Interacting display method and system for virtual arc-shaped large screen between buildings based on mixed reality

Abstract

The application provides a mixed reality-based inter-building virtual arc-shaped large-screen interactive display method, and belongs to the technical field of virtual reality / mixed reality. The mixed reality technology is used to position virtual 3D content to at least two building facades, so that different virtual screens on different building facades interact with each other in space, and the user is presented through a mobile terminal. As an application form of city MR, the application breaks the aesthetic fatigue of people to plane advertisements, uses diversified crowds to build more dynamic public spaces, and provides a new idea for the new form application of mixed reality in a city environment.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality / mixed reality technology, specifically relating to a method, system, electronic device, and storage medium for interactive display of a virtual curved large screen based on mixed reality. Background Technology

[0002] Currently, naked-eye curved 3D screens are gradually becoming a popular urban landscape. Their basic form is a 3D LED curved screen composed of a large number of display modules. The content displayed is either pre-produced video tailored to the screen's aspect ratio or real-time rendered 3D content. Naked-eye curved 3D screens utilize the principle of visual displacement and simultaneously adjust the sound system and lighting according to the video content to present a realistic 3D effect to the viewer. However, in practical applications, naked-eye curved 3D screens are limited by the viewer's viewing position, geographical constraints, and high costs, making it difficult to meet the commercial needs of large-scale deployment. Furthermore, they lack user interactivity.

[0003] Mixed Reality (MR) effects based on urban environments are also a hot application of virtual reality technology. They combine virtually generated 3D or 2D elements with real-world urban scenes such as buildings to create visual or interactive experiences that surpass the real environment. However, most existing urban MR experiences still lean towards simpler gamified or artistic experiences, and more practical applications, such as commercial advertising, art, and public information dissemination, are yet to be seen.

[0004] Therefore, there is an urgent need for a technology that can improve the interactivity of 3D curved screens and the market practicality of urban MR. Summary of the Invention

[0005] This invention provides a method, system, electronic device, and storage medium for interactive display of virtual curved screens between buildings based on mixed reality, in order to overcome at least one technical problem existing in the prior art.

[0006] To achieve the above objectives, the present invention provides a method for interactive display of virtual curved screens between buildings based on mixed reality, comprising:

[0007] Based on a deep learning object detection model, the location of buildings in selected image frames is detected in real time to identify at least two buildings participating in the virtual display.

[0008] Based on a deep learning planar recognition model, the position and angle of the building facade are extracted, and the placement of the virtual curved screen on the building facade and its placement position on the building facade are determined according to the position and angle.

[0009] A pre-fabricated curved screen based on pre-constructed 3D spatial geometric primitives is attached to the placement position of the placement surface to form a virtual 3D curved screen. The virtual 3D curved screen presents the display effect of a 3D curved screen at the placement position.

[0010] Customized 3D elements are placed in the virtual 3D curved screen to create a virtual-real fusion between the curved screen 3D space and the customized 3D elements.

[0011] Furthermore, preferably, after forming the virtual 3D curved screen, it also includes:

[0012] The interactive elements provided by the client are placed in the virtual 3D curved screen to form a virtual-real fusion between the 3D space of the curved screen and the client.

[0013] To address the aforementioned problems, this invention also provides a virtual curved large-screen interactive display system for buildings based on mixed reality, comprising:

[0014] The virtual building identification unit is used to detect the location of buildings in a selected image frame in real time based on a deep learning object detection algorithm, so as to identify at least two buildings participating in the virtual display.

[0015] The display location determination unit is used to extract the position and angle of the exterior facade of the building to be displayed based on a deep learning plane recognition algorithm, and determine the placement facade of the virtual curved screen to be displayed and the placement position on the placement facade based on the position and angle.

[0016] The virtual space bonding unit is used to bond a pre-constructed 3D curved screen based on pre-built 3D spatial geometric primitives to the placement position of the placement surface to form a virtual 3D curved screen. The virtual 3D curved screen presents the display effect of a 3D curved screen at the placement position.

[0017] A customized element fusion unit is used to place customized 3D elements in the virtual 3D curved screen to form a virtual-real fusion of the curved screen 3D space and the customized 3D elements.

[0018] To address the above problems, the present invention also provides an electronic device, comprising:

[0019] Memory, storing at least one instruction; and

[0020] The processor executes the instructions stored in the memory to implement the steps in the above-described method for interactive display of virtual curved screens between buildings based on mixed reality.

[0021] To address the aforementioned issues, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned method for interactive display of virtual curved screens between buildings based on mixed reality.

[0022] This invention provides a method, system, electronic device, and storage medium for interactive display of virtual curved screens between buildings based on mixed reality. By using mixed reality technology, virtual 3D content is positioned on the facades of at least two physical buildings, creating virtual screens on different building facades that interact spatially with each other, and then presented to the user via a mobile device. As an application of urban MR, this invention breaks through people's aesthetic fatigue with print advertising, while using the experience of the new technology itself as a driving force for users. It utilizes diverse groups to create more vibrant public spaces, providing new ideas for the application of mixed reality in urban environments. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating a method for interactive display of virtual curved screens between buildings based on mixed reality, according to an embodiment of the present invention.

[0025] Figure 2 A schematic diagram of the application architecture of a virtual curved screen interactive display method between buildings based on mixed reality provided in an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of the basic process and division of labor of each module for generating and processing user interaction on a virtual curved screen, provided by an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the display and interaction effects rendered by the curved screen 3D space interaction and rendering module provided in an embodiment of the present invention.

[0028] Figure 5 A schematic diagram of the logical structure of a virtual curved screen interactive display system between buildings based on mixed reality, provided in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the internal structure of an electronic device that implements a virtual curved screen interactive display method between buildings based on mixed reality, according to an embodiment of the present invention.

[0030] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0031] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0032] The purpose of this invention is to provide a method for rendering multiple virtual curved screens with good fusion effects in real time based on the facades of multiple buildings in an urban environment, and a corresponding system that enables multi-mode interaction between several virtual curved screens in multiple application scenarios. The virtual curved screen interaction effect refers to using mixed reality technology to position virtual 3D content onto the facades of physical buildings, forming multiple virtual screens that interact spatially with each other, and presenting this to the user via a mobile device.

[0033] It is evident that the interactive virtual screen provided by the method of this invention, as an application form of urban MR, can break people's aesthetic fatigue with print advertisements, thereby using the experience of the new technology itself as the driving force for users, utilizing diverse groups to build more vibrant public spaces, and providing new ideas for the new forms of mixed reality application in urban environments.

[0034] The specific implementation of the present invention will be described in further detail below with reference to the accompanying drawings.

[0035] Figure 1 This is a flowchart illustrating a method for interactive display of a virtual curved screen between buildings based on mixed reality, according to an embodiment of the present invention. This method can be executed by a system, which can be implemented in software and / or hardware. (Refer to...) Figure 1 As shown, in this embodiment, the virtual curved screen interactive display method between buildings based on mixed reality includes steps S110 to S140.

[0036] S110: Based on a deep learning object detection model, the location of buildings in selected image frames is detected in real time to determine at least two buildings participating in the virtual display.

[0037] S120: Based on a deep learning planar recognition model, extract the position and angle of the building facade, and determine the placement surface of the virtual curved screen to be displayed on the building facade and the placement position on the placement surface based on the position and angle.

[0038] S130: The pre-fabricated curved screen 3D space based on the pre-constructed 3D spatial geometric primitives is attached to the placement position of the placement surface to form a virtual 3D curved screen. The virtual 3D curved screen presents the display effect of a 3D curved screen at the placement position.

[0039] S140: Place customized 3D elements in the virtual 3D curved screen to form a virtual-real fusion of the curved screen 3D space and the customized 3D elements.

[0040] Furthermore, in order to enhance the client's participation and / or interaction with the displayed virtual 3D content, after the virtual 3D curved screen is formed, an interactive element fusion step may be included, that is, the interactive elements provided by the client are placed in the virtual 3D curved screen to form a virtual-real fusion of the curved screen 3D space and the client.

[0041] Figure 2 This is a schematic diagram of the application architecture of a virtual curved screen interactive display method between buildings based on mixed reality, provided in an embodiment of the present invention.

[0042] like Figure 2 As shown, the application architecture of this invention mainly consists of two parts: a "mobile terminal" and a "model training terminal." The "mobile terminal" can be a smartphone, tablet, or other device, while the "model training terminal" is the host providing deep learning training services. In the "model training terminal," deep learning neural networks are trained for object detection and planar detection of urban buildings, and the results are exported to the corresponding recognition module on the mobile terminal in a generalized deep neural network format. The overall architecture adopts a modular design pattern. Figure 3 This demonstrates the basic process of generating and processing user interactions on a virtual curved screen, as well as the division of labor among the modules, when a user scans a building with their mobile phone.

[0043] like Figure 1 , Figure 2 and Figure 3As shown in the diagram, when a user uses a mobile hardware platform (such as a smartphone or tablet) to capture video footage of city buildings, the system first uses a deep learning object detection algorithm to detect the location of buildings in the image frame in real time. Next, using a deep learning planar recognition algorithm, the system extracts the position and angle of the building facades. After sorting and filtering, suitable facades and specific locations for placing the virtual curved screen are determined. Then, pre-constructed 3D spatial geometric primitives are manipulated to merge and display the building facades on the virtual curved screen, creating a 3D curved screen effect. Various necessary 3D elements are then placed on the virtual 3D curved screen, resulting in a highly realistic viewing experience with excellent virtual-real fusion. Furthermore, in addition to the one-way presentation on the virtual 3D curved screen, several interactive modes are provided to allow users to interact with multiple 3D virtual curved screens based on rich control and presentation elements.

[0044] Since virtual curved screens do not require the construction of corresponding media on-site, and the intelligent manipulation of geometric primitives can automatically fit the real-world building facades, this means significant savings in hardware costs during development. Furthermore, the development of interactive content is no longer limited by screens with fixed aspect ratios. Simultaneous recognition of multiple virtual screens also allows content developers to effectively utilize the space between buildings, creating bolder and more innovative urban MR content with less geographical constraint, better meeting diverse application needs including advertising, art displays, and public information dissemination.

[0045] Meanwhile, since 3D objects in virtual space change their orientation during initialization based on the mobile user's location and the screen's display position to ensure a good viewing experience (similar to automatically adjusting for the optimal viewpoint), the generation logic for virtual content with variable orientation can be programmed into the 3D engine, allowing its generation angle to be adjusted based on data transmitted by the planar recognition component. This arrangement of virtual content with variable orientation breaks the limitations on the user's field of view during 3D glasses-free large-screen viewing. This is another major improvement of this invention compared to existing 3D glasses-free large-screen displays.

[0046] Specifically, as an example, when a user takes a picture of a building using a mobile terminal such as a smartphone, the various modules on the mobile terminal perform the following series of processing on the photographed building.

[0047] First, the shooting and frame extraction module on the mobile terminal continuously extracts image frames from the real-time images and / or video stream captured by the user to determine the selected image frames. The basis for image frame extraction can be a preset time interval, such as selecting an image frame for detection every 25ms. Of course, other specific parameters can also be used as the basis for image frame extraction, such as the angle of movement of the mobile terminal lens and the range of movement.

[0048] Then, the building's 2D object detection module is based on... Figure 1 The deep learning object detection model trained on the mid-model training end identifies at least two credible building targets from the image frames (selected image frames) extracted by the shooting and frame extraction modules as the buildings to be displayed in the virtual presentation, and provides the corresponding bounding boxes. For situations with numerous buildings, the 2D object detection module for each building can select the building bounding box that is most visually appealing to the user based on factors such as bounding box area and bounding box position, and then divide the extracted image frames into blocks based on the selected building bounding boxes.

[0049] Specifically, as an example, the YOLOv5 model can be used as a deep learning object detection model for object detection. The basic architecture of the YOLOv5 model includes four parts: input, backbone, neck, and prediction.

[0050] (1) Input: Mosaic data augmentation, adaptive anchor box calculation, adaptive image scaling;

[0051] (2) Backbone: Focus structure, CSP structure;

[0052] (3) Neck: FPN+PAN structure;

[0053] (4) Prediction: GIOU_Loss.

[0054] The YOLOv5 model determines whether an object in an image belongs to a certain category and what the confidence level is. Detected objects are displayed in the image as bounding boxes, and their category and corresponding confidence level are given; therefore, the position and size of the bounding boxes can be determined.

[0055] The training method for this YOLOv5 model is as follows:

[0056] (1) Prepare relevant video materials, set the interval for taking pictures at a certain number of frames, and form a picture set;

[0057] (2) Confirm the categories to be labeled in the image set (including buildings) and draw the target boxes for the objects in the image set, generating data such as border position and category;

[0058] (3) Divide the labeled materials into test set, training set and validation set, and feed them into YOLOv5 model for training.

[0059] When using the YOLOv5 model for real-world target recognition, if multiple buildings exist simultaneously, objects with larger bounding box areas and relatively centered on the screen will be selected for retention. These retained targets will be segmented based on their bounding box boundary information and then fed into the system for further processing.

[0060] More specifically, the locations of buildings in selected image frames are detected in real time to identify at least two buildings participating in the virtual display, including:

[0061] S121: Perform 2D building target detection on the selected image frame to determine the RGB image of each building in the selected image frame;

[0062] S122: Based on the RGB image of the building, determine at least two buildings to participate in the virtual display and mark them with bounding boxes.

[0063] In the building plan detection module, it will be based on... Figure 1 The deep learning planar recognition model trained on the model training end detects the main facade of the building in the segmented image to extract the position and angle of the building's facade.

[0064] Specifically, as an example, in one specific implementation of the present invention, the deep learning plane recognition model employs a Transformer-based network structure. Based on a given input image and its detected line segments, the line segments and contextual features are encoded into two sets of tokenized sequences, respectively. Then, a set of learnable plane query information is used to enable a structure-guided plane decoder to interact holistically with the context. This plane decoder can output a set of tokenized plane instances. Finally, a segmentation strategy is used to assign each pixel to its nearest plane instance to produce a pixel-by-pixel plane segmentation result. During the training of the deep learning plane model, a dataset containing plane information can be used for training, and the plane (pixel) recall rate can be applied to evaluate the plane detection performance.

[0065] For cases where the exterior surface of a building is uneven, the building's planar inspection module also undertakes the task of integrating the scattered planar surfaces and outputting data such as the angle, position, and vertices of the main facade of the building based on the integrated planar surfaces.

[0066] Specifically, as an example, a distance threshold is set in the plane detection algorithm used, and then the distance between pixel embedding and instance plane embedding is compared based on the distance threshold. If the distance between the two is lower than the set distance threshold, the pixel is assigned to this instance plane. Since the detection object of this invention is a building, which is far away, there is a gap between the distance of protruding scattered planes and building facades and mobile terminals such as mobile phones. For the determination of the distance threshold, multiple buildings with complex facades can be used as samples. Through repeated experiments and adjustments to the threshold, the plane detection results are observed and the integration is recorded until the plane shown in the result is integrated into the main facade of the building visually, and excellent integration results are presented in multiple samples. The new distance threshold is then stored.

[0067] The planar group filtering module then determines the placement surface of the virtual curved screen to be displayed on the building and its placement position on the building based on the position and angle.

[0068] The method of determining the placement of the virtual curved screen in the building under display, based on the stated position and angle, further includes:

[0069] S131: Determine the placement facade of the virtual curved screen on the building to be displayed based on the position and angle;

[0070] S132: Determine the placement position of the virtual curved screen on the placement surface.

[0071] Specifically, firstly, based on the location and angle information of the identified plane, groups of protruding facades of adjacent buildings are selected. Within the same group of protruding facades, the plane with the shorter side among the adjacent sides is used as a reference, and the length of the adjacent side is used as the upper limit of the width of the virtual curved screen to be displayed; the distance between the adjacent side and its parallel boundary is used as the upper limit of the length of the virtual curved screen. This determines the placement facade of the virtual curved screen to be displayed on the building and its placement position on the facade at that location.

[0072] As an example, the planar group filtering module can integrate and group scattered planes based on facade information, including facade angles, edge information, and distance from the camera, to determine the most suitable planar group for placing the virtual curved screen, and store the relevant data information of these planar groups. During the integration, grouping, and filtering process, the planar group filtering module will determine the number of faces of the corresponding virtual screen and the size ratio of each face based on the above data, thereby determining the parameter information of each scattered plane in the planar group and the information of the planar group after combining the scattered planes. This information is then output as the relevant data information of the planar group (real-time registration of planar information) to the curved screen 3D spatial positioning and fitting module.

[0073] Since the prefabricated curved screen 3D space is a cuboid space with controllable vertices and textures based on pre-constructed 3D spatial geometric primitives, the curved screen 3D space positioning and fitting module uses the vertices, plane normals, and position data of the selected plane group to position and fit the virtual space on the building. That is, the prefabricated curved screen 3D space is fitted to the placement position of the facade to form a virtual 3D curved screen, so that the virtual 3D curved screen presents the display effect of a 3D curved screen at the placement position.

[0074] The foundation of the pre-built curved screen 3D space is a pre-set 3D content library. This library stores relevant materials provided by the screen projection client or created according to their needs, such as object models, skeletal rigging files, animations, textures, and lighting files. The curved screen 3D space is initialized based on this pre-set 3D content library. In the 3D engine, initial positions are set for each element, they are arranged to form a scene, behavioral logic is set for elements that appear only after interaction, and lighting is set and baked.

[0075] Finally, in the curved screen 3D space interaction and rendering module, based on the user's mobile phone location and the building location, customized 3D elements are placed in the virtual 3D curved screen. After correcting the orientation of the pre-made 3D elements, the complete scene is rendered in real time on the user's mobile phone, thus forming a virtual-real fusion of the curved screen 3D space and the customized 3D elements. The logic for placement, correction, and rendering is similar to the aforementioned logic for generating virtual content with variable orientation in the 3D engine. The virtual scene and some 3D elements requiring orientation adjustment are arranged in the 3D engine. During initialization, these 3D elements receive the location of the user's mobile terminal (such as a phone) and the building location, controlling their generated angle to ensure they are rendered on the user's screen with the correct orientation.

[0076] The customized 3D elements include at least one of advertising 3D elements, art display 3D elements, and public information 3D elements.

[0077] Furthermore, to enhance client interactivity, it can also respond to user touch screen operations, meet user control needs in different interaction modes, and present a high-quality interactive effect between multiple virtual curved screens.

[0078] Specifically, as an example, the processing method for responding to user touch screen operations is as follows: in the case of a single point, the position and number of times the user taps; in the case of dragging / sliding, the position of the user pressing and the position of releasing are recorded, and the pressing and releasing actions can be recorded.

[0079] Figure 4The document showcases four example display and interaction effects rendered by the curved screen 3D space interaction and rendering module during the aforementioned process. Effect A, comparable to a 3D naked-eye large screen, displays virtual content and advertisements within the virtual curved screens on each building facade. Effect B features virtual objects automatically traversing between the virtual curved screens on two building facades, accompanied by pre-made animations and path randomness. Effect C allows users to perform an "aiming-launch" operation similar to "Angry Birds." Effect D displays 3D characters resembling small animals automatically jumping from one virtual screen space to another, requiring users to dynamically create aerial passages between the two buildings at appropriate times using touch swipes on the screen, providing landing points for the jumping characters. Figure 4 The application examples shown demonstrate the potential and value of virtual 3D curved screens and their interactive effects in various applications such as advertising, art displays, public information dissemination, and entertainment.

[0080] In summary, the mixed reality-based virtual curved screen interactive display method for buildings provided by this invention uses mixed reality technology to position virtual 3D content onto the facade of a physical building, forming multiple virtual screens that interact spatially with each other. This is then presented to users via mobile devices, breaking through the aesthetic fatigue of traditional print advertising. Furthermore, by using the experience of the new technology itself as a driving force for users, and leveraging diverse demographics to create more vibrant public spaces, this invention provides new ideas for the application of mixed reality in urban environments.

[0081] Corresponding to the above-mentioned interactive display method for virtual curved screens between buildings based on mixed reality, the present invention also provides an interactive display system for virtual curved screens between buildings based on mixed reality.

[0082] like Figure 5 As shown, this invention provides a mixed reality-based virtual curved large-screen interactive display system 500 for buildings, which can be installed in an electronic device. Depending on the functions implemented, the mixed reality-based virtual curved large-screen interactive display system 500 may include a virtual display building determination unit 510, a display position determination unit 520, a virtual space fitting unit 530, and a customized element fusion unit 540. The unit described in this invention can also be called a module, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0083] In this embodiment, the functions of each module / unit are as follows:

[0084] The virtual building identification unit 510 is used to detect the location of buildings in a selected image frame in real time based on a deep learning object detection algorithm, so as to identify at least two buildings participating in the virtual display.

[0085] The display location determination unit 520 is used to extract the position and angle of the exterior facade of the building to be displayed based on a deep learning plane recognition algorithm, and determine the placement facade of the virtual curved screen to be displayed and the placement position on the placement facade based on the position and angle.

[0086] The virtual space bonding unit 530 is used to bond the pre-constructed curved screen 3D space based on the pre-built 3D spatial geometric primitives to the placement position of the placement surface to form a virtual 3D curved screen. The virtual 3D curved screen presents the display effect of a 3D curved screen at the placement position.

[0087] The customized element fusion unit 540 is used to place customized 3D elements in the virtual 3D curved screen to form a virtual-real fusion of the curved screen 3D space and the customized 3D elements.

[0088] The specific implementation details of the building virtual curved screen interactive display system 500 based on mixed reality of the present invention can be referred to the above description of the embodiments of the building virtual curved screen interactive display method based on mixed reality, and will not be described in detail here.

[0089] Similarly, this building-based virtual curved screen interactive display system, based on mixed reality, positions virtual 3D content onto the facade of a physical building using mixed reality technology. This creates multiple virtual screens that interact spatially and are presented to users via mobile devices, breaking the aesthetic fatigue associated with print advertising. Furthermore, by using the experience of the new technology itself as a driving force for users, it leverages diverse demographics to create more vibrant public spaces, providing new ideas for the application of mixed reality in urban environments.

[0090] Figure 6 An electronic device 6 is provided for applying a mixed reality-based virtual curved screen interactive display method between buildings according to an embodiment of the present invention.

[0091] like Figure 6As shown, the electronic device 6 may include a processor 60, a memory 61, and a bus. It may also include a computer program stored in the memory 61 and executable on the processor 60, such as a mixed reality-based virtual curved screen interactive display program 62 for buildings. The memory 61 may include both internal storage units for the mixed reality-based virtual curved screen interactive display method and external storage devices. The memory 61 can be used not only to store application software and various types of data installed, such as the code for the mixed reality-based virtual curved screen interactive display program, but also to temporarily store data that has been output or will be output.

[0092] The memory 61 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the electronic device 6, such as the portable hard drive of the electronic device 6. In other embodiments, the memory 61 can be an external storage device of the electronic device 6, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 6. Furthermore, the memory 61 can include both internal and external storage units of the electronic device 6. The memory 61 can be used not only to store application software and various types of data installed on the electronic device 6, such as the code of a virtual curved screen interactive display program based on mixed reality, but also to temporarily store data that has been output or will be output.

[0093] In some embodiments, the processor 60 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 60 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 61 (e.g., a mixed reality-based virtual curved screen interactive display program between buildings), and calls data stored in the memory 61 to perform various functions and process data in the electronic device 5.

[0094] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 61 and at least one processor 60, etc.

[0095] Figure 6 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 6 The structure shown does not constitute a limitation on the electronic device 6, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0096] For example, although not shown, the electronic device 6 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 60 through a power management system, thereby enabling functions such as charging management, discharging management, and power consumption management through the power management system. The power supply may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 6 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0097] Furthermore, the electronic device 6 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 6 and other electronic devices.

[0098] Optionally, the electronic device 6 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 6 and to display a visual user interface.

[0099] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0100] The memory 61 in the electronic device 6 stores a mixed reality-based virtual curved screen interactive display program 62, which is a combination of multiple instructions. When running in the processor 60, it can achieve the following: S110: Based on a deep learning object detection model, the location of buildings in a selected image frame is detected in real time to determine at least two buildings participating in the virtual display; S120: Based on a deep learning plane recognition model, the position and angle of the facade of the display building are extracted, and the placement surface of the virtual curved screen to be displayed on the display building and the placement position on the placement surface are determined according to the position and angle; S130: The 3D space of the curved screen, prefabricated based on pre-constructed 3D spatial geometric primitives, is attached to the placement position on the placement surface to form a virtual 3D curved screen, which presents the display effect of a 3D curved screen at the placement position; S140: Customized 3D elements are placed in the virtual 3D curved screen to form a virtual-real fusion of the curved screen 3D space and the customized 3D elements.

[0101] Specifically, the specific implementation method of the processor 0 for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here. It should be emphasized that, to further ensure the privacy and security of the above-mentioned virtual curved screen interactive display program based on mixed reality between buildings, the high-availability processing data of the above database is stored in the blockchain node where this server cluster is located.

[0102] Furthermore, if the modules / units integrated in the electronic device 6 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0103] This invention also provides a computer-readable storage medium, which may be non-volatile or volatile. The storage medium stores a computer program, which, when executed by a processor, implements the following: S110: Based on a deep learning object detection model, real-time detection is performed on the building positions in a selected image frame to determine at least two buildings participating in the virtual display; S120: Based on a deep learning planar recognition model, the position and angle of the facade of the display buildings are extracted, and the placement surface of the virtual curved screen to be displayed and its placement position on the placement surface are determined according to the position and angle; S130: The 3D space of the curved screen, prefabricated based on pre-constructed 3D spatial geometric primitives, is attached to the placement position on the placement surface to form a virtual 3D curved screen, which presents the display effect of a 3D curved screen at the placement position; S140: Customized 3D elements are placed in the virtual 3D curved screen to form a virtual-real fusion of the curved screen 3D space and the customized 3D elements.

[0104] Specifically, the specific implementation method of the computer program when executed by the processor can be referred to the description of the relevant steps in the embodiment of the virtual curved screen interactive display method between buildings based on mixed reality, which will not be repeated here.

[0105] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0106] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0108] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0109] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0110] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0111] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim may also be implemented by a single unit or system through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for interactive display of virtual curved screens between buildings based on mixed reality, characterized in that, Using mixed reality technology, virtual 3D content is positioned onto the exterior facade of a physical building, creating multiple virtual screens that interact spatially with each other. Methods include: Based on a deep learning object detection model, the location of buildings in selected image frames is detected in real time to identify at least two buildings participating in the virtual display. Based on a deep learning planar recognition model, the position and angle of the building facade are extracted, and the placement of the virtual curved screen to be displayed on the building facade and its placement position on the building facade are determined according to the position and angle. A pre-fabricated curved screen based on pre-constructed 3D spatial geometric primitives is attached to the placement position of the placement surface to form a virtual 3D curved screen. The virtual 3D curved screen presents the display effect of a 3D curved screen at the placement position. Customized 3D elements are placed in the virtual 3D curved screen to create a virtual-real fusion between the curved screen 3D space and the customized 3D elements.

2. The interactive display method for virtual curved screens between buildings based on mixed reality according to claim 1, characterized in that, The training method for the deep learning object detection model includes: Extract image frames from video footage according to a preset frame extraction interval to form an image frame set; The image frame set is categorized, and the objects in the image frame set are drawn with bounding boxes to generate the bounding box positions and category parameters of the building targets. Based on the bounding box position and category parameters, the image frame set after category labeling and target box drawing is divided into a test set, a training set, and a validation set. The deep learning object detection model is trained based on the test set, training set, and validation set.

3. The interactive display method for virtual curved screens between buildings based on mixed reality according to claim 1 or 2, characterized in that, The selected image frame is determined based on frame extraction from the captured image and / or video stream.

4. The interactive display method for virtual curved screens between buildings based on mixed reality according to claim 1 or 2, characterized in that, Real-time detection of building locations within selected image frames is performed to identify at least two buildings participating in the virtual display, including: 2D building target detection is performed on the selected image frame to determine the RGB image of each building in the selected image frame; Based on the RGB images of the buildings, at least two buildings are identified to participate in the virtual display.

5. The interactive display method for virtual curved screens between buildings based on mixed reality according to claim 1, characterized in that, The customized 3D elements include at least one of advertising 3D elements, art display 3D elements, and public information 3D elements.

6. The interactive display method for virtual curved screens between buildings based on mixed reality according to claim 1, characterized in that, The deep learning object detection model and the deep learning plane recognition model are deep learning neural network models trained based on deep learning neural networks.

7. A virtual curved large-screen interactive display system for buildings based on mixed reality, characterized in that, The system uses mixed reality technology to project virtual 3D content onto the exterior of a physical building, creating multiple virtual screens that interact with each other in space. The system includes: The virtual building identification unit is used to detect the location of buildings in a selected image frame in real time based on a deep learning object detection algorithm, so as to identify at least two buildings participating in the virtual display. The display location determination unit is used to extract the position and angle of the exterior facade of the building to be displayed based on a deep learning plane recognition algorithm, and determine the placement facade of the virtual curved screen to be displayed and the placement position on the placement facade based on the position and angle. The virtual space bonding unit is used to bond a pre-constructed 3D curved screen based on pre-built 3D spatial geometric primitives to the placement position of the placement surface to form a virtual 3D curved screen. The virtual 3D curved screen presents the display effect of a 3D curved screen at the placement position. A customized element fusion unit is used to place customized 3D elements in the virtual 3D curved screen to form a virtual-real fusion of the curved screen 3D space and the customized 3D elements.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the steps of the building virtual curved screen interactive display method based on mixed reality as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the interactive display method for virtual curved screens between buildings based on mixed reality as described in any one of claims 1 to 6.

Images