An Augmented Reality-Based Cultural Interaction Method, Device, and Storage Medium

Abstract

The present invention discloses a cultural interaction method, device and storage medium based on augmented reality. The device includes: a physical display device and an augmented reality system deployed in a mobile terminal; the physical display device includes a light painting module, and the light painting module can rotate on a turntable; the light painting module is surrounded by several display surfaces, and display boards are arranged on the display surfaces; the display boards include a character foreground board and a background board; a light source is arranged in the cavity surrounded by the display surfaces. The augmented reality system includes an appreciation mode, an interaction mode and a creative mode. In the present invention, the role is not only defined as the audience, but also he is a participant in the performance. The preset action data can improve the performance effect of the user controlling the virtual character, and the real-time action editing supported by the multi-touch technology allows the user to modify the generated action content on the basis of the preset actions, increasing the cultural participation degree.

Description

[0001] Priority Application

[0002] This application claims priority from the Chinese invention patent application [Application No.: 202410166119X] titled "

Title of Invention: An Augmented Reality-based Cultural Interaction Method, Device, and Storage Medium

[0003] The present invention belongs to the field of augmented reality technology, and particularly relates to an augmented reality-based cultural interaction method, device, and computer-readable storage medium. Background Art

[0004] Chinese opera culture is world-famous for its unique singing and narrative style. Its performances delve into the narrative and character relationships, serving as a bridge to convey China's rich cultural thoughts and historical depth, and reproducing historical events and development trajectories of ancient Chinese cultural thoughts.

[0005] For decades, Chinese institutions have been committed to promoting cultural inheritance. Digital resources such as texts, images, and videos have been used as valuable media to preserve local opera performances and their hidden cultural thoughts, transforming abstract theories into art and bringing history and cultural thoughts to life. However, this passive viewing mode may not effectively attract the general public. Therefore, it is necessary to explore a new communication mode to present cultural performances in a more immersive way. An emerging trend is to use augmented reality (AR) to share experiences in cultural narratives because of its advantages in display scene creation, storytelling, and active exploration, and its combination with tangible interfaces, situational visualization, game mechanisms, etc., to promote multi-sensory embodied interaction.

[0006] On the other hand, cultural performances are often limited by time and space. Therefore, we want to design a hybrid cultural and creative product as a carrier in a shared cultural experience, apply it to the display scenes of users' daily life, and promote a cultural interaction experience in which users actively participate.

[0007] The prior art CN109692487A discloses an augmented reality operation method for Huangmei Opera. The technical solutions described in this document are mainly used for real theaters and rely on the installation of equipment such as cameras, sensors, computing and storage devices, and display devices on-site. Motion capture is also based on the real-time motion data of actors.

[0008] The prior art CN102688603A discloses a system and method for real-time magic stage performances based on augmented reality and action recognition technologies. The technical solutions recorded therein are mainly used for real theaters and rely on the installation of devices such as cameras, projectors, and holographic images on-site. In stage performances, it is composed of the interaction between the performances of some real actors and the augmented information images.

[0009] The prior art CN109725706A discloses an augmented reality-based character interaction system. The technical solutions recorded therein only involve presenting augmented reality content of multiple recognition images through non-functional devices in the shape made of different physical materials. Summary of the Invention

[0010] The purpose of the present invention is to provide an augmented reality-based cultural interaction method and interaction device, which improve the efficiency of cultural interaction through augmented reality technology.

[0011] To solve the above-mentioned technical problems, the present invention specifically adopts the following technical solutions: An augmented reality-based cultural interaction device, including: a physical display device and an augmented reality system deployed in a mobile terminal;

[0012] The physical display device includes a light painting module, and the light painting module can rotate on a turntable;

[0013] The light painting module is surrounded by several display surfaces, and display boards are arranged on the display surfaces; the display boards include a character foreground board and a background board; a light source is arranged in the cavity surrounded by the display surfaces, so that only the pattern on the character foreground board is visible when the light source is turned off, and the patterns on both the character foreground board and the background board are visible when the light source is turned on;

[0014] The augmented reality system includes three modes;

[0015] In Mode I, in response to the user's scanning of the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the physical display device image in the display scene for display. The virtual component three-dimensional model includes event signs in the display scene; the event signs are generated according to the events occurring in the corresponding time period of the display board, and each event sign represents an event that a character has in the corresponding time period of the display board; the event signs can move with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event signs are located on the occurrence ground of the event on the map, and the distance between the event signs and the map plane is determined according to the event occurrence time;

[0016] In Mode II, in response to the user's scanning of the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; according to the display board image data scanned by the user, a virtual character three-dimensional model of the character in the corresponding time period is displayed; and in response to the user's operation, the virtual object three-dimensional model is used for setting the scene;

[0017] In Mode III, in response to the user's scanning of the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture is editable.

[0018] As an improvement, the foreground board of the character is made of PP paper, and the background board is made of light sheet; the display board further includes a transparent acrylic board laid in front of the foreground board of the character.

[0019] As an improvement, the light painting module is square-columnar, and the side wall of the square-columnar light painting module is the display surface; a hollow is opened on the display surface, and the display board is embedded in the hollow.

[0020] The present invention also provides an augmented reality-based cultural interaction method, including:

[0021] Obtaining the classification result of the humanistic theme of interest to the user;

[0022] Analyzing the corresponding character, a number of events centered on the character, and the time when the events occur according to the classification result of the humanistic theme of interest to the user;

[0023] Segmenting a number of events according to the occurrence time to form a number of time periods, and designing corresponding character shapes and backgrounds for each time period;

[0024] Preparing a display board according to the character shape and background of each time period, and fixing the display board to the display surface of the physical display device;

[0025] In Mode I, in response to the user's scanning of the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display, and the virtual component three-dimensional model includes an event board in the display scene; the event board is generated according to the events occurring in the corresponding time period of the display board, and each event board represents an event that the character has in the time period corresponding to the display board; the event board can move along with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event board is located on the place where the event occurs on the map, and the distance between the event board and the map plane is determined according to the event occurrence time;

[0026] In Mode II, in response to the user's scanning of the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; according to the display board image data scanned by the user, a virtual character three-dimensional model of the character in the corresponding time period is displayed; and in response to the user's operation, the virtual object three-dimensional model is used for setting the scene;

[0027] In Mode III, in response to the user's scanning of the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture is editable.

[0028] As an improvement, in Mode I, the virtual component three-dimensional model includes the event board in the display scene; the event board is generated according to the events occurring in the corresponding time period of the display board, and each event board represents an event that the character has in the corresponding time period of the display board; the event board can move along with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event board is located at the place where the event occurs on the map, and the distance between the event board and the map plane is determined according to the event occurrence time.

[0029] As an improvement, the creation of the event board specifically includes:

[0030] S501. Establish the three-dimensional coordinates of the display board in the display scene based on the position of the display board in the physical display device, where the length and width of the display board correspond to the x and y axes, and the thickness direction of the display board corresponds to the z axis;

[0031] S502. According to the events occurring in the corresponding time period on each display board, use Adobe illustrator to draw the corresponding event board and model it through Rhino;

[0032] S504. The place where the event corresponding to the event board occurs is used as the x coordinate and y coordinate of the event board on the corresponding display board, and the event occurrence time is used as the z coordinate.

[0033] As an improvement, in Mode I, by rotating the physical display device or / and adjusting the scanning angle of the mobile terminal, the scanned display board is switched, so as to display different scanned display boards, and according to the scanned display board image data, the virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display.

[0034] As an improvement, in Mode I, in response to the user's operation, a Chinese opera video is played.

[0035] As an improvement, in Mode II, in response to the user's operation, corresponding addition, deletion, dragging, rotation, and scaling operations are performed on the virtual object three-dimensional model.

[0036] As an improvement, in Mode II, in response to a user's operation, a text introduction corresponding to an event on a display board in the current display scene is presented.

[0037] As an improvement, in Mode II, in response to a user's operation, the singing mode is entered; the singing track corresponds to the event on the display board in the current display scene; in the singing mode, different users are respectively responsible for singing and scene shooting.

[0038] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned cultural interaction method based on augmented reality is implemented.

[0039] The advantages of the present invention are as follows:

[0040] The entity display device in the present invention is based on light sources, such as LEDs and special paper materials, so that the images recognized in different states (for example, when the light source is on and when the light source is off) can be presented at the same position. The static images in the two lighting states in the present invention have picture relevance, and both the metaphor and narrative of cultural interaction are coherent, which is suitable for presenting multi-level cultural interaction content. Moreover, by scanning the picture on the display board under the light source, a corresponding event card is generated (this event card represents the location and time of the event), so that users can deeply understand the corresponding document, and different display boards can also be switched by rotating the entity display device, so as to obtain the locations and times of different events, and further deeply understand the document. At the same time, this interaction method also greatly improves the user's participation and enhances the user experience. The present invention not only involves 2D image recognition, but also involves 3D image recognition of the whole device. At the same time, the lighting function design of the device gives the present invention a certain practicality in life.

[0041] The cultural interaction method in the present invention uses physical objects as carriers for image positioning, and the motion capture is also based on the professional actor motions pre-captured during development, which greatly reduces the requirements for equipment. As long as a mobile device such as a mobile phone or a tablet is used, the user can participate in the experience without being restricted by time and space. And as an amateur cultural experiencer, the role in the present invention is not only defined as an audience, but also as a participant in the performance. The preset motion data can improve the performance effect of the user controlling the virtual character, and the real-time motion editing supported by the multi-touch technology allows the user to modify the generated motion content on the basis of the preset motions, increasing the cultural participation. Description of the Drawings

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale. Obviously, the following-described drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0043] Figure 1a Structural schematic diagram of the entity display device in the present invention;

[0044] Figure 1b For Figure 1a Exploded view of the shown entity display device;

[0045] Figure 2a Flowchart of the augmented reality-based cultural interaction method according to an embodiment of the present invention;

[0046] Figure 2b Flowchart of an embodiment for segmenting several events;

[0047] Figure 3 Schematic diagram of the generation process of the event card.

[0048] Markings in the figure: 1 acrylic board, 2 foreground board of the figure, 3 background board, 4 acrylic board, 5 frame, 6 top shape, 7 connecting part shape, 8 base, 9 light source, 10 USB charging port, 11 touch button, 12 bearing, 13 bolt, 14 anti-slip pad. Detailed implementation manners

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0050] In this article, suffixes such as "module", "component", or "unit" used to represent elements are only for the convenience of the description of the present invention, and they have no specific meaning themselves. Therefore, "module", "component", or "unit" can be used interchangeably.

[0051] In this text, terms such as "upper", "lower", "inner", "outer", "front", "rear", "one end", "the other end", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0052] In this text, unless otherwise clearly specified and defined, terms such as "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] In this text, "and / or" includes any and all combinations of one or more of the listed related items.

[0054] In this text, "a plurality of" means two or more, that is, it includes two, three, four, five, etc.

[0055] Embodiment 1

[0056] As Figure 1a and Figure 1b shown, the present invention provides an augmented reality-based cultural interaction device (or interaction system), including: a physical display device and an augmented reality system deployed in a mobile terminal.

[0057] Through the mutual cooperation of the physical display device and the augmented reality system, a scene combining reality and virtuality is displayed on the display screen of the mobile terminal, including smartphones, tablets, etc. Simply put, the camera on the mobile terminal is used to collect the real scene including the physical display device, and the augmented reality system adds virtual content to the above real scene and fuses the virtual content with the real scene, so as to create a display scene combining display and virtuality on the screen of the mobile terminal.

[0058] The physical display device is a physical device existing in reality, which includes a light painting module, and the light painting module can rotate on a turntable.

[0059] The light painting module is surrounded by several display surfaces, and display boards are arranged on the display surfaces; the display boards include a foreground figure board 2 and a background board 3; a light source 9 is arranged in the cavity surrounded by the display surfaces, so that only the pattern on the foreground figure board 2 is visible when the light source 9 is turned off, and the patterns on both the foreground figure board 2 and the background board 3 are visible when the light source is turned on. When the user rotates the turntable, the foreground figure board 2 and the background board 3 representing different scenes can be presented to the user in the user interface displayed on the above-mentioned mobile terminal. Preferably, a figure is drawn on the foreground figure board 2. For example, Wang Yangming. Various scenes where different events occur are drawn on the background board 3.

[0060] Specifically, the foreground figure board 2 is made of PP paper, and the background board 3 is made of light sheet; the display board further includes a transparent acrylic board 1 laid in front of the foreground figure board 2. In order to improve the ornamental value, the pattern on the foreground figure board 2, that is, the PP paper, can be a black and white line drawing, and the pattern on the background board 3, that is, the light sheet, can be a color painting.

[0061] The light source 9 in this embodiment is an LED light source. For convenient control, a touch switch 11 is also provided. In addition, for the purpose of being portable, a rechargeable battery is also equipped. Of course, a USB charging port 10 for charging the battery is also provided.

[0062] The specific shape of the light painting module can be various. In this embodiment, in order to fit in with traditional culture, the light painting module is set in the shape of an ancient building, that is, the light painting module is square-columnar, and its main body is a wooden frame 5; the side wall of the square-columnar light painting module is the display surface; a hollow is opened on the display surface, and the display board is inlaid in the hollow. In addition, a decorative styling structure, such as a pavilion, etc., is arranged on the top of the light painting module. An acrylic board 1 is also arranged between the styling structure and the light painting module. More specifically, the styling structure includes a top styling 6 and a connecting part styling 7.

[0063] In addition, the light painting module can rotate on the turntable. The turntable specifically includes a base 8, and the base 8 is connected to the light painting module by a bearing 12 and a bolt 13. In addition, an anti-slip pad 14 is arranged at the bottom of the base 8.

[0064] The augmented reality system is software deployed on a mobile terminal and is used to provide corresponding virtual content according to the content of the display board in the physical display device. This is prior art and will not be elaborated here. However, differently, the augmented reality system in this embodiment includes three modes;

[0065] In Mode I, in response to the user scanning the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the physical display device image in the display scene for display; wherein, the virtual component three-dimensional model includes an event board in the display scene; the event board is generated according to the events occurring in the corresponding time period of the display board, and each event board represents an event that a person has in the corresponding time period of the display board; the event board can move along with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event board is located at the place where the event occurred on the map, and the distance between the event board and the map plane is determined according to the event occurrence time;

[0066] In Mode II, in response to the user scanning the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; according to the display board image data scanned by the user, a virtual human three-dimensional model of the person in the corresponding time period is displayed; and in response to the user's operation, a virtual object three-dimensional model is used for setting the scene;

[0067] In Mode III, in response to the user scanning the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture is editable.

[0068] Preferably, the augmented reality system specifically includes:

[0069] The first response module is used for, in Mode I, in response to the user scanning the display board through the mobile terminal, displaying a display scene including the physical display device on the user interface of the user's mobile terminal; and according to the display board image data scanned by the user, superimposing a virtual component three-dimensional model on the physical display device image in the display scene for display;

[0070] The second response module is used for, in Mode II, in response to the user scanning the display board through the mobile terminal, displaying a display scene including the physical display device on the user's mobile terminal; according to the display board image data scanned by the user, displaying a virtual human three-dimensional model of the person in the corresponding time period; and in response to the user's operation, using a virtual object three-dimensional model for setting the scene;

[0071] The third response module is used for, in Mode III, in response to the user scanning the display board through the mobile terminal, displaying a display scene including the physical display device on the user's mobile terminal; in response to the user's operation, taking a screenshot of the current display scene and saving it; the captured picture is editable.

[0072] Furthermore, the cultural interaction device further includes:

[0073] A data acquisition module for acquiring the classification results of the cultural themes of interest to the user.

[0074] An event data analysis module for parsing out the corresponding characters, a number of events centered around the character, and the location and time of the event occurrence according to the classification results of the cultural themes of interest to the user.

[0075] An event segmentation module for segmenting a number of events according to the occurrence time to form a number of time periods, and designing corresponding character models and backgrounds for each time period; wherein, the background includes at least the location or scene information where one or more events occur.

[0076] After designing the character models and backgrounds, it is only necessary to draw them on PP paper and slides respectively. The drawn PP paper and slides are installed on the display surface of the physical display device, and finally encapsulated with an acrylic board to avoid damage.

[0077] Embodiment 2

[0078] As Figure 2a shown, the present invention also provides an augmented reality-based cultural interaction method, which is applied to the augmented reality-based cultural interaction device in Embodiment 1, and specifically includes:

[0079] S1 Acquire the classification results of the cultural themes of interest to the user.

[0080] Cultural themes generally refer to themes related to human culture, history, art, society, etc., such as literature, music, philosophy, historical figures, major events, and social customs. These themes reflect human thoughts, emotions, and values. In the above text, the classification results of cultural themes may include specific historical periods, cultural phenomena, or art genres, etc.

[0081] The main purpose of the present invention is to carry forward traditional culture, so this embodiment is described around Yue Opera in traditional culture.

[0082] Yue Opera is one of the Chinese opera genres, originating from the Shengzhou area of Zhejiang Province and formed in the late Qing Dynasty and the early Republic of China. As an intangible cultural heritage of China, Yue Opera carries a rich historical and cultural heritage. It integrates local folk music, southern opera, Huangmei Opera and other elements, and forms a unique performance style and music system in the development process. The popularity of Yue Opera is inseparable from the contribution of many classic repertoires. Among them, the newly compiled classic repertoire of Yue Opera, "Wang Yangming", not only has unique features in the performance form and music, but also conveys rich inner emotions and ideological connotations by narrating Wang Yangming's life experience that emphasizes both "reason" and "emotion".

[0083] However, traditional stage performance forms are difficult to fully and vividly display the charm of this story. Especially for young audiences, traditional viewing methods may seem dull and outdated. As an emerging information technology means, augmented reality technology provides the public with a brand-new way of cultural experience. It can immerse the audience in a virtual environment, which has stronger interactivity and immersion compared with traditional stage viewing. Therefore, combining the story of the Yue opera "Wang Yangming" with augmented reality technology is not only conducive to the protection and inheritance of intangible cultural heritage but also better meets the diverse needs of contemporary audiences for cultural experience.

[0084] In this embodiment, based on the IBM SPSS23.0 software, the average and standard deviation algorithms are used to analyze the data of online questionnaire surveys and expert interviews to obtain the interest clustering results of users in terms of cultural experience. The specific steps include:

[0085] S101: Conduct an online questionnaire survey based on the Likert scale, collect, organize, and delete abnormal data using Microsoft Forms to generate a cleaned dataset.

[0086] S102: Based on the cleaned dataset, use the average algorithm (M) and the mean difference (SD) for feature extraction to generate a feature set.

[0087] S103: Based on the feature set, use IBM SPSS23.0 for data analysis and evaluation to generate the user interest clustering results.

[0088] S104: Based on the user interest clustering results, generate the setting of the dual and multi-user interactive opera cultural experience mode and the setting of cultural content.

[0089] S2: Analyze the corresponding characters, several events centered around the character, and the location and time of the events according to the classification results of the humanistic themes that the user is interested in.

[0090] After obtaining the humanistic themes that the user is interested in, it is necessary to determine the display content related to the theme. Taking Wang Yangming as an example, through the collection and collation of relevant materials, four themes, namely "Discussing the Tao in the Academy", "Rebuking Liu Jin to His Face", "Being Flogged and Exiled", and "Enlightening in Longchang", are obtained as the display content for each display board. And the specific events included in each theme's corresponding time period, as well as the time and location of the events, are analyzed.

[0091] S3: Segment several events according to the occurrence time to form several time periods, and design the corresponding character models and backgrounds for each time period.

[0092] Specifically, referring to Figure 2b This step specifically includes:

[0093] S301, after performing text analysis on a user-selected topic (e.g., the Yue Opera "Wang Yangming") and its real historical background based on a machine learning algorithm, obtain the main emotional tone and specific data sets of the task in different time periods and event contexts;

[0094] S302: Based on the data set, classify it according to time and events to obtain the development tone and keyword images of Yangming's philosophy of mind and the protagonist Yangming's personal scale;

[0095] S303: Visually process the event based on the data set, determine the main color tone of each part of the scene based on the sentiment analysis result of step S301, and use Midjourney to generate a display screen; use Midjourney to draw a stylized sketch of the scene;

[0096] S304. Based on the stylized sketch of the scene, use software such as PS to perform secondary optimization design on the screen;

[0097] S305. Using Adobe Illustrator, vectorize the text design based on Mr. Yangming's quotations and calligraphy works, and thereby perform typesetting design on the screen described in S304 and add four-character name seals corresponding to the scenes, thereby obtaining four scene design screens.

[0098] During the calligraphy exhibition, a deep learning-based generative model for Wang Yangming's calligraphy was designed. This model is a single-word generation model that uses a conditional GAN to take text as input and a calligraphy image as input. The generated text image is then typeset to form the final calligraphy work.

[0099] S4 prepares a display board according to the character modeling and background of each time period, and fixes the display board to the display surface of the physical display device.

[0100] After designing the character model and background, you only need to draw them on PP paper and light sheets respectively. The drawn PP paper and light sheets are installed on the display surface of the physical display device and finally sealed with acrylic sheets to prevent damage.

[0101] After the physical display device S5 is prepared, interaction can be carried out through the augmented reality system deployed on the mobile terminal. In this embodiment, the augmented reality system includes three modes: Mode I appreciation mode, Mode II interaction mode, and Mode III creative mode. The three modes are introduced below.

[0102] In Mode I, in response to the user scanning the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component 3D model is superimposed on the physical display device image in the display scene for display.

[0103] More specifically, as Figure 3 shown, the virtual component 3D model includes an event board in the display scene; the event board is generated according to the events occurring in the corresponding time period of the display board, and each event board represents an event that a person has occurred in the corresponding time period of the display board; the event board can move along with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event board is located at the place where the event occurred on the map, and the distance between the event board and the map plane is determined according to the event occurrence time.

[0104] The creation process of the event board is as follows:

[0105] S501. Establish the three-dimensional coordinates of the display board in the display scene based on the position of the display board in the physical display device, where the length and width of the display board correspond to the x and y axes, and the thickness direction of the display board corresponds to the z axis.

[0106] Map the map of the place where the event occurred onto the display board, and the longitude and latitude of the map correspond to the x-axis direction and y-direction of the display board. The map can be used as a hidden layer and is not displayed in the scene.

[0107] S502. According to the events occurring in the corresponding time period on each display board, use Adobe illustrator to draw the corresponding event board and model it through Rhino; preferably, give the event board a glazed material.

[0108] S504. The place where the event corresponding to the event board occurred (longitude and latitude) is used as the x coordinate and y coordinate of the event board on the display board, and the event occurrence time is used as the z coordinate.

[0109] The geographical location (longitude and latitude) where each event occurred is used as the x coordinate and y coordinate of the event board on the display board. The time when the event occurred is the z coordinate of the event board on the display board, that is, the closer the event is to the present, the closer the event board is to the central axis of the physical display device, and the farther the event is from the present, the farther the event board is from the central axis of the physical display device, and vice versa. Thus, the place and time of the event are displayed through the three-dimensional spatial position of the event board.

[0110] In addition, in this mode, by rotating the physical display device or / and adjusting the scanning angle of the mobile terminal, the scanned display board is switched, so as to display different display boards that are scanned, and according to the image data of the scanned display board, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display.

[0111] For example, after selecting this mode in the system, user A scans the physical display device through the mobile terminal, obtains the image feature point information of the display board marker in the camera lens, and matches it with the information on the server side. At this time, several event cards will be generated on the display board in the camera lens, and the three-dimensional spatial positions of the event cards are determined according to the event occurrence location and time. Therefore, even if events occur in the same place, as long as the occurrence times are different, the event cards will not overlap.

[0112] In addition, since the positions of the event cards are determined according to each display board, their positions are relatively fixed with respect to the display board. Thus, when the camera rotates or the physical display device rotates, the time cards will move with the display board and will not be disordered.

[0113] When user A adjusts the scanning angle or user B rotates the physical display device at this time, the display board will be switched in the camera lens, and the content in the display scene will also change with the change of the display board. In this embodiment, the physical display device is a square column and includes four display surfaces, so there are also four display boards. When the scanning angle is adjusted or the physical display device rotates, different virtual component three-dimensional models including event cards will be switched and displayed.

[0114] Of course, the virtual component three-dimensional models in this mode can include other items such as virtual plaques, sculptures, etc. for beautifying the scene in addition to the event cards.

[0115] In this mode, in response to the user's operation, an opera video can also be played. Each display board has its own corresponding opera video clip, which can be played when the user clicks the play button.

[0116] In mode II, in response to the user's scanning of the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; according to the image data of the display board scanned by the user, a virtual character three-dimensional model of the character in the corresponding time period is displayed; and in response to the user's operation, a scene is set up using the virtual object three-dimensional model.

[0117] In this mode, during the time period corresponding to each display board, the core figure Wang Yangming will have a different virtual character appearance. By adjusting the scanning angle or rotating the physical display device, when different display boards come into the camera view, the 3D models of the virtual Wang Yangming with corresponding different appearances will be shown. The performance actions of the virtual characters can be professional opera character animations preset based on motion capture technology. These preset actions are collected from professional opera actors through motion capture devices during the program development process. Through motion capture technology, the virtual characters can perform the preset actions more realistically and smoothly.

[0118] In addition, in this mode, the scenes where each virtual character appearance is located can be decorated. For example, 3D models of mountains, trees, etc. can be added. It can be understood that in response to the user's operations, corresponding addition, deletion, dragging, rotation, and scaling operations are performed on the 3D models of virtual objects. For example, the user can drag the 3D model of a tree to any position in the scene, and can also change its angle by rotation and its size by scaling. Specifically, when the user clicks the scenery setting button, the virtual space can be set on the basis of the physical product and the virtual scene by selecting and dragging the decorative objects below; the position, zoom in / out, and rotation of the decorative objects can be adjusted through gesture control; clicking the previous interface button can step back the user's scenery setting behavior; clicking the scene restoration button can clear all scene settings and return to the original state; before taking a photo for memory, the user can click the backpack button to hide the selection bar of the decorative objects; after clicking the photo-taking button for memory, the system will record the current scene state and automatically enter the state of photo-taking review. The user can independently click the corresponding forwarding button to select to forward it to other social media or the local photo album of the mobile phone; clicking the return button can return to the main interface of the interactive mode.

[0119] This mode also includes a singing mode. In response to the user's operation, after entering the singing mode, the singing track corresponds to the event of the display board in the currently displayed scene; in the singing mode, different users are responsible for singing and scene shooting respectively. Specifically, after the user clicks the singing button, it automatically enters the Yue opera singing mode and the system starts automatic recording; through the performance of different roles, user A records the scene and sound, and user B is responsible for singing, and the two cooperate to complete an opera culture work; the user can click the lyrics button to control the switch of the lyrics below; the user clicks the enable chorus button to switch between chorus and accompaniment; after the user clicks the end performance button, the system will automatically enter the state of recording review. The user can independently click the corresponding forwarding button to select to forward it to other social media or the local photo album of the mobile phone; clicking the return button can return to the main interface of the interactive mode.

[0120] In Mode III, in response to the user's scanning of the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture is editable.

[0121] In this mode, different virtual character 3D models will also be displayed according to different scanned display boards.

[0122] Specifically, when the user clicks the photo button, the viewing angle can be adjusted and the photo shutter can be clicked to take a photo record of the current scene; the photos will all be saved in the gallery; clicking the return button can return to the main interface of the creative mode; when the user clicks the postcard button, postcards can be made; clicking the + button in the middle of the screen can enter the gallery to filter pictures, and text editing and stamp selection will be performed on the selected pictures; after clicking the complete button, the edited postcard can be forwarded to other social media or the mobile phone local by independently clicking the corresponding forwarding button; the user clicks the gallery button to enter and view the photos, and then selects to edit the postcard; clicking the return button can return to the main interface of the creative mode.

[0123] The steps of the virtual 3D model in the present invention are as follows:

[0124] S511. Select a framework through Vuforia to generate an AR (Augmented Reality) framework selection result;

[0125] S512. Based on the result of the AR framework selection, use Vuforia to perform image tracking and virtual object rendering on the black-and-white line drawing, color drawing, and overall 3D physical object respectively to generate a basic AR function;

[0126] Vuforia performs image recognition based on identifying image feature points. Therefore, if there are two completely identical pictures, one in black-and-white and the other in color, the recognition results will be the same. In the design of the present invention, the black-and-white line drawing and the color drawing are not completely consistent. The color drawing has a large number of new feature points added on the basis of the feature points of the black-and-white line drawing, so it will be recognized as another picture. Therefore, different augmented reality contents can be stored separately.

[0127] S513. Based on the basic AR function set, use baking and real-time GI technologies in unity to enhance the rendering effect and generate an AR model with enhanced visual effects;

[0128] S514. By means of depth camera technology, image data is acquired to improve the accuracy of AR in identifying physical objects. The present invention provides a virtual object registration method that combines depth data collected by a ToF camera. This method uses the depth data on a device that provides depth data to improve the detection accuracy of the posture of the physical display device and the matching accuracy of virtual objects during the movement (rotation) of physical objects.

[0129] The specific steps for generating an augmented reality dynamic scene associated with the physical display device in the present invention are as follows:

[0130] S521. Based on AR technology, identify physical objects (2D images and 3D wholes), and create a License Key in the certificate manager;

[0131] S522. Add a marker object in the target manager; perform enhancement processing on the added marker object target to obtain the enhanced marker object;

[0132] S523. Add the Unitypackage package of the marker object in the target manager. The Unitypackage package is a development package for Unity3D development;

[0133] S524. Paste the License Key created in the certificate manager into the QCARBehaviour script of the ARCamera in Unity3D;

[0134] S525. Import the Unitypackage package and the mobile AR development package into unity3D together to generate a local database. The mobile AR development package includes a Vuforia extension package, a model package, and a special effect package.

[0135] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0136] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a computer terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0137] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the purpose of the present invention and the scope protected by the claims. All of these are within the protection scope of the present invention.

Claims

1. An augmented reality-based cultural interaction device, characterized in that Comprising: A physical display device and an augmented reality system deployed in a mobile terminal; The physical display device includes a light painting module that can rotate on a turntable; The light painting module is surrounded by several display surfaces, and display boards are arranged on the display surfaces; the display boards include a character foreground board and a background board; a light source is arranged in the cavity surrounded by the display surfaces, so that only the pattern on the character foreground board is visible when the light source is turned off, and the patterns on both the character foreground board and the background board are visible when the light source is turned on; the patterns on the character foreground board and the background board are formed by segmenting several events according to the occurring events to form several time periods in advance, and the character shapes and backgrounds are designed according to each time period; The augmented reality system includes three modes; In Mode I, in response to the user scanning the display board through the mobile terminal, a real scene including the physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the image of the physical display device in the real scene for display. The virtual component three-dimensional model includes event cards generated according to the events occurring in the corresponding time period of the display board. Each event card represents an event that the character undergoes in the time period corresponding to the display board. The real scene and the virtual component three-dimensional model constitute a display scene that combines reality and virtuality; the event cards can move along with the corresponding display surface on the light painting module in the real scene in the display scene; a map where the event occurs is mapped on the display surface of the light painting module in the real scene in the display scene, and the longitude and latitude of the map correspond to the x-axis direction and y-axis direction of the display board; the event cards are located at the occurrence location of the event on the map, and the distance between the event cards and the map plane is determined according to the event occurrence time; specifically, the longitude and latitude corresponding to the event occurrence location of the event card are used as the x coordinate and y coordinate of the event card on the display board, and the event occurrence time is used as the z coordinate; and when the physical display device is rotated or / and the scanning angle of the mobile terminal is adjusted to switch the scanned display board, different scanned display boards are displayed, and according to the scanned display board image data, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display; In Mode II, in response to the user scanning the display board through the mobile terminal, a real scene including the physical display device is displayed on the user's mobile terminal; according to the display board image data scanned by the user, a virtual character three-dimensional model of the character in the corresponding time period is displayed, so as to obtain a display scene that combines reality and virtuality; And in response to the user's operation, a virtual object three-dimensional model is used to set the scene in the display scene; In Mode III, in response to the user scanning the display board through the mobile terminal, a display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, the current display scene is captured and saved; the captured picture is editable.

2. The cultural interaction device based on augmented reality according to claim 1, characterized in that: The foreground board of the figure is made of PP paper, and the background board is made of light sheet; the display board further includes a transparent acrylic board laid in front of the foreground board of the figure.

3. The cultural interaction device based on augmented reality according to claim 1, characterized in that: The light painting module is in the shape of a square column, and the side wall of the square column light painting module is the display surface; a hollow is opened on the display surface, and the display board is embedded in the hollow.

4. An augmented reality-based cultural interaction method, characterized in that, Including: Obtain the classification result of the humanistic theme that the user is interested in; Parse out the corresponding figure, a number of events centered on the figure, and the time when the events occur according to the classification result of the humanistic theme that the user is interested in; Segment a number of events according to the occurrence time to form a number of time periods, and design corresponding figure shapes and backgrounds for each time period; Prepare a display board according to the figure shape and background of each time period, and fix the display board to the display surface of the physical display device; In Mode I, in response to the user scanning the display board through the mobile terminal, display the real scene including the physical display device on the user's mobile terminal; and according to the display board image data scanned by the user, superimpose the virtual component three-dimensional model on the image of the physical display device in the real scene for display. The virtual component three-dimensional model includes an event sign generated according to the events occurring in the corresponding time period of the display board. Each event sign represents an event that the figure experiences in the time period corresponding to the display board. The real scene and the virtual component three-dimensional model constitute a display scene that combines reality and virtuality; the event sign can move along with the corresponding display surface in the display scene; a map where the event occurs is mapped on the display surface in the display scene, and the longitude and latitude of the map correspond to the x-axis direction and y-axis direction of the display board; the event sign is located at the place where the event occurs on the map, and the distance between the event sign and the map plane is determined according to the event occurrence time; specifically, the longitude and latitude corresponding to the event occurrence place of the event sign are used as the x coordinate and y coordinate of the event sign on the display board, and the event occurrence time is used as the z coordinate; and when the physical display device is rotated or / and the scanning angle of the mobile terminal is adjusted to switch the scanned display board, so as to display different scanned display boards, and according to the scanned display board image data, superimpose the virtual component three-dimensional model on the image of the physical display device in the display scene for display; In Mode II, in response to the user scanning the display board through the mobile terminal, display the real scene including the physical display device on the user's mobile terminal; according to the display board image data scanned by the user, display the virtual figure three-dimensional model of the figure in the corresponding time period, so as to obtain a display scene that combines reality and virtuality; And in response to the user's operation, use the virtual object three-dimensional model to set the scene in the display scene; In Mode III, in response to the user scanning the display board through the mobile terminal, display the display scene including the physical display device on the user's mobile terminal; in response to the user's operation, take a screenshot of the current display scene and save it; the captured picture is editable; Among them, the creation of the event sign specifically includes: S501. Establish the three-dimensional coordinates of the display board in the physical display device in the display scene, where the length and width of the display board correspond to the x and y axes, and the thickness direction of the display board corresponds to the z axis; S502. According to the events that occurred during the corresponding time period on each display board, use Adobe illustrator to draw the corresponding event cards and model them through Rhino; S504. The occurrence location of the event corresponding to the event card serves as the x coordinate and y coordinate of the event card on the corresponding display board, and the event occurrence time serves as the z coordinate.

5. The cultural interaction method based on augmented reality according to claim 4, characterized in that: In mode I, in response to the user's operation, play an opera video; or, in mode II, in response to the user's operation, perform corresponding addition, deletion, dragging, rotation, and scaling operations on the three-dimensional model of the virtual object.

6. The cultural interaction method based on augmented reality according to claim 4, characterized in that: In mode II, in response to the user's operation, display the text introduction of the event corresponding to the display board in the current display scene.

7. An augmented reality-based cultural interaction method according to claim 4, characterized in that: In mode II, in response to the user's operation, enter the singing mode; the singing track corresponds to the event of the display board in the current display scene; in the singing mode, different users are responsible for singing and scene shooting respectively.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any one of the augmented reality-based cultural interaction methods described in claims 4 to 7.

Images