Augmented reality-based culture interaction method and device, and storage medium

The AR-based cultural interaction method and device enhance user engagement and understanding of cultural performances by using a physical display device and mobile terminal to overlay virtual content, addressing the limitations of traditional viewing and promoting immersive experiences.

CN119068150AActive Publication Date: 2024-12-03ZHEJIANG WANLI UNIV
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411159364.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-02-05
Filing Date
2024-08-21
Publication Date
2024-12-03
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

Existing cultural performances, such as Chinese opera, are limited by time and space, and passive viewing modes fail to engage the general public effectively, necessitating an immersive and interactive way to present cultural experiences.

Method used

A cultural interaction method and device using augmented reality (AR) with a physical display device and mobile terminal, featuring a light painting module and augmented reality system, allowing users to scan display boards for virtual content overlay, character models, and interactive scene creation.

Benefits of technology

Enhances user participation and understanding of cultural content through multi-sensory interactions, reducing equipment requirements and enabling participation without spatial constraints, and providing a more immersive and interactive experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068150A_ABST
    Figure CN119068150A_ABST
Patent Text Reader

Abstract

The invention discloses a culture interaction method and device based on augmented reality and a storage medium. The device comprises an entity display device and an augmented reality system deployed in a mobile terminal. The entity display device comprises a light picture module, and the light picture module can rotate on the rotary table; the lamplight picture module is defined by a plurality of display faces, and display boards are arranged on the display faces. The display board comprises a character foreground board and a background board; a light source is arranged in a cavity defined by the display face. The augmented reality system comprises an appreciation mode, an interaction mode and a creative mode. The role in the method is not only defined as an audience, but also is a performance participant, the preset action data can improve the performance effect of the virtual character controlled by the user, the real-time action editing supported by the multi-point touch control technology can enable the user to modify the generated action content on the basis of the preset action, and the user experience is improved. And the culture participation degree is increased.
Need to check novelty before this filing date? Find Prior Art

Description

Priority application

[0001] This application claims priority to the Chinese invention patent application [Application Number: 202410166119X] filed on February 5, 2024, "[Invention Name: A cultural interaction method, device and storage medium based on augmented reality]", which is incorporated by reference in its entirety. Technical Field

[0002] The present invention belongs to the field of augmented reality technology, and in particular relates to an augmented reality-based cultural interaction method, device and computer-readable storage medium. Background Art

[0003] Chinese opera culture is world-renowned for its unique singing and storytelling style. Its performances go deep into the narrative and character relationships. It is a bridge to convey China's rich cultural thoughts and historical depth, and reproduces the historical events and development trajectory of ancient Chinese cultural thoughts.

[0004] For decades, Chinese institutions have been committed to promoting cultural heritage. Digital resources such as text, images, and videos have been used as a valuable medium to preserve local opera performances and their hidden cultural ideas, transforming abstract theories into art and bringing historical and cultural ideas to life. However, this passive viewing mode may not be effective in attracting the general public. Therefore, it is necessary to explore a new communication model to present cultural performances in a more immersive way. An emerging trend is to use augmented reality (AR) to share experiences in cultural narratives because it has advantages in displaying scene creation, storytelling, and active exploration, and is combined with tangible interfaces, situational visualization, game mechanisms, etc. to promote multi-sensory embodied interactions.

[0005] On the other hand, cultural performances are often limited by time and space. Therefore, we want to design a hybrid cultural and creative product as a carrier in a shared cultural experience, apply it to users' daily life display scenarios, and encourage users to actively participate in cultural interactive experiences.

[0006] Prior art CN109692487A discloses an augmented reality operation method for Huangmei Opera. The technical solution recorded in the document is mainly used in real theaters and relies on the installation of cameras, sensors, computing and storage devices, display devices and other equipment on site. Motion capture is also based on the real-time motion data of the actors.

[0007] Prior art CN102688603A discloses a system and method for real-time magic stage performances based on augmented reality and motion recognition technology. The technical solution recorded therein is mainly used in real theaters and relies on the on-site installation of equipment such as cameras, projectors, holographic imaging and other display devices. In stage performances, the performances of some real actors are superimposed on the interaction of enhanced information images.

[0008] Prior art CN109725706A discloses a role interaction system based on augmented reality, and the technical solution recorded therein only involves presenting augmented reality content of multiple recognition images through non-functional devices of shapes made of different physical materials. Summary of the invention

[0009] The purpose of the present invention is to provide a cultural interaction method and an interaction device based on augmented reality, so as to improve the efficiency of cultural interaction through augmented reality technology.

[0010] In order to solve the above-mentioned technical problems, the present invention specifically adopts the following technical solutions: a cultural interaction device based on augmented reality, comprising: a physical display device and an augmented reality system deployed in a mobile terminal;

[0011] The physical display device includes a light painting module, and the light painting module can rotate on a turntable;

[0012] The light painting module is surrounded by a plurality of display surfaces, on which display boards are arranged; the display boards include a character foreground board and a background board; a light source is arranged in the cavity surrounded by the display surfaces, so that when the light source is turned off, only the pattern on the character foreground board is visible, and when the light source is turned on, the patterns on both the character foreground board and the background board are visible;

[0013] The augmented reality system includes three modes;

[0014] In mode I, in response to a user scanning a display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display, and the virtual component three-dimensional model includes an event card in the display scene; the event card is generated according to the events occurring in the time period corresponding to the display board, and each event card represents an event occurring to a person in the time period corresponding to the display board; the event card can move with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event card is located at the place where the event occurs on the map, and the distance between the event card and the map plane is determined according to the time when the event occurs;

[0015] In mode II, in response to a user scanning the display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; a virtual character three-dimensional model of a character in a corresponding time period is displayed according to the display board image data scanned by the user; and in response to the user's operation, the virtual object three-dimensional model is used for setting the scene;

[0016] In mode III, in response to the user scanning the display board through the mobile terminal, the display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture can be edited.

[0017] As an improvement, the character foreground board is made of PP paper, and the background board is made of light sheet; the display board also includes a transparent acrylic board laid in front of the character foreground board.

[0018] As an improvement, the light painting module is in the shape of a square column, and the side wall of the square column light painting module is a display surface; the display surface is hollowed out, and the display board is embedded in the hollow.

[0019] The present invention also provides a cultural interaction method based on augmented reality, comprising:

[0020] Get the classification results of humanities topics that the user is interested in;

[0021] According to the classification results of the humanities topics that the user is interested in, the corresponding person, several events centered on the person, and the time when the events occurred are analyzed;

[0022] Divide several events into several time periods according to the time of occurrence, and design corresponding character models and backgrounds for each time period;

[0023] Prepare display boards according to the character modeling and background of each time period, and fix the display boards to the display surface of the physical display device;

[0024] In mode I, in response to a user scanning a display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display, and the virtual component three-dimensional model includes an event card in the display scene; the event card is generated according to the events occurring in the time period corresponding to the display board, and each event card represents an event occurring to a person in the time period corresponding to the display board; the event card can move with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event card is located at the place where the event occurs on the map, and the distance between the event card and the map plane is determined according to the time when the event occurs;

[0025] In mode II, in response to a user scanning the display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; a virtual character three-dimensional model of a character in a corresponding time period is displayed according to the display board image data scanned by the user; and in response to the user's operation, the virtual object three-dimensional model is used for setting the scene;

[0026] In mode III, in response to the user scanning the display board through the mobile terminal, the display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture can be edited.

[0027] As an improvement, in mode I, the virtual component three-dimensional model includes event boards in the display scene; the event boards are generated based on events occurring in a time period corresponding to the display board, and each event board represents an event occurring to a character in a time period corresponding to the display board; the event boards can move with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event board is located at the location on the map where the event occurs, and the distance between the event board and the map plane is determined based on the time of occurrence of the event.

[0028] As an improvement, the creation of the event card specifically includes:

[0029] S501, establishing the three-dimensional coordinates of the display board in the display scene based on the position of the display board in the physical display device, wherein the length and width of the display board correspond to the x-axis and the y-axis, and the thickness direction of the display board corresponds to the z-axis;

[0030] S502, according to the events occurring in the corresponding time period on each display board, draw the corresponding event board using Adobe illustrator and model it using Rhino;

[0031] S504: The place where the event occurs corresponding to the event card is used as the x-coordinate and y-coordinate of the event card on the corresponding display board, and the time when the event occurs is used as the z-coordinate.

[0032] As an improvement, in mode I, the scanned display board is switched by rotating the physical display device and / or adjusting the scanning angle of the mobile terminal, thereby displaying different scanned display boards, and based on the scanned display board image data, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display.

[0033] As an improvement, in mode I, an opera video is played in response to a user's operation.

[0034] As an improvement, in mode II, in response to the user's operation, corresponding adding, deleting, dragging, rotating, and scaling operations are performed on the virtual object three-dimensional model.

[0035] As an improvement, in mode II, in response to the user's operation, a text introduction of the corresponding event on the display board in the current display scene is displayed.

[0036] As an improvement, in mode II, in response to the user's operation, the singing mode is entered; the songs sung correspond to the events on the display board in the current display scene; in the singing mode, different users are responsible for singing and scene filming respectively.

[0037] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned cultural interaction method based on augmented reality is implemented.

[0038] The present invention is beneficial in that:

[0039] The physical display device in the present invention is based on a light source, such as an LED and a special paper material, so that the picture at the same position can present different recognition images in different states (for example, the light source is turned on and the light source is turned off). The static images under the two lighting states in the present invention have picture correlation, which is coherent for the metaphor and narrative of cultural interaction, and is suitable for multi-level cultural interaction content presentation. In addition, the picture on the scanning display board under the light source generates a corresponding event card (the event card represents the place and time of the event), so that the user can have an in-depth understanding of the corresponding file, and can also switch different display boards by rotating the physical display device, so as to obtain the place and time of different events, and then have an in-depth understanding of the file. At the same time, this interactive method also greatly improves the user's participation and improves the user experience. The present invention not only involves 2D image recognition, but also involves 3D image recognition of the entire device. At the same time, the lighting function design of the device gives the present invention a certain practicality in life.

[0040] The cultural interaction method in the present invention uses physical objects as carriers for image positioning, and motion capture is also based on the professional actor movements captured in advance during development, which greatly reduces the requirements for equipment. As long as mobile devices such as mobile phones and tablets are used, the experience can be participated in, without being restricted by time and space. And as an amateur cultural experiencer, the role in the present invention is not only defined as an audience, but also a participant in the performance. The preset action data can enhance the performance effect of the user controlling the virtual character, and the real-time action editing supported by multi-touch technology allows users to modify the generated action content based on the preset action, thereby increasing cultural participation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without paying creative labor.

[0042] Figure 1a It is a structural schematic diagram of the physical display device in the present invention;

[0043] Figure 1b for Figure 1a An exploded view of the physical display device shown;

[0044] Figure 2a A flowchart of a cultural interaction method based on augmented reality according to an embodiment of the present invention;

[0045] Figure 2b A flow chart of an embodiment for segmenting a plurality of events;

[0046] Figure 3 This is a schematic diagram of the process of generating event cards.

[0047] Markings in the figure: 1 acrylic plate, 2 character foreground plate, 3 background plate, 4 acrylic plate, 5 frame, 6 top shape, 7 connection shape, 8 base, 9 light source, 10 USB charging port, 11 touch button, 12 bearing, 13 bolt, 14 anti-slip pad. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] Herein, suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings by themselves. Therefore, "module", "component" or "unit" can be used mixedly.

[0050] In this document, the terms "upper", "lower", "inner", "outer", "front", "back", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0051] In this document, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0052] Herein "and / or" includes any and all combinations of one or more of the associated listed items.

[0053] Herein, "plurality" means two or more than two, ie, it includes two, three, four, five, etc.

[0054] Embodiment 1

[0055] like Figure 1a and Figure 1b As shown, the present invention provides a cultural interaction device (or interaction system) based on augmented reality, including: a physical display device and an augmented reality system deployed in a mobile terminal.

[0056] Through the cooperation between the physical display device and the augmented reality system, a scene combining reality and virtuality is displayed on the display screen of mobile terminals including smart phones, tablet computers, etc. In simple terms, the camera on the mobile terminal is used to capture the real scene including the physical display device, and the augmented reality system adds virtual content to the real scene and merges the virtual content with the real scene, thereby creating a display scene combining reality and virtuality on the screen of the mobile terminal.

[0057] The physical display device is a physical device existing in reality, and comprises a light painting module, and the light painting module can be rotated on a turntable.

[0058] The light painting module is surrounded by a plurality of display surfaces, on which display boards are arranged; the display boards include a character foreground board 2 and a background board 3; a light source 9 is arranged in the cavity surrounded by the display surfaces, so that when the light source 9 is turned off, only the pattern on the character foreground board 2 is visible, and when the light source is turned on, the patterns on both the character foreground board 2 and the background board 3 are visible. When the user turns the turntable, the character foreground board 2 and the background board 3 representing different scenes can be presented to the user in the user interface displayed by the mobile terminal. Preferably, a character is drawn on the character foreground board 2. For example, Wang Yangming. Various scenes of different events are drawn on the background board 3.

[0059] Specifically, the character foreground board 2 is made of PP paper, and the background board 3 is made of light sheet; the display board also includes a transparent acrylic board 1 laid in front of the character foreground board 2. In order to improve the viewing experience, the pattern on the character foreground board 2, i.e., the PP paper, can be a black and white line drawing, and the pattern on the background board 3, i.e., the light sheet, can be a color drawing.

[0060] The light source 9 in this embodiment is an LED light source. For convenient control, a touch switch 11 is also provided. In addition, in order to achieve the purpose of portability, a rechargeable battery is also provided, and of course, a USB charging port 10 for charging the battery is also provided.

[0061] The specific shape of the light painting module can be varied. In this embodiment, in order to conform to traditional culture, the light painting module is set to the shape of an antique building, that is, the light painting module is a square column, and its main body is a wooden frame 5; the side wall of the square column light painting module is a display surface; the display surface is hollowed out, and the display board is embedded in the hollow. In addition, a decorative modeling structure, such as a pavilion, is provided on the top of the light painting module. An acrylic plate 1 is also provided between the modeling structure and the light painting module. More specifically, the modeling structure includes a top modeling 6 and a connecting part modeling 7.

[0062] In addition, the light painting module can rotate on a turntable, which specifically includes a base 8 , which is connected to the light painting module by means of a bearing 12 and a bolt 13 , and an anti-slip pad 14 is also provided at the bottom of the base 8 .

[0063] The augmented reality system is software deployed on a mobile terminal, which is used to provide corresponding virtual content according to the content of the display board in the physical display device. This is a prior art and will not be described here. However, the difference is that the augmented reality system in this embodiment includes three modes;

[0064] In mode I, in response to a user scanning a display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display; wherein the virtual component three-dimensional model includes an event card in the display scene; the event card is generated according to the events occurring in the time period corresponding to the display board, and each event card represents an event occurring to a person in the time period corresponding to the display board; the event card can move with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event card is located at the place where the event occurs on the map, and the distance between the event card and the map plane is determined according to the time when the event occurs;

[0065] In mode II, in response to a user scanning the display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; a virtual character three-dimensional model of a character in a corresponding time period is displayed according to the display board image data scanned by the user; and in response to the user's operation, the virtual object three-dimensional model is used for setting the scene;

[0066] In mode III, in response to the user scanning the display board through the mobile terminal, the display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture can be edited.

[0067] Preferably, the augmented reality system specifically includes:

[0068] The first response module is used to display a display scene including the physical display device on the user interface of the user's mobile terminal in response to the user scanning the display board through the mobile terminal in mode I; and to overlay a virtual component three-dimensional model on the image of the physical display device in the display scene for display according to the display board image data scanned by the user;

[0069] The second response module is used to display a display scene including a physical display device on the user's mobile terminal in response to the user scanning the display board through the mobile terminal in mode II; display a virtual character three-dimensional model of the character in the corresponding time period according to the display board image data scanned by the user; and use the virtual object three-dimensional model to set the scene in response to the user's operation;

[0070] The third response module is used to display the display scene including the physical display device on the user's mobile terminal in response to the user scanning the display board through the mobile terminal in mode III; in response to the user's operation, take a screenshot of the current display scene and save it; the captured picture can be edited.

[0071] Furthermore, the cultural interaction device also includes:

[0072] The data acquisition module is used to obtain the classification results of humanities topics that the user is interested in;

[0073] The event data analysis module is used to analyze the corresponding person, several events centered on the person, and the place and time of the event according to the classification results of the humanities topics that the user is interested in;

[0074] The event segmentation module is used to segment several events into several time periods according to the time of occurrence, and design corresponding character modeling and background for each time period; wherein the background includes the location or scene information of at least one event.

[0075] After designing the character modeling and background, you only need to draw them on PP paper and light sheets respectively. The drawn PP paper and light sheets are installed on the display surface of the physical display device, and finally encapsulated with acrylic sheets to avoid damage.

[0076] Embodiment 2

[0077] like Figure 2a As shown, the present invention also provides a method for cultural interaction based on augmented reality, which is applied to the device for cultural interaction based on augmented reality in the first embodiment, and specifically includes:

[0078] S1 obtains the classification results of humanities topics that the user is interested in.

[0079] Humanities topics usually refer to topics related to human culture, history, art, society, etc., such as literature, music, philosophy, historical figures, major events, and social customs. These topics reflect human thoughts, emotions, and values. In the above text, the humanities topic classification results may include specific historical periods, cultural phenomena, or artistic schools.

[0080] The main purpose of the present invention is to promote traditional culture, so this embodiment is described around Yue Opera in traditional culture.

[0081] Yue Opera is one of the Chinese operas. It originated in Shengzhou, Zhejiang, and was formed in the late Qing Dynasty and early Republic of China. As China's intangible cultural heritage, Yue Opera carries a rich historical and cultural heritage. It integrates local folk music, southern opera, Huangmei opera and other elements, and has formed a unique performance style and music system in the process of development. The popularity of Yue Opera is inseparable from the contribution of many classic repertoires. Among them, the newly compiled classic repertoire of Yue Opera "Wang Yangming" is not only unique in performance form and music, but also conveys rich inner emotions and ideological connotations by narrating Wang Yangming's life experience of "reason" and "emotion".

[0082] However, it is difficult for traditional stage performances to fully and vividly present the charm of this story. Especially for young audiences, the traditional way of viewing may seem boring and outdated. As an emerging information technology, augmented reality technology provides the public with a new way of cultural experience. It can put the audience in a virtual way, which is more interactive and immersive than traditional stage viewing. Therefore, combining the story of the Yue Opera "Wang Yangming" with augmented reality technology is not only conducive to the protection and inheritance of intangible cultural heritage, but also can better meet the diverse needs of contemporary audiences for cultural experience.

[0083] In this embodiment, based on IBM SPSS23.0 software, the mean and standard deviation algorithm is used to analyze the data of online questionnaire surveys and expert interviews to obtain the clustering results of users' interests in cultural experience. The specific steps include:

[0084] S101, some online questionnaires based on Likert scale, used Microsoft Forms to collect, organize, delete abnormal data, and generate cleaned data sets;

[0085] S102, based on the cleaned data set, using the mean algorithm (M) and the mean difference (SD) to perform feature extraction and generate a feature set;

[0086] S103, based on the feature set, using IBM SPSS23.0 to analyze and evaluate the data and generate user interest clustering results;

[0087] S104. Based on the user interest clustering results, generate settings for two-person and multi-person user interactive opera culture experience modes and cultural content settings.

[0088] S2 analyzes the corresponding person, several events centered on the person, and the place and time of the event based on the classification results of the humanities topics that the user is interested in.

[0089] After obtaining the humanities topics that users are interested in, it is necessary to determine the display content related to the topic. Taking Wang Yangming as an example, through the collection and organization of relevant materials, four topics were obtained as the display content of each display board: "Discussing Taoism in the Academy", "Rebuking Liu Jin", "Exiled with a Cane in the Court", and "Enlightenment in Longchang". And the specific events included in the time period corresponding to each topic, as well as the time and place of the events were analyzed.

[0090] S3 divides several events into several time periods according to the time of occurrence, and designs corresponding character modeling and background for each time period.

[0091] Specifically, see Figure 2b , the step specifically includes:

[0092] S301, after performing text analysis on the selected topic (e.g., the Yue Opera "Wang Yangming") and the real historical background based on the machine learning algorithm, the main emotional tone and specific data set of the task in different time periods and different event backgrounds are obtained;

[0093] S302, based on the data set, classify it according to time and event, and obtain the development tone and keyword images of Yangming's philosophy of mind and the protagonist Mr. Yangming's personal scale;

[0094] S303, visually processing the event based on the data set, determining the main color tone of each part of the scene based on the sentiment analysis result of step S301, and using Midjourney to generate a display screen; using Midjourney to draw a stylized sketch of the scene;

[0095] S304. Based on the stylized sketch of the scene, use software such as PS to perform secondary optimization design on the screen;

[0096] S305. Using Adobe Illustrator, vectorize the text design based on Mr. Yangming's quotations and calligraphy works, and thereby perform typeset design on the screen described in S304 and add four-character name seals corresponding to the scenes, thereby obtaining four scene design screens.

[0097] In the process of calligraphy display, a Wang Yangming calligraphy generation model based on deep learning was designed. This model is a single-word generation model. Through conditional GAN, the text is used as the model input and the calligraphy text image is used as the input. The generated text image is typed and becomes the final calligraphy work.

[0098] S4 prepares a display board according to the character modeling and background of each time period, and fixes the display board to the display surface of the physical display device.

[0099] After designing the character modeling and background, you only need to draw them on PP paper and light sheets respectively. The drawn PP paper and light sheets are installed on the display surface of the physical display device, and finally encapsulated with acrylic sheets to avoid damage.

[0100] After the physical display device S5 is prepared, the augmented reality system deployed on the mobile terminal can be used for interaction. In this embodiment, the augmented reality system includes three modes: Mode I appreciation mode, Mode II interaction mode, and Mode III creative mode. The three modes are introduced below.

[0101] In mode I, in response to the user scanning the display board through the mobile terminal, the display scene including the physical display device is displayed on the user's mobile terminal; and based on the display board image data scanned by the user, the virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display.

[0102] More specifically, Figure 3 As shown, the virtual component three-dimensional model includes an event board in the display scene; the event board is generated according to the events occurring in the time period corresponding to the display board, and each event board represents an event occurring to a character in the time period corresponding to the display board; the event board can move with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event board is located at the place where the event occurs on the map, and the distance between the event board and the map plane is determined according to the time when the event occurs.

[0103] The process of creating an event card is as follows:

[0104] S501. Establishing the three-dimensional coordinates of a display board in a display scene based on the position of the display board in the physical display device, wherein the length and width of the display board correspond to the x-axis and the y-axis, and the thickness direction of the display board corresponds to the z-axis.

[0105] Map the location of the event onto the display board, with the longitude and latitude of the map corresponding to the x-axis and y-axis of the display board. The map can be used as a hidden layer and is not displayed in the scene.

[0106] S502, according to the events occurring in the corresponding time period on each display board, draw the corresponding event board using Adobe illustrator and model it using Rhino; preferably, give the event board a glass material;

[0107] S504. The location (longitude and latitude) of the event corresponding to the event card is used as the x-coordinate and y-coordinate of the event card on the display board, and the time of event occurrence is used as the z-coordinate.

[0108] The geographical location (longitude and latitude) where each event occurred is used as the x-coordinate and y-coordinate of the event card on the display board. The time when the event occurred is the z-coordinate of the event card on the display board, that is, the closer the time is, the closer the event card is to the central axis of the physical display device, and the farther the time is, the farther the event card is from the central axis of the physical display device, and vice versa. Thus, the location and time of the event are displayed through the three-dimensional spatial position of the event card.

[0109] In addition, in this mode, by rotating the physical display device and / or adjusting the scanning angle of the mobile terminal, the scanned display board is switched to display different scanned display boards, and based on the scanned display board image data, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display.

[0110] For example, after selecting this mode in the system, user A scans the physical display device through the mobile terminal, obtains the image feature point information of the display board marker in the lens, and matches it with the server-side information. At this time, several event cards will be generated on the display board in the lens, and the three-dimensional spatial position of the event card is determined according to the location and time of the event. Therefore, even if there are events that occur in the same place, as long as the time of occurrence is different, the event cards will not overlap.

[0111] In addition, since the position of the event card is determined according to each display board, its position relative to the display board is fixed. Therefore, when the camera rotates or the physical display device rotates, the time card will also move with the display board so that there will be no confusion.

[0112] When user A adjusts the scanning angle or user B rotates the physical display device, the display board will switch in the camera, and the content in the displayed scene will also change with the change of the display board. In this embodiment, the physical display device is a square prism, including four display surfaces, and therefore also has four display boards. When the scanning angle is adjusted or the physical display device is rotated, different virtual component three-dimensional models including event cards will be switched.

[0113] Of course, the virtual component three-dimensional model in this mode can include other virtual plaques, sculptures, etc. in addition to event signs to beautify the scene.

[0114] In this mode, in response to the user's operation, the opera video can also be played. Each display board has its own corresponding opera video clip, which can be played when the user clicks the play button.

[0115] In mode II, in response to the user scanning the display board through the mobile terminal, the display scene including the physical display device is displayed on the user's mobile terminal; based on the image data of the display board scanned by the user, the virtual character three-dimensional model of the character in the corresponding time period is displayed; and in response to the user's operation, the virtual object three-dimensional model is used for setting the scene.

[0116] In this mode, the core character Wang Yangming will have a different virtual character shape in the time period corresponding to each display board. By adjusting the scanning angle or rotating the physical display device, when different display boards enter the camera, the corresponding three-dimensional model of the virtual Wang Yangming with different shapes will be displayed. The performance actions of the virtual character can be based on the professional opera character animation preset by motion capture technology. These preset actions are collected from professional opera actors through motion capture equipment during the program development process. Through motion capture technology, the virtual character can perform the preset actions more realistically and smoothly.

[0117] In addition, in this mode, the scene where each virtual character model is located can be arranged, such as adding three-dimensional models of mountains and trees. It can be understood that in response to the user's operation, the virtual object three-dimensional model is added, deleted, dragged, rotated, and scaled accordingly. For example, the user can drag the three-dimensional model of the tree to any position in the scene, and can also change its angle by rotating it, and change its size by scaling it. Specifically, the user clicks the scenery button, and can set the virtual space based on the physical product and the virtual scene by selecting and dragging the decorative objects below; adjust the position, zoom in and out, and rotate the decorative objects through gesture control; click the previous interface button to return one step to the user's scenery behavior; click the scene recovery button to clear all scene arrangements and return to the original state; before taking a photo, you can click the backpack button to hide the selection bar of the decorative object; after clicking the photo button, the system will record the current scene state and automatically enter the photo revisit state, and the user can click the corresponding forwarding button to choose to forward it to other social media or the local album of the mobile phone; click the return button to return to the main interface of the interactive mode.

[0118] This mode also includes a singing mode. In response to the user's operation, after entering the singing mode, the songs sung correspond to the events on the display board in the current display scene; in the singing mode, different users are responsible for singing and scene filming respectively. Specifically, after the user clicks the singing button, the Yue Opera singing mode is automatically entered, and the system starts automatic recording; through the role-playing of different roles, user A records the scene and sound, and user B is responsible for singing, and the two people collaborate to complete a cultural work of opera; the user can click the lyrics button to control the switch of the lyrics below; the user clicks the enable accompaniment button to switch the accompaniment or accompaniment; after the user clicks the end performance button, the system will automatically enter the recording and follow-up state, and the user can click the corresponding forwarding button to choose to forward to other social media or the local album of the mobile phone; click the return button to return to the main interface of the interactive mode.

[0119] In mode III, in response to the user scanning the display board through the mobile terminal, the display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture can be edited.

[0120] In this mode, different virtual character three-dimensional models will also be displayed according to the different display boards scanned.

[0121] Specifically, the user clicks the photo button to adjust the viewing angle and clicks the shutter to take a photo of the current scene; all photos will be saved in the gallery; clicking the back button returns to the main interface of the creative mode; the user clicks the postcard button to make a postcard; clicking the + button in the middle of the screen enters the gallery to filter pictures, and edits the text and selects a stamp on the selected picture; after clicking the Done button, the edited postcard can be forwarded to other social media by clicking the corresponding forward button or the local mobile phone user can click the gallery button to view photos and choose to edit the postcard; clicking the back button returns to the main interface of the creative mode.

[0122] The steps of creating a virtual three-dimensional model in the present invention are as follows:

[0123] S511, select a frame through Vuforia to generate an AR (Augmented Reality) frame selection result;

[0124] S512, based on the result of the AR framework selection, using Vuforia to respectively perform image tracking of the black and white line drawing, the color drawing, and the entire 3D object and rendering of the virtual object to generate basic AR functions;

[0125] Vuforia recognizes images based on identifying image feature points. Therefore, if the images are exactly the same, one is a black and white draft and the other is a color draft, the recognition results will be the same. In the design of the present invention, the black and white line draft and the color draft are not completely consistent. The color draft has a large number of new feature points added to the feature points of the black and white draft, so it will be recognized as another image. Therefore, different augmented reality contents can be stored separately.

[0126] S513, based on the basic AR function set, using baking and real-time GI technology in Unity to enhance the rendering effect and generate an AR model with enhanced visual effects;

[0127] S514. Using depth camera technology, image data is acquired to improve the accuracy of AR recognition of physical objects. The present invention provides a virtual object registration method in combination with depth data collected by a ToF camera. The method uses depth data on a device that provides depth data to improve the detection accuracy of the posture of a physical display device and the matching accuracy of virtual objects when the physical object moves (rotates).

[0128] The specific steps of generating an augmented reality dynamic scene associated with a physical display device in the present invention are:

[0129] S521. Identify the physical object (2D image and 3D whole) based on AR technology and create a License Key in the certificate manager;

[0130] S522, adding a marker object in the target manager; performing enhancement processing on the added marker object target to obtain the enhanced marker object;

[0131] S523, adding the Unitypackage package of the marker object in the target manager, where the Unitypackage package is a development package used for Unity3D development;

[0132] S524. Paste the License Key created in the Certificate Manager into the QCARBehaviour script of ARCamera in Unity3D;

[0133] S525, import the Unitypackage package and the mobile AR development package into unity3D to generate a local database; the mobile AR development package includes the Vuforia extension package, the model package, and the special effect package.

[0134] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a computer terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0136] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A cultural interaction device based on augmented reality, characterized in that include: Physical display devices and augmented reality systems deployed in mobile terminals; The physical display device includes a light painting module, and the light painting module can rotate on a turntable; The light painting module is surrounded by a plurality of display surfaces, on which display boards are arranged; the display boards include a character foreground board and a background board; a light source is arranged in the cavity surrounded by the display surfaces, so that when the light source is turned off, only the pattern on the character foreground board is visible, and when the light source is turned on, the patterns on both the character foreground board and the background board are visible; The augmented reality system includes three modes; In mode I, in response to a user scanning a display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display, and the virtual component three-dimensional model includes an event card in the display scene; the event card is generated according to the events occurring in the time period corresponding to the display board, and each event card represents an event occurring to a person in the time period corresponding to the display board; the event card can move with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event card is located at the place where the event occurs on the map, and the distance between the event card and the map plane is determined according to the time when the event occurs; In mode II, in response to a user scanning the display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; based on the display board image data scanned by the user, a virtual character three-dimensional model of the character in the corresponding time period is displayed; and In response to the user's operation, the three-dimensional model of the virtual object is used to set the scene; In mode III, in response to the user scanning the display board through the mobile terminal, the display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture can be edited.

2. The cultural interaction device based on augmented reality according to claim 1, characterized in that: The character foreground board is made of PP paper, and the background board is made of light sheet; the display board also includes a transparent acrylic board laid in front of the character foreground board.

3. The cultural interaction device based on augmented reality according to claim 1, characterized in that: The light painting module is in the shape of a square column, and the side wall of the square column light painting module is a display surface; the display surface is hollowed out, and the display board is embedded in the hollow.

4. A cultural interaction method based on augmented reality, characterized in that: include: Get the classification results of humanities topics that the user is interested in; According to the classification results of the humanities topics that the user is interested in, the corresponding person, several events centered on the person, and the time when the events occurred are analyzed; Divide several events into several time periods according to the time of occurrence, and design corresponding character models and backgrounds for each time period; Prepare display boards according to the character modeling and background of each time period, and fix the display boards to the display surface of the physical display device; In mode I, in response to a user scanning a display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; and according to the display board image data scanned by the user, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display, and the virtual component three-dimensional model includes an event card in the display scene; the event card is generated according to the events occurring in the time period corresponding to the display board, and each event card represents an event occurring to a person in the time period corresponding to the display board; the event card can move with the corresponding display surface in the display scene; a map is mapped on the display surface in the display scene, the event card is located at the place where the event occurs on the map, and the distance between the event card and the map plane is determined according to the time when the event occurs; In mode II, in response to a user scanning the display board through a mobile terminal, a display scene including a physical display device is displayed on the user's mobile terminal; based on the display board image data scanned by the user, a virtual character three-dimensional model of the character in the corresponding time period is displayed; and In response to the user's operation, the three-dimensional model of the virtual object is used to set the scene; In mode III, in response to the user scanning the display board through the mobile terminal, the display scene including the physical display device is displayed on the user's mobile terminal; in response to the user's operation, a screenshot of the current display scene is taken and saved; the captured picture can be edited.

5. The method for cultural interaction based on augmented reality according to claim 4, characterized in that: In mode I, the scanned display board is switched by rotating the physical display device and / or adjusting the scanning angle of the mobile terminal, thereby displaying different scanned display boards, and based on the scanned display board image data, a virtual component three-dimensional model is superimposed on the image of the physical display device in the display scene for display.

6. The method for cultural interaction based on augmented reality according to claim 4, characterized in that: The creation of the event card specifically includes: S501, establishing the three-dimensional coordinates of the display board in the display scene based on the position of the display board in the physical display device, wherein the length and width of the display board correspond to the x-axis and the y-axis, and the thickness direction of the display board corresponds to the z-axis; S502, according to the events occurring in the corresponding time period on each display board, draw the corresponding event board using Adobe illustrator and model it using Rhino; S504: The place where the event occurs corresponding to the event card is used as the x-coordinate and y-coordinate of the event card on the corresponding display board, and the time when the event occurs is used as the z-coordinate.

7. The method for cultural interaction based on augmented reality according to claim 4, characterized in that: In mode I, in response to the user's operation, the opera video is played; or, in mode II, in response to the user's operation, the virtual object three-dimensional model is subjected to corresponding adding, deleting, dragging, rotating, and scaling operations.

8. The method for cultural interaction based on augmented reality according to claim 4, characterized in that: In mode II, in response to the user's operation, the text introduction of the corresponding event on the display board in the current display scene is displayed.

9. The method for cultural interaction based on augmented reality according to claim 4, characterized in that: In mode II, in response to the user's operation, the singing mode is entered; the songs sung correspond to the events on the display board in the current display scene; in the singing mode, different users are responsible for singing and scene filming respectively.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for cultural interaction based on augmented reality described in any one of claims 4 to 9 is implemented.

Citation Information

Patent Citations

  • Method and system for interactive display of cultivation culture virtual scene

    CN106530392A

  • Public cultural resource virtual reality panoramic display system based on multiple terminals

    CN107393015A

  • Cultural relic virtuality and reality combined display method and system, computer equipment and terminal

    CN113485552A

  • Digital vow tablet device

    CN115454256A

  • Method and device for inheriting cultural data in virtual reality, equipment and medium

    CN116977147A