Cartoon interaction method and device and cartoon reader

Abstract

The invention discloses a cartoon interaction method and device and a cartoon reader, and the method comprises the steps: carrying out the AR scene initialization based on the cartoon content, and carrying out the 3D rendering of a cartoon object contained in the cartoon content; acquiring real-time environment information, constructing a real-scene environment according to the real-time environment information, and performing real-scene superposition on the real-scene environment and the cartoon object; and obtaining interaction information of the user, and displaying interaction with the cartoon content according to the interaction information of the user. After the cartoon content is subjected to 3D rendering, the cartoon content and the real-scene environment constructed by the real-time environment information are subjected to real-scene superposition, so that the cartoon is presented in the real-scene environment, and the sense of distance between the cartoon and a user is reduced. According to the interaction information of the user, the cartoon content can be dynamically adjusted, and the display effect of interaction with the cartoon content is presented to the user.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular, to a comic interaction method, device, and comic reader. Background Art

[0002] Comic readers help users break away from the limitations of paper comic books and can read comics anytime and anywhere, which is more portable. Comic readers can also provide users with operations such as freely adjusting the display size of comics, making comic reading more personalized.

[0003] However, existing comic readers are only limited to providing users with the display of flat images, lacking interactivity with users, and the immersion of users during reading is poor. Summary of the Invention

[0004] In view of the above problems, embodiments of this application are proposed to provide a comic interaction method, device, and comic reader that overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of the embodiments of this application, a comic interaction method is provided, which includes:

[0006] Initialize an AR scene based on comic content and perform 3D rendering on comic objects included in the comic content;

[0007] Obtain real-time environment information, construct a real scene environment according to the real-time environment information, and perform real-scene overlay with comic objects;

[0008] Obtain user interaction information, and display the interaction with the comic content according to the user interaction information.

[0009] Optionally, initializing an AR scene based on comic content and performing 3D rendering on comic objects included in the comic content further includes:

[0010] Start the comic reader and obtain comic content;

[0011] Use a 3D graphics engine to perform 3D rendering on the comic content to obtain a 3D display of comic objects included in the comic content.

[0012] Optionally, obtaining real-time environment information, constructing a real scene environment according to the real-time environment information, and performing real-scene overlay with comic objects further includes:

[0013] Use a preset environment perception device to obtain real-time environment information and construct a matching real scene environment based on the real-time environment information;

[0014] Perform dynamic real-scene overlay of the real scene environment and comic objects so that the comic objects are presented in the real scene environment in 3D form.

[0015] Optionally, performing dynamic live-action overlay of the real-world environment and the comic object so that the comic object is presented in the real-world environment in 3D further includes:

[0016] Based on the user's perspective, dynamically overlaying the comic object onto the real-world environment, where the comic object is displayed at different positions in the real-world environment according to the change of the user's perspective.

[0017] Optionally, the interaction information includes user emotion information;

[0018] Obtaining the user's interaction information and, based on the user's interaction information, displaying the interaction with the comic content further includes:

[0019] Obtaining the user's facial expression information and / or body movement information, and identifying and determining the user's emotion information based on the facial expression information and / or body movement information; and / or, obtaining the user's physiological information using a preset sensing device and determining the user's emotion information based on the physiological information;

[0020] Based on the user's emotion information, displaying the comic content based on a preset dynamic adjustment mode; where the preset dynamic adjustment mode includes one or more of the following: displaying dynamic dialogue information with the comic object, adjusting the actions and / or expressions of the displayed comic object, adjusting the background and / or music of the comic content, displaying at least one interaction option with the comic object.

[0021] Optionally, the interaction information includes the user's operation information;

[0022] Obtaining the user's interaction information and, based on the user's interaction information, displaying the interaction with the comic content further includes:

[0023] Obtaining the user's operation information; the operation information includes the user's text input, voice input, and / or gesture movement information;

[0024] Identifying the operation information, determining an interaction instruction based on the operation information, and displaying the comic content based on the interaction instruction; the interaction instruction includes one or more of the following: the comic object moves, the comic object speaks, the comic object performs a specified action, triggers a comic event, determines a plot development option.

[0025] Optionally, the interaction information includes bullet screen interaction information;

[0026] Obtaining the user's interaction information and, based on the user's interaction information, displaying the interaction with the comic content further includes:

[0027] Obtaining the bullet screen interaction information input by the user and performing 3D rendering display of the bullet screen interaction information.

[0028] According to the second aspect of the embodiments of the present application, there is provided a comic interaction device, which includes:

[0029] An initial rendering module, adapted to initialize an AR scene based on comic content and perform 3D rendering on comic objects included in the comic content;

[0030] An environment overlay module, adapted to obtain real-time environment information, construct a real scene environment based on the real-time environment information, and perform real scene overlay with comic objects;

[0031] An interaction module, adapted to obtain user interaction information and display the interaction with the comic content according to the user interaction information.

[0032] According to a third aspect of the embodiments of the present application, there is provided a comic reader, which includes: a display device, an AR device, and a sensing device;

[0033] The display device is used for: displaying comic content;

[0034] The AR device is used for displaying the 3D-rendered comic objects. The AR device further includes an environment perception device, which is used for obtaining real-time environment information to construct a real scene environment and perform real scene overlay with the comic content;

[0035] The sensing device is used for: obtaining user interaction information to display the interaction with the comic content according to the interaction information.

[0036] According to a fourth aspect of the embodiments of the present application, there is provided a computing device, including: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface complete communication with each other through the communication bus;

[0037] The memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operations corresponding to the above-mentioned comic interaction method.

[0038] According to a fifth aspect of the embodiments of the present application, there is provided a computer storage medium, in which at least one executable instruction is stored, and the executable instruction causes a processor to execute the operations corresponding to the above-mentioned comic interaction method.

[0039] According to a sixth aspect of the embodiments of the present application, there is provided a computer program product, including at least one executable instruction, and the executable instruction causes a processor to execute the operations corresponding to the above-mentioned comic interaction method.

[0040] According to the comic interaction method, device, and comic reader provided by the present application, after the comic content is 3D-rendered, it is subjected to real scene overlay with the real scene environment constructed from real-time environment information, so that the comic is presented in the real scene environment, narrowing the sense of distance from the user. According to the user interaction information, the comic content can be dynamically adjusted to present a display effect of interacting with the comic content to the user.

[0041] The above description is only an overview of the technical solution of the present application. In order to better understand the technical means of the present application, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present application more obvious and understandable, the specific embodiments of the present application are hereinafter specifically exemplified. Description of the Drawings

[0042] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0043] Figure 1 Shows a flowchart of a comic interaction method according to an embodiment of the present application;

[0044] Figure 2 Shows a flowchart of a comic interaction method according to another embodiment of the present application;

[0045] Figure 3 Shows a schematic structural diagram of a comic interaction device according to an embodiment of the present application;

[0046] Figure 4 Shows a schematic diagram of the hardware device of a comic reader according to an embodiment of the present application;

[0047] Figure 5 Shows a schematic structural diagram of a computing device according to an embodiment of the present application. Detailed Embodiments

[0048] The exemplary embodiments of the present application will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be fully conveyed to those skilled in the art.

[0049] First, the noun terms related to one or more embodiments of the present application are explained.

[0050] OpenGL: Open Graphics Library, an open graphics library, a cross-language and cross-platform application programming interface (API) for rendering 2D and 3D vector graphics, consisting of nearly 350 different function calls for drawing from simple graphics to relatively complex three-dimensional scenes.

[0051] Unity3D: Cross-platform real-time 3D interactive content creation software.

[0052] SLAM: Simultaneous Localization and Mapping, an algorithm for simultaneous localization and mapping. It is a technology that obtains environmental information through sensors (such as lidar, cameras, inertial measurement units, etc.) to achieve localization and map building.

[0053] Figure 1 The flowchart of a comic interaction method according to an embodiment of the present application is shown, as Figure 1 shown, the method includes the following steps:

[0054] Step S101, initialize the AR scene based on the comic content and perform 3D rendering on the comic objects included in the comic content.

[0055] In this embodiment, the user can use a display device to display the comic. The display device can be a mobile device such as a mobile phone or a pad, which is convenient to carry and use. When reading the comic, the user can use an AR device to present a 3D display effect of the comic content, which is also convenient for subsequent user interaction.

[0056] Specifically, initialize the AR scene according to the comic content. The AR scene initialization can present the original 2D display effect of the comic content in 3D form. When performing 3D rendering, graphics engines such as OpenGL and Unity3D can be used to combine the 2D image of the comic content with the 3D model to complete the 3D rendering of the comic content. The implementation method can be as follows:

[0057] using UnityEngine;

[0058] using Vuforia;

[0059] public class ARComicRenderer:MonoBehaviour

[0060] {

[0061] public Texture2D comicPageTexture;

[0062] public GameObject comicPageObject;

[0063] void Start()

[0064] {

[0065] / / AR scene initialization

[0066] ARSceneTracker tracker=

[0067] TrackerManager.Instance.GetTracker <arscenetracker>();

[0068] tracker.Start();

[0069] / / 3D rendering of comic object

[0070] comicPageObject = new GameObject("ComicPage");

[0071] comicPageObject.transform.position =

[0072] Camera.main.transform.position + Camera.main.transform.forward *

[0073] 5f;

[0074] comicPageObject.AddComponent <meshrenderer>();

[0075] comicPageObject.GetComponent <meshrenderer>().material.mainTexture

[0076] = comicPageTexture;

[0077] }

[0078] }

[0079] The above is for illustrative purposes. It shall be set according to the actual implementation situation and is not limited here.

[0080] Through 3D rendering of comic content, a 3D comic display can be presented to users, providing a better visual reading effect for users. The comic objects in the comic content can be various characters in the comic, such as various objects like characters, animals, and items in the comic. When 3D rendering the comic objects, multiple characters in the comic content can be rendered, or only one character in the comic content can be rendered. The character can be selected and determined by the user before rendering, or determined according to the proportion of the character in the comic content, which is not limited here.

[0081] Step S102: Obtain real-time environment information, construct a real scene environment based on the real-time environment information, and perform real-scene overlay with the comic object.

[0082] To enhance the immersive experience of users reading comics, the comic content can be overlaid with the real environment where the user is located. When users read comics, the sense of immersive experience of users can be increased.

[0083] Specifically, based on a camera device, etc., obtain the real-time environment information of the user. For example, if the user is in a natural environment, obtain natural environment information such as the information of trees, rivers, stones, etc. Or, if the user is in an indoor environment, obtain indoor environment information such as the room layout, furniture, decorations, etc. When obtaining the real-time environment information, the real-time environment information from the user's perspective is obtained, so that the comic object can be combined with the real-scene environment from the user's perspective. From the user's perspective, the effect that the comic object and the user are in the same space is presented, shortening the distance between the user and the comic object and making the comic object closer to the user. The above is for illustrative purposes. When specifically obtaining the real-time environment information, it can be set according to the actual implementation situation and is not limited here.

[0084] According to the acquired real-time environmental information, a corresponding real scene environment can be constructed. Computer vision technologies such as SLAM can be used to construct the real scene environment based on the real-time environmental information. For example, the natural environment where the user is located can be constructed according to the natural environmental information, and the indoor environment where the user is located can be constructed according to the indoor environmental information, etc. Taking the indoor environment as an example, if there are furniture such as tables and chairs in the room, when combining the comic object with the real scene environment, the comic object can be superimposed on the furniture in the user's perspective in real scene, such as the comic object standing behind the table, the comic object sitting on the chair, etc., presenting the effect that the comic object "transforms" from the comic content into the real scene environment and communicates face to face with the user.

[0085] The corresponding real scene environment can be constructed according to the real-time environmental information, which can be implemented according to the following code:

[0086]

[0087]

[0088] The above is for illustrative purposes, and the specific implementation can be set according to the actual situation and is not limited here.

[0089] Step S103, obtain the interaction information of the user, and according to the interaction information of the user, display the interaction with the comic content.

[0090] After presenting the comic object in 3D form in the user's perspective, in addition to watching the comic object, the user can also interact with the comic object, increasing the user's reading pleasure and enhancing the user's reading experience.

[0091] The interaction between the user and the comic object can be carried out in various ways. For example, the user can input information such as text or voice to have a conversation with the comic object, the user can communicate with the comic object through different actions, determine the user's current emotion according to the user's expression, and the comic object can give different feedbacks to the user according to the emotion. The user can also interact with the comic object and other users through bullet screen interaction information, etc. The above interaction forms are for illustrative purposes and are specifically set according to the actual situation and are not limited here.

[0092] The interactive information of the user can be obtained through a text input box, a voice input box, etc. of a display device, and can also be obtained through a camera device, etc. After obtaining the interactive information, the interactive information can be parsed to determine the dynamic adjustment of the display of the comic content based on the interactive information, and present the comic content that interacts with the user. For example, when the user has a conversation with a comic object, corresponding answers can be generated based on the user's input information and presented in a text box or a voice box, etc. to reply to the user and interact; or, according to the user's actions, such as when the user selects a comic object to move, the comic object can be moved according to the moving direction specified by the user's action; or, when the user's barrage interactive information contains information such as a scene, music, etc., the scene can be changed according to the scene specified by the user, and the corresponding music effect can be added according to the music specified by the user, etc. The above is for illustrative purposes, and specific settings are made according to the implementation situation to increase the interaction between the user and the comic object, and no limitation is made here.

[0093] It should be noted that the user information (including but not limited to user terminal information, user account information, etc.) and user data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for the user to choose to authorize or refuse.

[0094] According to the comic interaction method provided by the present application, after the comic content is 3D rendered, it is superimposed on the real scene constructed by the real-time environmental information, so that the comic is presented in the real scene, and the sense of distance from the user is shortened. According to the user's interactive information, the comic content can be dynamically adjusted to present a display effect of interacting with the comic content to the user.

[0095] Figure 2 The flowchart of the comic interaction method according to an embodiment of the present application is shown, as Figure 2 shown, the method includes the following steps:

[0096] Step S201, obtain comic content, and use a 3D graphics engine to perform 3D rendering on the comic content to obtain a 3D display of the comic objects included in the comic content.

[0097] After the user starts the comic reader, the obtained comic content can be 3D rendered using a 3D graphics engine. The 3D rendering can be performed on any comic object in the comic content, and a 3D effect of the comic object can be obtained based on the rendering model. When rendering, graphics engines such as OpenGL and Unity3D can be used, and the rendering result can be set according to the implementation situation, and no limitation is made here.

[0098] In step S202, use a preset environmental perception device to obtain real-time environmental information, construct a matching real scene environment based on the real-time environmental information, and perform dynamic real-scene overlay on the real scene environment and the comic object, so that the comic object is presented in the real scene environment in 3D form.

[0099] The real-time environmental information is the real-time environmental information based on the user's perspective, which can be obtained by using a preset environmental perception device. For example, the camera of an AR glasses can be used to obtain the real-time environmental information from the user's perspective, that is, the environment where the user is currently located from the user's perspective, which is convenient for dynamically overlaying the environment where the user is currently located with the comic object, presenting the effect that the comic object comes to the real world, appears around the user, and has an intimate contact with the user. The real-time environmental information may include, for example, information about each object in the environment, the distance information between each object and the user, etc.

[0100] After obtaining the environmental information, technologies such as SLAM can be used to construct the real-time environmental information into a real scene environment, and the real scene environment presented is the same as the real scene environment viewed from the user's perspective. When presenting the real scene environment, the 3D display effect of each object in the environment can be presented according to the distance between each object in the environment and the user and the different environments viewed from different user perspectives. The specific implementation can be based on SLAM construction and will not be elaborated here.

[0101] After obtaining the real scene environment, based on the user's perspective, the comic object can be dynamically overlaid on the real scene environment. The comic object can be displayed at different positions in the real scene environment according to the change of the user's perspective. For example, if the real scene environment includes furniture such as a table and a chair from the user's perspective, when the 3D comic object is displayed, it can appear around different furniture, such as standing behind the table or sitting on the chair. The size of the 3D comic object can also be adjusted based on the real scene environment, so that when the comic object is overlaid on the real scene environment, it can be more integrated into the real scene environment and avoid the problem of mismatch between the comic object and the real scene environment after the real-scene overlay. The specific method is as follows:

[0102]

[0103]

[0104] As shown above, if the SLAM technology of Vuforia is used by the ARComicRenderer class to track the user's perspective in real time, the position of the comic object is dynamically adjusted based on the user's perspective. When the real scene is overlaid, the comic object is overlaid with the real scene environment under the user's perspective, and the user interacts based on the real scene environment. The real scene environment can change with the user's perspective. When the user reads the comic using, for example, AR glasses, the corresponding real-time environment information is obtained through the camera of the AR glasses based on the user's perspective. The constructed real scene environment matches the real-time environment information of the user's perspective. After the comic object is overlaid with the real scene environment, as the user's perspective changes, the real scene environment changes, and the comic object also moves accordingly, so that it can be correctly presented within the user's perspective. The comic object can also interact with the user based on the real scene environment. For example, based on the real scene environment, it can play hide-and-seek with the user according to the user's interaction operation; or, it can move according to the user's interaction operation, etc., to enhance the fun of interaction. The above is for illustrative purposes, and it is specifically set according to the implementation situation, and is not limited here.

[0105] Step S203, based on the preset dynamic adjustment mode, display the comic content according to the user's emotional information.

[0106] In addition to reading comics, this embodiment provides an interaction between the user and the comic. The interaction can be achieved in various ways, such as by responding to the user's emotional information to interact with the user. The user's emotional information can be obtained by, for example, acquiring the user's facial expression information, body movement information, etc., and the user's emotional information is identified and determined based on the facial expression information and body movement information. The user's facial expression information and body movement information can be used to collect the user's facial expression information, the user's limb movement information, etc. through a camera, and the user's emotional information is obtained by using methods such as model recognition. For example, the user's expression is smiling, crying, etc., and the user's body movement is waving, giving a thumbs up, etc. The acquired user's facial expression information, body movement information, etc. are input into the trained machine learning model to obtain the user's emotional information. In addition to the user's facial expression information and body movement information, a preset sensing device, such as a wearable sensing device, can also be used to acquire the user's physiological information, and the user's emotional information is determined based on the physiological information. The physiological information includes, for example, heart rate, pulse, etc., and based on the physiological information, the user's current emotional information can be determined as excited, relaxed, etc. Specifically, it is as follows:

[0107]

[0108]

[0109] The analyze_facial_expression function can detect the user's face using, for example, OpenCV, and determine the detected expression as, for example, "happy". The analyze_heart_rate function can determine the user's emotional information based on the obtained heart rate value, etc. The above are examples for illustration, and specific settings are made according to the implementation situation, which are not limited here.

[0110] After obtaining the user's emotional information, the display of the comic content can be adjusted based on a preset dynamic adjustment mode to interact with the user. The preset dynamic adjustment mode can include, for example, displaying dynamic dialogue information with the comic object, adjusting the actions and / or expressions of the displayed comic object, adjusting the background and / or music of the comic content, displaying at least one interaction option with the comic object, etc. Displaying dynamic dialogue information with the comic object can include, for example, according to different emotional information of the user, the comic object displays different dialogues. For example, when it is recognized and determined that the user's emotional information is nervous, the comic character displays or inputs comforting dialogue information, etc. Adjusting the actions and / or expressions of the displayed comic object includes, for example, when the user's emotional information is surprise, the comic object also shows a surprised expression, etc. When the user's emotional information is happy, the actions of the comic object can be dancing and so on. Adjusting the background and / or music of the comic content includes, for example, when the user's emotional information is happy, dynamically adjusting the background of the comic content to make it brighter, with richer colors, and adding a lively background score, etc. Displaying at least one interaction option with the comic object includes, for example, when the user's emotional information is curiosity, providing one or more interaction options to the user, and displaying different effects according to the selected interaction options by the user. The above are examples for illustration, and specific settings can be made according to the implementation situation, adopting different dynamic adjustment modes to provide different interactive pleasures to the user, which are not limited here.

[0111] Step S204: Obtain the user's operation information, identify the operation information, determine an interaction instruction based on the operation information, and display the comic content based on the interaction instruction.

[0112] This embodiment can also provide different interaction results according to the user's operation information. The user's operation information includes, for example, the user's text input, voice input, gesture action information, etc., and the user's operation information can be obtained through the input device, voice collection device, camera device, etc. of the mobile device. For the user's text input and voice input, the operation information can be directly obtained. For the user's gesture action information, the corresponding gesture action can be recognized through image recognition and other means to obtain the operation information.

[0113] Based on the operation information, the interaction instructions for the comic object can be determined. For example, by semantic recognition of the operation information, or based on the association relationship between gesture actions and interaction instructions, etc., the interaction instructions can be determined. The interaction instructions include, for example, moving the comic object, the comic object speaking, the comic object performing a specified action, triggering a comic event, determining plot development options, etc. The interaction instruction for the comic object to move can be determined based on semantic recognition of words such as "move", "shift", "run", "walk", etc. included in the operation information, or a swipe gesture, a gesture of clicking on the destination of movement, etc. The interaction instruction can also include information such as the direction of movement, the distance of movement, the destination of movement, etc. If the operation information contains the above information, it is determined according to the operation information; if not, it can be determined based on preset information, which is not limited here. The comic object speaking can be determined based on semantic recognition of greetings, asking questions, etc. included in the operation information, or a wave gesture, a handshake gesture, etc. to determine the interaction instruction for the comic object to move. Based on the interaction instruction, the comic object is instructed to speak, and the specific conversation content can be replied by parsing the specific content in the user's operation information through, for example, a large model, which is not limited here. The comic object performing a specified action issues a specific specified action to the comic object according to the specific content recognized in the user's operation information. Triggering a comic event can preset different comic event branches, and according to the keywords and key gesture actions in the user's operation information, determine different comic event directions to trigger and display the corresponding comic events. Determining plot development options can provide the user with different options for the subsequent plot development of the comic. The user can select a plot development option through the operation information, and according to the selected plot development option by the user, display the subsequent plot development and experience different story endings, etc. The above is for illustrative purposes, and it is specifically set according to the implementation situation, which is not limited here.

[0114] Step S205, obtain the barrage interaction information input by the user, and perform 3D rendering and display on the barrage interaction information.

[0115] The interaction in this embodiment also provides a barrage interaction method for the user. Specifically, the user can input barrage interaction information in ways such as text, voice, etc., and the barrage interaction information can be 3D rendered and displayed in the AR scene, as follows:

[0116]

[0117]

[0118]

[0119] In the AR scenario, determine the position for displaying bullet screen interaction information, the moving speed of the bullet screen interaction information, etc. Here, when the bullet screen interaction information is displayed, it is presented in the real environment from the user's perspective. When the display position of the bullet screen interaction information is outside the real environment after it moves, the bullet screen interaction information is removed.

[0120] Here, the bullet screen interaction information not only includes the bullet screen interaction information sent by the user himself / herself, but also includes the bullet screen interaction information sent by other users. Displaying the bullet screen interaction information in the AR scenario, on the one hand, the bullet screen interaction information is displayed in the AR scenario, presenting a 3D display effect; on the other hand, the interaction is not limited to the comic object, but can also interact with other users, increasing the social interaction experience.

[0121] The execution order of steps S203 - S205 is not limited, and the corresponding steps are executed according to the implementation situation.

[0122] According to the comic interaction method provided by the present application, the 2D comic content is 3D rendered, and the comic object with a 3D display effect is superimposed on the real environment constructed by the real-time environment information. The comic object is presented in the real environment from the user's perspective, and its position in the real environment is adjusted according to the change of the user's perspective, narrowing the sense of distance from the user. According to different interaction information of the user, such as user emotion information, user operation information, bullet screen interaction information, etc., the dynamic adjustment of the comic content is realized, vividly presenting the interaction display effect between the user and the comic content in different situations, and increasing the user's interest in reading comics.

[0123] Figure 3 The structural schematic diagram of a comic interaction device provided by an embodiment of the present application is shown. As Figure 3 shown, the device includes:

[0124] An initial rendering module 310, adapted to initialize the AR scenario based on the comic content and perform 3D rendering on the comic object included in the comic content;

[0125] An environment superimposing module 320, adapted to obtain real-time environment information, construct a real environment according to the real-time environment information, and perform real environment superimposition with the comic object;

[0126] An interaction module 330, adapted to obtain the interaction information of the user and display the interaction with the comic content according to the interaction information of the user.

[0127] Optionally, the initial rendering module 310 is further adapted to:

[0128] Obtain the comic content;

[0129] Use a 3D graphics engine to perform 3D rendering on the comic content to obtain a 3D display of the comic object included in the comic content.

[0130] Optionally, the environmental overlay module 320 is further adapted to:

[0131] Obtain real-time environmental information using a preset environmental sensing device, and construct a matching real scene environment based on the real-time environmental information;

[0132] Perform dynamic real-scene overlay of the real scene environment and the comic object, so that the comic object is presented in the real scene environment in 3D form.

[0133] Optionally, the environmental overlay module 320 is further adapted to:

[0134] Based on the user's perspective, dynamically overlay the comic object onto the real scene environment, where the comic object is displayed at different positions in the real scene environment according to the change of the user's perspective.

[0135] Optionally, the interaction information includes user emotion information;

[0136] The interaction module 330 is further adapted to:

[0137] Obtain the user's facial expression information and / or body movement information, and identify and determine the user's emotion information according to the facial expression information and / or body movement information; and / or, obtain the user's physiological information using a preset sensing device, and determine the user's emotion information according to the physiological information;

[0138] According to the user's emotion information, display the comic content based on a preset dynamic adjustment mode; where the preset dynamic adjustment mode includes one or more of the following: display dynamic dialogue information with the comic object, adjust the actions and / or expressions of the displayed comic object, adjust the background and / or music of the comic content, display at least one interaction option with the comic object.

[0139] Optionally, the interaction information includes the user's operation information;

[0140] The interaction module 330 is further adapted to:

[0141] Obtain the user's operation information; the operation information includes the user's text input, voice input, and / or gesture movement information;

[0142] Identify the operation information, determine an interaction instruction according to the operation information, and display the comic content based on the interaction instruction; the interaction instruction includes one or more of the following: the comic object moves, the comic object speaks, the comic object performs a specified action, triggers a comic event, determines an option for plot development.

[0143] Optionally, the interaction information includes barrage interaction information;

[0144] The interaction module 330 is further adapted to:

[0145] Obtain the barrage interaction information input by the user, and perform 3D rendering display on the barrage interaction information.

[0146] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be elaborated here.

[0147] According to the comic interaction device provided by the present application, after 3D rendering the comic content, it is superimposed on the real scene environment constructed by the real-time environment information, so that the comic is presented in the real scene environment, narrowing the sense of distance from the user. According to the interaction information of the user, the comic content can be dynamically adjusted to present a display effect of interacting with the comic content to the user.

[0148] Figure 4 The structural schematic diagram of a comic reader provided by an embodiment of the present application is shown. As Figure 4 shown, the comic reader 400 includes: a display device 410, an AR device 420, and a sensing device 430.

[0149] The display device 410 is used to: display comic content. The display device 410 can be, for example, a mobile device, etc., to display a 2D effect comic on a plane, and can also be used for the user to input text or voice information, obtain the user's gesture operation information, user expression information, body movement information, etc. through the camera of the mobile device.

[0150] The AR device 420 can be used to: provide a 3D effect comic display for the user under the AR scenario, and display the 3D rendered comic object to the user for convenient interaction with the user. The AR device 420 may further include an environment perception device 421 for obtaining real-time environment information to construct a real scene environment, so as to realize real scene superposition with the comic content and facilitate the interaction between the comic object and the user in the real scene environment.

[0151] The sensing device 430 is used to obtain the interaction information of the user, such as the physiological information of the user, etc., so as to display the interaction with the comic content according to the interaction information.

[0152] The data transmission between the above hardware devices can be carried out through transmission methods such as wireless, Bluetooth, etc., which is not limited here.

[0153] According to the comic reader provided by the present application, a comic object with a 3D rendering effect is provided for the user, and the real scene environment constructed by the real-time environment information is superimposed on the comic object, so that the comic is presented in the real scene environment, narrowing the sense of distance from the user. According to the interaction information of the user, the comic content can be dynamically adjusted to present a display effect of interacting with the comic content to the user.

[0154] The present application also provides a non-volatile computer storage medium, which stores at least one executable instruction, and the executable instruction can perform the operations corresponding to the comic interaction method in any of the above method embodiments.

[0155] The present application also provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform the operations corresponding to the comic interaction method in any of the above method embodiments.

[0156] Figure 5 The structural schematic diagram of a computing device according to an embodiment of the present application is shown. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0157] As Figure 5 shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communication bus 508.

[0158] Wherein:

[0159] The processor 502, the communications interface 504, and the memory 506 communicate with each other through the communication bus 508.

[0160] The communications interface 504 is used to communicate with network elements of other devices such as clients or other servers.

[0161] The processor 502 is used to execute the program 510, and specifically can execute the relevant steps in the above comic interaction method embodiments.

[0162] Specifically, the program 510 may include program code, and the program code includes computer operation instructions.

[0163] The processor 502 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the present application. One or more processors included in the computing device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0164] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

[0165] The program 510 can specifically be used to cause the processor 502 to execute the comic interaction method in any of the above method embodiments. For the specific implementation of each step in the program 510, reference can be made to the corresponding steps and units in the above comic interaction embodiments, which will not be elaborated here. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the foregoing method embodiments, which will not be repeated here.

[0166] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings provided herein. Based on the above description, the structure required to construct such systems is obvious. In addition, the present application is not directed to any specific programming language. It should be understood that the content of the present application described herein can be implemented using various programming languages, and the description of the specific language above is for disclosing the preferred embodiments of the present application.

[0167] In the specification provided herein, a large number of specific details are set forth. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and technologies have not been shown in detail so as not to obscure the understanding of this specification.

[0168] Similarly, it should be understood that in order to streamline the present application and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting the intention that the claimed present application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all the features of the single foregoing embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, where each claim stands on its own as a separate embodiment of the present application.

[0169] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all the features disclosed in this specification (including the accompanying claims, abstract and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0170] In addition, those skilled in the art can understand that although some of the embodiments herein include certain features included in other embodiments rather than other features, the combination of the features of different embodiments means being within the scope of this application and forms different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0171] Each component embodiment of this application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some or all of the components according to this application. This application can also be implemented as a device or device program (such as a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0172] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the unit claims listing several devices, several of these devices may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.< / meshrenderer> < / meshrenderer> < / arscenetracker>

Claims

1. A comic interaction method, comprising: Initializing an AR scene based on the comic content, and performing 3D rendering on comic objects contained in the comic content; Acquire real-time environment information, construct a real-scene environment according to the real-time environment information, and superimpose the real-scene environment with the cartoon object; The user's interactive information is obtained, and based on the user's interactive information, the interaction with the comic content is displayed.

2. The method according to claim 1, wherein: The initializing the AR scene based on the comic content and performing 3D rendering on the comic objects contained in the comic content further includes: Get comic content; The comic content is 3D rendered using a 3D graphics engine to obtain a 3D display of the comic objects contained in the comic content.

3. The method according to claim 1 or 2, wherein: The acquiring of real-time environment information, constructing a real-scene environment according to the real-time environment information, and superimposing the real-scene environment with the cartoon object further comprises: Acquire real-time environmental information using a preset environmental perception device, and construct a matching real-scene environment based on the real-time environmental information; The real-scene environment and the cartoon object are dynamically superimposed on each other so that the cartoon object is presented in the real-scene environment in a 3D form.

4. The method according to claim 3, wherein: The dynamically superimposing the real scene environment and the cartoon object so that the cartoon object is presented in the real scene environment in a 3D form further comprises: Based on the user's perspective, the comic object is dynamically superimposed on the real-scene environment, wherein the comic object is displayed at different positions in the real-scene environment according to changes in the user's perspective.

5. The method according to claim 1, wherein: The interactive information includes user emotion information; The obtaining of user interaction information and displaying the interaction with the comic content according to the user interaction information further includes: Acquiring the user's facial expression information and / or body movement information, and identifying and determining the user's emotional information based on the facial expression information and / or body movement information; and / or acquiring the user's physiological information using a preset sensor device, and determining the user's emotional information based on the physiological information; According to the user's emotional information, the comic content is displayed based on a preset dynamic adjustment mode; wherein the preset dynamic adjustment mode includes one or more of the following: displaying dynamic dialogue information with the comic object, adjusting the displayed actions and / or expressions of the comic object, adjusting the background and / or music of the comic content, and displaying at least one interactive option with the comic object.

6. The method according to claim 1, wherein: The interactive information includes user operation information; The obtaining of user interaction information and displaying the interaction with the comic content according to the user interaction information further includes: Acquiring user operation information; the operation information includes user text input, voice input and / or gesture action information; Identify the operation information, determine interactive instructions according to the operation information, and display the comic content based on the interactive instructions; the interactive instructions include one or more of the following: the comic object moves, the comic object speaks, the comic object performs a specified action, triggers a comic event, and determines a plot development option.

7. The method according to claim 1, wherein: The interactive information includes barrage interactive information; The obtaining of user interaction information and displaying the interaction with the comic content according to the user interaction information further includes: Obtain the bullet screen interaction information input by the user, and perform 3D rendering and display of the bullet screen interaction information.

8. A comic interactive device, comprising: An initial rendering module, adapted to initialize an AR scene based on comic content and perform 3D rendering on comic objects contained in the comic content; An environment superposition module, adapted to obtain real-time environment information, construct a real-scene environment according to the real-time environment information, and superimpose the real-scene environment with the cartoon object; The interactive module is adapted to obtain the user's interactive information and display the interaction with the comic content according to the user's interactive information.

9. A comic reader, comprising: Display devices, AR devices, and sensor devices; The display device is used to: display the comic content; The AR device is used to display the 3D rendered comic object, and the AR device also includes an environment perception device for acquiring real-time environment information to construct a real-scene environment and superimpose the real-scene environment with the comic content; The sensing device is used to obtain user interaction information to display the interaction with the comic content according to the interaction information.

10. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the comic interaction method according to any one of claims 1-7.

11. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute an operation corresponding to the comic interaction method according to any one of claims 1 to 7.

12. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the comic interaction method according to any one of claims 1 to 7.

Images