An augmented reality-based virtual scene interaction method and system, and a storage medium

Abstract

The application provides a kind of virtual scene interaction method, system and storage medium based on augmented reality, it is related to image processing technical field, the method comprises: when user completes to build the interactive building block corresponding to cultural relic, obtains the two-dimensional image data of interactive building block at any angle;According to two-dimensional image data, the current three-dimensional structure of interactive building block is identified;Determine whether the current three-dimensional structure reaches preset condition;If yes, then according to the current three-dimensional structure, generate the scene model corresponding to the current three-dimensional structure and preset related action;According to scene model and preset related action, obtain the image, animation and text introduction corresponding to cultural relic.The application utilizes image processing and machine learning technology, identifies the three-dimensional structure of interactive building block, generates preset action based on real three-dimensional structure, to improve the experience of user in obtaining cultural relic related knowledge.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a virtual scene interaction method, system and storage medium based on augmented reality. Background Art

[0002] In today's digital field, augmented reality technology is applied in many fields, including the popularization of historical relics with augmented reality technology to meet users' needs for knowledge related to relics. Generally, augmented reality and virtual reality technologies are used to experience and apply cultural relics. For example, by building blocks corresponding to cultural relics and scanning specific marks or pictures on the blocks, a virtual scene model of the cultural relics is obtained, and the scene model is integrated with the real scene to obtain image animation and text introduction based on augmented reality, which is convenient for users to interact and obtain knowledge related to cultural relics.

[0003] In the existing technology, knowledge is acquired only by scanning fixed patterns, resulting in no necessary connection between the user's action of building blocks and the acquisition of knowledge. This lacks interactivity and fun with the user and is unable to attract the user's sustained attention. Summary of the Invention

[0004] The problem solved by the present invention is how to improve the user's experience in acquiring knowledge related to cultural relics.

[0005] To solve the above problems, the present invention provides a virtual scene interaction method based on augmented reality, comprising:

[0006] When the user completes building the interactive building blocks corresponding to the cultural relics, two-dimensional image data of the interactive building blocks at any angle is obtained;

[0007] identifying a current three-dimensional structure of the interactive building block according to the two-dimensional image data;

[0008] Determining whether the current three-dimensional structure meets a preset condition;

[0009] If yes, generating a scene model and preset related actions corresponding to the current three-dimensional structure according to the current three-dimensional structure;

[0010] According to the scene model and the preset related actions, images, animations and text introductions corresponding to the cultural relics are obtained.

[0011] Optionally, identifying the current three-dimensional structure of the interactive building block according to the two-dimensional image data includes:

[0012] Inputting the two-dimensional image data into a three-dimensional convolutional neural network;

[0013] Acquiring high-level features of the two-dimensional image data through the three-dimensional convolutional neural network;

[0014] The current three-dimensional structure is generated according to the high-level features.

[0015] Optionally, acquiring high-level features of the two-dimensional image data through the three-dimensional convolutional neural network includes:

[0016] Performing feature extraction on the two-dimensional image data through the convolution layer and the pooling layer of the three-dimensional convolutional neural network to obtain multiple feature representations;

[0017] The feature representation is summarized through the fully connected layer of the three-dimensional convolutional neural network, and the feature representation having a characterization of the three-dimensional structure is used as the high-level feature.

[0018] Optionally, generating the current three-dimensional structure according to the high-level features includes:

[0019] Summarizing the high-level features to obtain a high-level feature vector;

[0020] Mapping the high-level feature vector to a three-dimensional space to obtain a structural representation;

[0021] The structural representation is optimized using a loss function, and the optimized structural representation is used as the current three-dimensional structure.

[0022] Optionally, determining whether the current three-dimensional structure meets a preset condition includes:

[0023] By comparing the current three-dimensional structure with a preset standard structure, determining whether the current three-dimensional structure meets the preset condition;

[0024] Wherein, when the current three-dimensional structure is the same as the preset standard structure, it is determined that the current three-dimensional structure meets the preset condition;

[0025] When the current three-dimensional structure is different from the preset standard structure, it is determined that the current three-dimensional structure does not meet the preset condition.

[0026] Optionally, generating a scene model and preset related actions corresponding to the current three-dimensional structure according to the current three-dimensional structure includes:

[0027] dividing the current three-dimensional structure into a plurality of geometric objects;

[0028] Establishing an initial model according to the relative positions of the geometric objects and a preset scene layout rule; wherein the three-dimensional structure of the initial model is the same as the current three-dimensional structure;

[0029] Rendering the initial model to obtain the scene model;

[0030] The preset related actions are generated according to the scenario model.

[0031] Optionally, generating the preset related action according to the scenario model includes:

[0032] Adjusting the relative position of the preset related action in the scene model to a preset standard position;

[0033] The preset related actions are rendered and integrated with the scene model.

[0034] Optionally, obtaining an image, animation, and text introduction corresponding to the cultural relic based on the scene model and the preset related actions includes:

[0035] Identifying the position and posture of the interactive building block in the real environment based on the two-dimensional image data;

[0036] By means of augmented reality technology, the scene model and the preset related actions are integrated with the real environment where the interactive building blocks are located according to the position and the posture, so as to obtain images, animations and text introductions corresponding to the cultural relics.

[0037] The present invention also provides a virtual scene interaction system based on augmented reality, comprising a memory and a processor;

[0038] The memory is used to store computer programs;

[0039] The processor is configured to implement the above-mentioned virtual scene interaction method based on augmented reality when executing the computer program.

[0040] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the virtual scene interaction method based on augmented reality as described above is implemented.

[0041] The present invention utilizes an augmented reality-based virtual scene interaction method, system, and storage medium to build cultural relic building blocks and learn related knowledge through augmented reality. When a user completes building the interactive building blocks corresponding to the cultural relic, two-dimensional image data of the interactive building blocks at any angle is captured. Based on the two-dimensional image data, the current three-dimensional structure of the interactive building blocks is identified. Image processing and machine learning techniques are applied to quickly and accurately identify the three-dimensional structure of the interactive building blocks from captured two-dimensional images. Simultaneously, the system determines whether the current three-dimensional structure meets preset requirements and provides real-time dynamic feedback to the user on whether the requirements are met, helping them optimize the interactive building block construction process. When the current three-dimensional structure meets the preset requirements, a highly realistic and engaging scene model and related animation effects are generated. This not only highlights the historical value of the cultural relic but also increases the educational interest. The present invention utilizes image processing and machine learning techniques to analyze images of the interactive building blocks built by the user and, based on the user's building status, generates preset actions in real time. This eliminates the need for fixed markers to generate fixed preset actions, closely linking the user's building process with subsequent action generation. This allows users to identify building block models without requiring specific angles or ranges, facilitating user operation and providing a more user-friendly experience. Through real-time analysis and feedback, users can immediately see the results of their operations, which enhances interactivity and user satisfaction and improves the user experience in acquiring knowledge related to cultural relics. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flowchart of a virtual scene interaction method based on augmented reality in one embodiment of the present invention;

[0043] Figure 2 A schematic diagram of images, animations and text introductions generated in another embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of a scene model generated according to the current three-dimensional structure in another embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0046] Combine Figure 1 As shown, the present invention provides a virtual scene interaction method, system and storage medium based on augmented reality, including:

[0047] When the user completes building the interactive building blocks corresponding to the cultural relics, two-dimensional image data of the interactive building blocks at any angle is obtained.

[0048] Specifically, in a preferred embodiment of the present invention, first, after the user completes building the building blocks, he or she can use a mobile phone or other terminal to take a photo of the completed interactive building blocks to obtain corresponding two-dimensional image data. It is worth mentioning that mobile phone photography can obtain two-dimensional image data at any angle.

[0049] The current three-dimensional structure of the interactive building block is identified according to the two-dimensional image data.

[0050] Specifically, in a preferred embodiment of the present invention, after acquiring two-dimensional image data, an augmented reality experience that integrates virtuality and reality can be achieved through the combination of AR technology and 3D modeling technology. The mobile phone camera is activated to capture an image of a building block toy. The application uses an image processing algorithm to pre-process the image and uses a deep learning model to identify the three-dimensional structure of the building blocks, replacing the specific markings that rely on scanning in traditional AR technology. A network structure suitable for three-dimensional model recognition is designed through a deep learning framework, and a three-dimensional convolutional neural network is selected for model training using a labeled data set. The input is a two-dimensional image frame or video frame of the building block model, and the output is the current three-dimensional structure of the interactive building block.

[0051] It is determined whether the current three-dimensional structure meets a preset condition.

[0052] Specifically, based on the identified three-dimensional structure, but taking into account that the building blocks built by the user may have errors, resulting in the three-dimensional structure being different from the standard structure, therefore, in a preferred embodiment of the present invention, it is necessary to compare the identified three-dimensional structure with the standard model to determine whether the current three-dimensional structure meets the preset conditions.

[0053] If so, a scene model and preset related actions corresponding to the current three-dimensional structure are generated according to the current three-dimensional structure.

[0054] Specifically, in a preferred embodiment of the present invention, if the current 3D structure is determined to meet preset conditions, a model of the current 3D structure is created and rendered according to a preset rendering method to obtain a scene model corresponding to the current 3D structure and preset related actions, where the preset related actions are generally animations, text, or image motion trajectories. If not, it means that the 3D structure of the interactive building block does not meet the preset requirements. Therefore, in a preferred embodiment of the present invention, when the 3D structure does not meet the preset requirements, a text prompt can be provided to the user.

[0055] According to the scene model and the preset related actions, images, animations and text introductions corresponding to the cultural relics are obtained.

[0056] Specifically, once the 3D model meets the requirements, the system generates a corresponding cultural relic scene model based on the model and, in conjunction with preset actions, creates a dynamic display effect. Simultaneously, the system extracts relevant images, animations, and detailed text descriptions of the cultural relic, enriching the user's learning experience. Once the relevant images, animations, and text descriptions are generated, users can interact with them by touching a mobile phone or other device. These images, animations, and text descriptions can include animations of characters running, jumping, and colliding, enabling users to learn about the cultural relic during the game. In a preferred embodiment of the present invention, the generated images, animations, and text descriptions are imported into the Unity engine, allowing users to control these images, animations, and detailed text descriptions through touchscreen swipes or accelerometers. Upon receiving user input, the Unity engine utilizes efficient graphics rendering technology to render realistic 3D models and scene models. Unity's built-in physics engine simulates real-world physical effects, such as gravity, collisions, and friction. Unity can handle various user inputs, such as touch, swipe, or click, providing users with diverse interaction options.

[0057] In summary, in a preferred embodiment of the present invention, a virtual scene interaction method based on augmented reality can be applied to the popularization of salt-making portrait bricks. First, the camera of a mobile device is used to capture an image of the building block model. The image is subjected to basic preprocessing, such as denoising and contrast enhancement. Then, high-level features are gradually abstracted through multiple convolutional layers and pooling layers of a three-dimensional convolutional neural network. Based on the input image, the three-dimensional structure of the building block is reconstructed through a deep learning model. The reconstructed three-dimensional model is compared with the preset standard model to determine whether the assembly is correct. The image captured by the camera is analyzed in real time using the computing power and deep learning model of the mobile phone. When the building block model is assembled correctly, the application is triggered to enter the AR experience mode, showing an augmented reality effect combining virtuality and reality, and generating preset related actions. On this basis, users can interact with the images, animations and text introductions corresponding to the cultural relics through sliding on the touch screen or using the acceleration sensor with preset related actions, thereby realizing user games. For the popularization of salt-making portrait bricks, combined Figure 2 The resulting images, animations, and text descriptions of the augmented reality-based virtual scene interaction method are shown. Users can control characters in a parkour game, running, jumping, and avoiding obstacles, simulating the challenges and difficulties of salt production and gaining a deeper understanding of salt-making bricks. This not only provides digital products such as 3D modeling, popular science animation, and parkour games, but also incorporates AR technology to provide users with a comprehensive, multi-dimensional learning experience, enhancing the fun and interactivity of traditional cultural education.

[0058] The present invention uses augmented reality to interact with virtual scenes. This method uses augmented reality to build cultural relic building blocks and learn related knowledge. When a user completes building the interactive building blocks corresponding to the cultural relic, two-dimensional image data of the interactive building blocks at any angle is captured. Based on the two-dimensional image data, the current three-dimensional structure of the interactive building blocks is identified. Image processing and machine learning techniques are applied to quickly and accurately identify the three-dimensional structure of the interactive building blocks from captured two-dimensional images. Simultaneously, the method determines whether the current three-dimensional structure meets preset requirements and provides real-time dynamic feedback to the user on whether the requirements are met, helping them optimize the interactive building block construction process. When the current three-dimensional structure meets the preset requirements, a highly realistic and engaging scene model and related animation effects are generated. This not only highlights the historical value of the cultural relic but also increases the educational interest. The present invention utilizes image processing and machine learning techniques to analyze images of the interactive building blocks built by the user. Based on the user's building status, preset actions are generated in real time. This eliminates the need for fixed markers to generate fixed preset actions, closely linking the user's building process with subsequent action generation. This allows users to identify building block models without requiring specific angles or ranges, facilitating user operation and providing a more user-friendly experience. Through real-time analysis and feedback, users can immediately see the results of their operations, which enhances interactivity and user satisfaction and improves the user experience in acquiring knowledge related to cultural relics.

[0059] In an embodiment of the present invention, identifying the current three-dimensional structure of the interactive building block based on the two-dimensional image data includes:

[0060] Inputting the two-dimensional image data into a three-dimensional convolutional neural network;

[0061] Acquiring high-level features of the two-dimensional image data through the three-dimensional convolutional neural network;

[0062] The current three-dimensional structure is generated according to the high-level features.

[0063] In this embodiment, to convert two-dimensional image data into a three-dimensional structure, the two-dimensional image data is first input into a three-dimensional convolutional neural network. This special type of neural network is designed to process data with three-dimensional structures. In this case, the images used to train the neural network may come from various angles of interactive building blocks built by users. By learning from this data, the network extracts key information describing the three-dimensional shape, which may include the shape, size, position, and relationship between other building blocks. These extracted high-level features are then input into a generative model such as a generative adversarial network or an autoencoder. This model can generate the three-dimensional structure corresponding to the interactive building blocks based on the high-level features of the two-dimensional image.

[0064] The virtual scene interaction method based on augmented reality of the present invention uses a three-dimensional convolutional neural network to identify high-level features of two-dimensional images and generate corresponding three-dimensional structures, which can achieve a series of beneficial effects. Secondly, by combining other algorithms, while ensuring the quality of the model, the speed of model generation is improved, thereby providing users with a smooth and rich interactive experience.

[0065] In an embodiment of the present invention, obtaining high-level features of the two-dimensional image data through the three-dimensional convolutional neural network includes:

[0066] Performing feature extraction on the two-dimensional image data through the convolution layer and the pooling layer of the three-dimensional convolutional neural network to obtain multiple feature representations;

[0067] The feature representation is summarized through the fully connected layer of the three-dimensional convolutional neural network, and the feature representation having a characterization of the three-dimensional structure is used as the high-level feature.

[0068] In this embodiment, the network first extracts features from the input image data through a series of convolutional and pooling layers. The convolutional layer uses convolution operations to identify local patterns in the image, such as edges and textures, while the pooling layer is responsible for reducing the amount of computation and extracting more abstract feature representations, while enhancing the network's generalization capabilities. Through this series of operations, each layer produces multiple feature representations, which can be understood as abstractions of different dimensions and levels of image data. The network then aggregates the feature representations of all layers through a fully connected layer to form a comprehensive, high-level feature representation, or high-level feature, which contains comprehensive information about the three-dimensional structure of the original image.

[0069] The present invention's augmented reality-based virtual scene interaction method improves the model's applicability and flexibility by extracting high-level features and generating 3D structures. By accumulating multiple layers of feature representations and summarizing them in a fully connected layer, the network is able to capture more complex and deep image features, significantly enhancing the quality and reliability of 3D reconstruction.

[0070] In the embodiment of the present invention, generating the current three-dimensional structure according to the high-level features includes:

[0071] Summarizing the high-level features to obtain a high-level feature vector;

[0072] Mapping the high-level feature vector to a three-dimensional space to obtain a structural representation;

[0073] The structural representation is optimized using a loss function, and the optimized structural representation is used as the current three-dimensional structure.

[0074] In this embodiment, the high-level features extracted by a 3D convolutional neural network undergo a further feature aggregation process to produce a compact high-level feature vector. This feature aggregation makes it possible to extract the most critical information from a large number of features, thereby reducing the resources required for subsequent calculations and improving the operational efficiency of the entire system. This process typically involves various statistical techniques, such as mean aggregation, maximum aggregation, or direct transformation via fully connected layers. This high-level feature vector then needs to be mapped to a structural representation in 3D space. This can be achieved through various generative models, typically using reversible networks or variational autoencoders. In a preferred embodiment of the present invention, the generative model may use this high-level feature vector as coordinates in the latent space to generate a grid representation or voxel representation of the 3D model. The loss function used in this mapping process ensures that the generated 3D structure is as close to the real-world object structure as possible. To refine this 3D structure, the loss function evaluates the difference between the predicted model and the actual model and continuously adjusts the model weights through a backpropagation algorithm, ultimately resulting in an optimized structural representation.

[0075] The virtual scene interaction method based on augmented reality of the present invention obtains corresponding three-dimensional structures by processing high-level features, thereby improving user experience and meeting immediate three-dimensional modeling needs.

[0076] In an embodiment of the present invention, determining whether the current three-dimensional structure meets a preset condition includes:

[0077] By comparing the current three-dimensional structure with a preset standard structure, determining whether the current three-dimensional structure meets the preset condition;

[0078] Wherein, when the current three-dimensional structure is the same as the preset standard structure, it is determined that the current three-dimensional structure meets the preset condition;

[0079] When the current three-dimensional structure is different from the preset standard structure, it is determined that the current three-dimensional structure does not meet the preset condition.

[0080] In this embodiment, first, the system requires a standard three-dimensional model as a comparison benchmark. This standard model represents the ideal or correct structure. When the user completes the construction of the three-dimensional model, the system generates a representation of the current three-dimensional structure through the technology described above. Then, this representation is compared with the preset standard structure. Generally, the current three-dimensional structure is matched with the preset standard structure by measuring the similarity of shape, volume, and surface features, or using a feature matching algorithm. If the current three-dimensional structure and the preset structure are the same within the allowable error range, the system determines that the preset conditions are met; if they are different, it is determined that the conditions are not met, and feedback is provided to guide the user to make adjustments.

[0081] The virtual scene interaction method based on augmented reality of the present invention provides an objective and standardized evaluation mechanism to ensure the quality and accuracy of the constructed three-dimensional structure.

[0082] In the embodiment of the present invention, generating a scene model corresponding to the current three-dimensional structure and preset related actions according to the current three-dimensional structure includes:

[0083] dividing the current three-dimensional structure into a plurality of geometric objects;

[0084] Establishing an initial model according to the relative positions of the geometric objects and a preset scene layout rule; wherein the three-dimensional structure of the initial model is the same as the current three-dimensional structure;

[0085] Rendering the initial model to obtain the scene model;

[0086] The preset related actions are generated according to the scenario model.

[0087] In this embodiment, the current three-dimensional structure is first subdivided into multiple geometric objects. This process is similar to breaking up the components of a complete model, and each component corresponds to an independent geometric shape. Figure 3 As shown, each building block is treated as an independent geometric shape, and the 3D structure is subdivided according to the building block shape to recreate the same scene model as the building blocks. This allows for more flexible processing and manipulation of each part, such as in subsequent rendering and animation. Next, based on the relative positions of these geometric objects and pre-defined scene layout rules, including placement, distances between objects, and angles, an initial model is created. This initial model maintains the same 3D form as the actual 3D structure built by the user. Subsequently, through modern computer graphics rendering techniques such as ray tracing or real-time rendering, the initial model is given texture, light and shadow, and color, making it a vivid scene model. Finally, based on this scene model, a series of preset actions, such as opening, closing, and rotation, are designed and generated in 3D software or augmented reality. These actions can be pre-defined by the user or the algorithm.

[0088] The virtual scene interaction method based on augmented reality of the present invention not only facilitates system understanding and processing by decomposing and reorganizing three-dimensional structures, but also allows detailed customization and optimization of independent geometric objects when necessary, greatly improving the realism and attractiveness of the final scene.

[0089] In the embodiment of the present invention, generating the preset related action according to the scenario model includes:

[0090] adjusting the relative positions of the preset related actions in the scene model to preset standard positions;

[0091] rendering the preset related actions and fusing them with the scene model.

[0092] In the embodiment, the process of generating the preset related actions first involves action design and key frame setting, which is usually performed in three-dimensional modeling and animation software. Here, the so-called preset related actions can be understood as a series of programmed actions that will be activated when certain conditions are met, in the preferred embodiment of the application, when the user interacts with the model in a specific way. In order to enable these actions to be correctly exhibited in the scene model, the relative positions of the actions in the model need to be adjusted to ensure that they are in the preset standard positions. After the adjustment work is completed, the preset actions are then rendered. During the rendering process, these actions are given real-time texture, lighting and shadow effects in the form of animation, so as to be more naturally fused with the scene model. This requires fine-tuning of the speed, motion trajectory and interaction with the scene of the actions to ensure that the animation looks both realistic and physically consistent. By precisely placing the preset actions in the scene and fusing them with the scene model in high-quality rendering, the user can observe smooth and realistic animation effects as if the objects were moving in reality.

[0093] The virtual scene interaction method based on augmented reality of the application enables the user to improve the participation and immersion by intuitive actions, by precisely placing the preset actions in the scene and fusing them with the scene model in high-quality rendering.

[0094] In the embodiment of the application, the obtaining of the image, animation and text introduction corresponding to the cultural relic according to the scene model and the preset related actions comprises:

[0095] According to the two-dimensional image data, the position and pose of the interactive block in the real environment are identified.

[0096] According to the position and the pose, the scene model and the preset related actions are fused with the real environment where the interactive block is located by means of augmented reality technology, to obtain the image, animation and text introduction corresponding to the cultural relic.

[0097] In this embodiment, it is first necessary to identify the exact position and posture of the interactive building blocks in the real environment based on the two-dimensional image data. Obtaining this information usually involves aligning the two-dimensional image features of the target object with the three-dimensional coordinate system in the real world, which is usually accomplished through feature point matching, posture estimation, and simultaneous positioning and mapping technology. Once this information is determined, the augmented reality system can accurately superimpose the three-dimensional scene model and preset related actions on the user's real environment based on the real position and posture. In a typical embodiment, the user looks at the interactive building blocks through the camera of a device such as a smartphone, and the screen displays an environmental image that integrates virtual objects in real time. At the same time, the relevant actions on the interactive building blocks are activated, and a brief introduction to the cultural relics or a detailed story background introduction is demonstrated in the user's real environment.

[0098] The virtual scene interaction method based on augmented reality of the present invention relies on augmented reality technology to ensure the user's immersive experience and cleverly combines the historical background of cultural relics with real-life scenes.

[0099] The present invention also provides a virtual scene interaction system based on augmented reality, comprising a memory and a processor;

[0100] The memory is used to store computer programs;

[0101] The processor is configured to implement the above-mentioned virtual scene interaction method based on augmented reality when executing the computer program.

[0102] The virtual scene interaction system based on augmented reality of the present application builds cultural relics blocks through augmented reality and learns related knowledge, when the user completes the building of the interactive blocks corresponding to the cultural relics, acquires two-dimensional image data of the interactive blocks at any angle, and according to the two-dimensional image data, identifies the current three-dimensional structure of the interactive blocks, applies image processing and machine learning technology to quickly and accurately identify the three-dimensional structure of the interactive blocks from the two-dimensional image, at the same time, judges whether the current three-dimensional structure meets the preset condition, gives the user dynamic feedback whether the preset condition is met in real time, helps them optimize the construction process of the interactive blocks. When the current three-dimensional structure meets the preset requirement, a high realistic and fascinating scene model and related animation effects are generated. Not only highlights the historical value of cultural relics, but also increases the interest of education. The present application uses image processing and machine learning technology to analyze the image of the interactive blocks built by the user, and generates preset actions in real time according to the state of the interactive blocks built by the user, so as to get rid of the fixed preset actions generated by fixed markers, make the user's building process closely related to the subsequent action generation, make the user can identify the block model without specific angle and range, convenient for user operation, make the user experience more friendly. Through real-time analysis and feedback, the user can immediately see the operation result, enhance the interactivity and user satisfaction, improve the user experience in obtaining cultural relics related knowledge.

[0103] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the virtual scene interaction method based on augmented reality as described above.

[0104] The computer-readable storage medium of the present invention uses augmented reality to build cultural relic building blocks and learn related knowledge. When a user completes building the interactive building blocks corresponding to the cultural relic, the medium captures two-dimensional image data of the interactive building blocks at any angle and identifies the current three-dimensional structure of the interactive building blocks based on the two-dimensional image data. Image processing and machine learning techniques are applied to quickly and accurately identify the three-dimensional structure of the interactive building blocks from captured two-dimensional images. Simultaneously, the system determines whether the current three-dimensional structure meets preset requirements and provides real-time dynamic feedback to the user on whether the preset requirements are met, helping them optimize the interactive building block construction process. When the current three-dimensional structure meets the preset requirements, a highly realistic and engaging scene model and related animation effects are generated. This not only highlights the historical value of the cultural relic but also increases the fun of education. The present invention utilizes image processing and machine learning techniques to analyze images of the interactive building blocks built by the user and, based on the user's building status, determines and generates preset actions in real time. This eliminates the need for fixed markers to generate fixed preset actions, closely linking the user's building process with subsequent action generation. This allows users to identify building block models without requiring specific angles or ranges, facilitating user operation and providing a more user-friendly experience. Through real-time analysis and feedback, users can immediately see the results of their operations, which enhances interactivity and user satisfaction and improves the user experience in acquiring knowledge related to cultural relics.

[0105] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-described methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0106] It has to be noted that, in the present document, relational terms are intended only to convey a possible relationship between elements or

[0107] The foregoing is considered as illustrative only of the principles of the application. Numerous modifications and changes will readily occur to those skilled in the art, which modifications and changes are to be understood as intended to be encompassed by the general scope of the application. Accordingly, the application is not to be limited to the above described or illustrated embodiments that are merely given by way of example. It is also be understood that various combinations of the above described embodiments and variations thereof are encompassed by the application.

Claims

1. A virtual scene interaction method based on augmented reality, characterized in that: include: When the user completes building the interactive building block corresponding to the cultural relic, obtaining two-dimensional image data of the interactive building block at any angle; Identifying the current three-dimensional structure of the interactive building block based on the two-dimensional image data; specifically comprising: inputting the two-dimensional image data into a three-dimensional convolutional neural network; Acquiring high-level features of the two-dimensional image data through the three-dimensional convolutional neural network; generating the current three-dimensional structure according to the high-level features; Determining whether the current three-dimensional structure meets a preset condition; If so, generating a scene model and preset related actions corresponding to the current three-dimensional structure based on the current three-dimensional structure, specifically comprising: dividing the current three-dimensional structure into a plurality of geometric objects; establishing an initial model based on the relative positions of the geometric objects and a preset scene layout rule; wherein the three-dimensional structure of the initial model is the same as that of the current three-dimensional structure; rendering the initial model to obtain the scene model; and generating the preset related actions based on the scene model; Generating the preset related action according to the scene model specifically includes: adjusting the relative position of the preset related action in the scene model to a preset standard position; rendering the preset related action and fusing it with the scene model; According to the scene model and the preset related actions, an image, animation, and text introduction corresponding to the cultural relic are obtained, specifically including: identifying the position and posture of the interactive building block in the real environment according to the two-dimensional image data; and using augmented reality technology, according to the position and the posture, integrating the scene model and the preset related actions with the real environment where the interactive building block is located to obtain an image, animation, and text introduction corresponding to the cultural relic.

2. The virtual scene interaction method based on augmented reality according to claim 1, characterized in that: The obtaining of high-level features of the two-dimensional image data by the three-dimensional convolutional neural network includes: Performing feature extraction on the two-dimensional image data through the convolution layer and the pooling layer of the three-dimensional convolutional neural network to obtain multiple feature representations; The feature representation is summarized through the fully connected layer of the three-dimensional convolutional neural network, and the feature representation having a characterization of the three-dimensional structure is used as the high-level feature.

3. The virtual scene interaction method based on augmented reality according to claim 1, characterized in that: Generating the current three-dimensional structure according to the high-level features includes: Summarizing the high-level features to obtain a high-level feature vector; Mapping the high-level feature vector to a three-dimensional space to obtain a structural representation; The structural representation is optimized using a loss function, and the optimized structural representation is used as the current three-dimensional structure.

4. The virtual scene interaction method based on augmented reality according to claim 1, characterized in that: The determining whether the current three-dimensional structure meets the preset condition includes: By comparing the current three-dimensional structure with a preset standard structure, determining whether the current three-dimensional structure meets the preset condition; Wherein, when the current three-dimensional structure is the same as the preset standard structure, it is determined that the current three-dimensional structure meets the preset condition; When the current three-dimensional structure is different from the preset standard structure, it is determined that the current three-dimensional structure does not meet the preset condition.

5. A virtual scene interaction system based on augmented reality, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the augmented reality-based virtual scene interaction method according to any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the virtual scene interaction method based on augmented reality as described in any one of claims 1 to 4 is implemented.

Images