Display system and method based on augmented reality technology

Abstract

The invention relates to the field of augmented reality, and provides a display system and method based on the augmented reality technology, and the method comprises the steps: firstly obtaining a virtual scene image and a real scene image, carrying out the semantic-level feature fusion of the scene images, synchronously obtaining the natural language description of the interaction intention of a user, and carrying out the semantic understanding; then, semantic-level feature coding representation of virtual and real scenes is extracted, multi-granularity semantic embedding of natural language description is combined, user intention representation is enhanced layer by layer by means of a recursive cross-modal attention mechanism, and finally an interaction picture highly matched with the context of the scene is generated. Thus, through a cross-modal hierarchical interaction recursion mechanism, the limitation of traditional single-modal intention recognition is broken through, deep coupling of dynamic scene semantics and user language intentions is realized, a new technical path is provided for high-immersion AR interaction, and the sense of participation of the user is improved.

Description

Technical Field

[0001] The present application relates to the technical field of augmented reality, and more particularly, in embodiments of the present application, to a display system and method based on augmented reality technology. Background Art

[0002] In recent years, augmented reality (AR) technology, through its ability to blend virtual and real environments, has demonstrated tremendous potential in industries such as industry, education, healthcare, and entertainment. Its core approach is to overlay computer-generated virtual information (such as images and 3D models) with the real environment in real time, creating a multimodal interactive experience. However, existing AR systems still face key challenges in understanding user intent: Traditional approaches often rely on a single modality (such as gesture recognition or voice commands) to infer user intent, ignoring the dynamic constraints that the contextual semantics of the virtual and real scene exert on interaction behavior. This leads to a disconnect between user intent and the semantics of the virtual and real scene, limiting the accuracy of understanding user interaction intent and, in particular, prone to ambiguous feedback in complex scenarios.

[0003] For example, in a virtual product display scenario, a user might express their interaction intent through natural language descriptions (e.g., "Place the red sofa by the window"). However, the system must simultaneously analyze the window position and spatial layout in the real scene, as well as the features of the virtual sofa model, to generate an image that meets the user's expectations. Existing technologies often process virtual and real scene information independently, or achieve multimodal fusion through simple feature splicing, making it difficult to capture semantic relevance across scenarios. Furthermore, the ambiguity of natural language interactions and the dynamic nature of scenes further exacerbate the difficulty of understanding intent.

[0004] Therefore, an optimized display system based on augmented reality technology is desired. Summary of the Invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a display system and method based on augmented reality technology, which first obtains virtual scene images and real scene images and performs semantic-level feature fusion of scene images, and simultaneously obtains the natural language description of the user's interaction intention for semantic understanding, and then extracts the semantic-level feature encoding representation of the virtual and real scenes, combines the multi-granularity semantic embedding of the natural language description, and uses a recursive cross-modal attention mechanism to strengthen the user intention representation layer by layer, and finally generates an interactive screen that is highly adapted to the scene context. In this way, through the cross-modal hierarchical interaction recursive mechanism, it breaks through the limitations of traditional single-modal intent recognition, realizes the deep coupling of dynamic scene semantics and user language intention, provides a new technical path for highly immersive AR interaction, and enhances the user's sense of participation.

[0006] According to one aspect of the present application, a display system based on augmented reality technology is provided, which includes: A virtual scene image acquisition module, used for acquiring a virtual scene image; A real scene image acquisition module, used to acquire real scene images; Interaction intention text collection module, used to obtain the natural language description of user interaction intention; A scene semantic joint encoding module, configured to perform scene image semantic level feature fusion on the virtual scene image and the real scene image to obtain a virtual-real scene image semantic joint encoding representation; An interaction intention semantic understanding module is used to perform semantic understanding on the natural language description of the user interaction intention to obtain a semantic embedding coding representation of the user interaction intention; An interaction intention semantic enhancement module is configured to use the virtual-real scene image semantic joint coding representation as auxiliary information to perform semantic enhancement on the user interaction intention semantic embedding coding representation based on cross-modal information interaction to obtain a user interaction intention semantic enhancement coding representation; The virtual interaction display module is used to generate a virtual interaction display screen based on the semantic enhancement coding representation of the user interaction intention.

[0007] According to another aspect of the present application, a display method based on augmented reality technology is provided, which includes: Acquire a virtual scene image; Acquire real scene images; Obtain natural language description of user interaction intentions; Performing scene image semantic-level feature fusion on the virtual scene image and the real scene image to obtain a virtual-real scene image semantic joint coding representation; Performing semantic understanding on the natural language description of the user interaction intention to obtain a semantic embedding coding representation of the user interaction intention; Using the virtual-real scene image semantic joint coding representation as auxiliary information, the semantic embedded coding representation of the user interaction intention is semantically enhanced based on cross-modal information interaction to obtain a semantically enhanced coding representation of the user interaction intention; Based on the semantically enhanced coded representation of the user interaction intention, a virtual interaction display screen is generated.

[0008] Compared with the existing technology, the present application provides a display system and method based on augmented reality technology. It first obtains virtual scene images and real scene images and performs semantic-level feature fusion of the scene images, and simultaneously obtains the natural language description of the user's interaction intention for semantic understanding. Then, by extracting the semantic-level feature encoding representation of the virtual and real scenes, combined with the multi-granularity semantic embedding of the natural language description, a recursive cross-modal attention mechanism is used to strengthen the user intention representation layer by layer, and finally generate an interactive screen that is highly adapted to the scene context. In this way, through the cross-modal hierarchical interaction recursive mechanism, the limitations of traditional single-modal intent recognition are broken through, and the deep coupling of dynamic scene semantics and user language intention is achieved, providing a new technical path for highly immersive AR interaction and enhancing user participation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 4 is a system block diagram of a display system based on augmented reality technology according to an embodiment of the present application.

[0011] Figure 2 Schematic diagram of data flow of a display system based on augmented reality technology according to an embodiment of the present application.

[0012] Figure 3 4 is a block diagram of a scene semantic joint encoding module in a display system based on augmented reality technology according to an embodiment of the present application.

[0013] Figure 4 This is a block diagram of an interactive intention semantic enhancement module in a display system based on augmented reality technology according to an embodiment of the present application.

[0014] Figure 5 4 is a block diagram of a virtual interactive display module in a display system based on augmented reality technology according to an embodiment of the present application.

[0015] Figure 6 Flowchart of a display method based on augmented reality technology according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0017] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0018] In addition, numerous specific details are provided in the following detailed description to better illustrate the present application. Those skilled in the art will appreciate that the present application can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present application.

[0019] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0020] In recent years, augmented reality (AR) technology has demonstrated tremendous potential in multiple fields by integrating computer-generated virtual information with the real environment in real time. However, existing AR systems still face challenges in understanding user intent, primarily because they mostly rely on a single modality (such as gesture or voice) to infer intent and fail to fully consider the impact of the contextual semantics of the virtual and real scenes on interactive behaviors. This leads to a disconnect between intent and scene semantics, which is particularly prone to misunderstandings in complex scenes. For example, in a virtual product display, a user may describe the location of an item in natural language, but the system needs to understand both the real space layout and the characteristics of the virtual objects to respond accurately. Existing methods are limited by their difficulty in capturing semantic associations across scenes. This ambiguity in natural language and the dynamic changes in scenes further complicate intent understanding.

[0021] It should be understood that since augmented reality contains virtual scenes and real scenes, in order to understand the user's interaction intention more clearly, virtual scenes and real scenes can be used as auxiliary information to enhance the accuracy of understanding the user's interaction intention, so that a virtual picture that is more in line with the user's interaction intention can be displayed. Based on this, in the technical solution of the present application, a display system based on augmented reality technology is proposed, which constructs a virtual and real scene semantic joint coding framework, and realizes the deep coupling of dynamic scene semantics and user language intention through a cross-modal hierarchical interactive recursive mechanism. Specifically, by extracting the semantic-level feature coding representation of the virtual and real scenes, combining the multi-granularity semantic embedding of the natural language description, and using the recursive cross-modal attention mechanism to strengthen the user intention representation layer by layer, finally generating an interactive picture that is highly adapted to the scene context. This method breaks through the limitations of traditional unimodal intent recognition, provides a new technical path for highly immersive AR interaction, and helps to enhance user participation.

[0022] In response to the above technical problems, this application proposes a display system based on augmented reality technology. Figure 1 4 is a system block diagram of a display system based on augmented reality technology according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the display system based on augmented reality technology according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to an embodiment of the present application, the display system 100 based on augmented reality technology includes: a virtual scene image acquisition module 110, used to acquire a virtual scene image; a real scene image acquisition module 120, used to acquire a real scene image; an interaction intention text acquisition module 130, used to acquire a natural language description of the user interaction intention; a scene semantic joint encoding module 140, used to perform scene image semantic level feature fusion on the virtual scene image and the real scene image to obtain a virtual-real scene image semantic joint encoding representation; an interaction intention semantic understanding module 150, used to perform semantic understanding on the natural language description of the user interaction intention to obtain a user interaction intention semantic embedding encoding representation; an interaction intention semantic enhancement module 160, used to use the virtual-real scene image semantic joint encoding representation as auxiliary information, and perform semantic enhancement on the user interaction intention semantic embedding encoding representation based on cross-modal information interaction to obtain a user interaction intention semantic enhancement encoding representation; a virtual interaction display module 170, used to generate a virtual interaction display screen based on the user interaction intention semantic enhancement encoding representation.

[0023] In the aforementioned augmented reality display system 100, the virtual scene image acquisition module 110 is used to acquire virtual scene images. It should be understood that a virtual scene image refers to a computer-generated digital image containing a specific object or environment. These images can be three-dimensional models, animations, or other forms of digital content. Typically, the creation of a virtual scene image begins with modeling the target object or environment. This involves using professional 3D modeling software to accurately reproduce the object's physical properties, such as shape, size, color, and texture. For example, when building a virtual furniture display application, a high-precision 3D scanner is used to scan an actual sofa to obtain its precise geometry. Material properties and surface details are then fine-tuned to make the virtual sofa appear identical to the real thing. Furthermore, factors such as lighting conditions, shadow effects, and material reflectivity must be considered to ensure that the virtual object visually integrates seamlessly with the real world. Furthermore, for dynamic elements, such as moving characters or changing weather conditions, animation tools are used to add temporal variations to enhance their realism. Extracting semantic features from the virtual content is an essential step in acquiring virtual scene images. This requires not only capturing the basic geometric information of an object's surface but also deeply understanding its underlying semantic information. For example, in a furniture display application, in addition to accurately depicting the sofa's appearance, it is also necessary to recognize and record more abstract conceptual descriptions such as "This is a red leather sofa." This helps establish a connection between the virtual object and the natural language description, enabling the system to correctly interpret the user's intention based on verbal instructions (such as "Place the red sofa by the window") and respond accordingly. Furthermore, obtaining high-quality virtual scene images requires powerful rendering capabilities. Modern augmented reality systems often strive for high-resolution, low-latency, and real-time rendering. This requires the use of efficient algorithms and technologies to optimize the graphics rendering process. For example, GPU-accelerated computing can significantly improve rendering speed while maintaining image quality. Level of Detail (LOD) technology can dynamically adjust the model's complexity based on the distance between the viewer and the virtual object, reducing unnecessary computational overhead and improving overall performance.

[0024] In the aforementioned augmented reality display system 100, the real-scene image acquisition module 120 is used to acquire real-scene images. It should be understood that real-scene images capture the actual visual information of the user's environment, including but not limited to key elements such as spatial layout, object positions, lighting conditions, and color distribution. By accurately acquiring and interpreting this information, the system can create a digital mapping that is highly consistent with the user's actual environment. First, it needs to understand the user's actual physical environment. For example, in a home decoration application, the system needs to determine the optimal placement of virtual furniture based on the specific dimensions and layout of the user's room. This requires the system to not only identify the location of walls, floors, and ceilings, but also the specific locations of windows and doors to accurately simulate how light enters the virtual environment. Furthermore, given the varying light reflection properties of different materials, such as the distinct reflection properties of wooden and marble floors, these details must be meticulously captured to ensure that virtual objects appear to exist naturally within the environment. Second, acquiring real-scene images enhances the system's immersiveness. By capturing high-quality real-scene images, virtual objects can be more harmoniously integrated with their surroundings. This is particularly important for enhancing the user experience. When users wear AR devices and view virtual content, if there's a noticeable mismatch or dissonance between virtual objects and the background, the experience can be severely disrupted. For example, displaying a virtual character with strong shadows on a sunny day without considering the actual lighting conditions in the environment can cause the character to appear out of place, even undermining the authenticity and coherence of the entire experience. Conversely, by capturing the real scene with high precision and incorporating this information into the rendering of virtual objects, the interaction between virtual elements and the real world becomes more natural and fluid, significantly improving user engagement and satisfaction. Furthermore, capturing real-world imagery involves understanding and adapting to dynamic changes. The real world isn't static; over time, lighting conditions change, and objects can shift positions. Therefore, to make the augmented reality experience more vivid and lifelike, the system must be able to perceive and respond to these changes in real time. This requires not only capturing a one-time snapshot of the scene at the initial stage, but also continuously monitoring changes in the environmental state. For example, in an augmented reality game for outdoor activities, as the sun rises and sets, the lighting conditions will constantly change. At this time, the system needs to be able to automatically adjust the brightness and shadow effects of virtual elements to maintain consistency with the surrounding environment. At the same time, if other participants join the game, their position changes will also affect the display of virtual objects, which requires the system to have the ability to process and update quickly to ensure that all players have a synchronized and consistent experience. Specifically, high-quality camera equipment is needed to capture visual information about the user's surroundings.These cameras not only require high resolution to ensure clear image details, but also need to be able to adapt to varying lighting conditions to provide stable and reliable image output in all environments. For example, in bright outdoor sunlight or low indoor light conditions, the camera should automatically adjust parameters such as exposure time and gain to ensure that the captured image is neither overexposed nor understated, thereby maintaining consistent image quality. To further improve the quality of real-world images, a depth sensor is also included. This sensor measures the distance between objects and the camera, generating a map containing depth information. This map helps the system accurately identify the relative positions of different objects. For example, when deploying augmented reality applications in a room, a depth sensor can accurately depict the specific positions of walls, furniture, and other obstacles, allowing virtual objects to be placed appropriately on top of these entities without appearing obtrusive. Depth information can also be used to optimize shadow effects and reflected light, making virtual elements appear more integrated into the real environment.

[0025] In the above-mentioned display system 100 based on augmented reality technology, the interaction intention text collection module 130 is used to obtain a natural language description of the user's interaction intention. It should be understood that the "natural language description of the user's interaction intention" refers to the sentence content in which the user uses natural language to express his or her operation or interaction needs in the augmented reality scene. For example, in the virtual product display scene mentioned above, "Put the red sofa by the window" is a natural language description of the user's interaction intention. It clearly expresses the user's intention to adjust the position of the virtual red sofa model and clearly points out that the target placement position is by the window. Therefore, by obtaining the natural language description of the user's interaction intention, the display system can conduct in-depth analysis and processing on it, thereby more accurately understanding the user's true intention.

[0026] Figure 3 FIG is a block diagram of a scene semantic joint encoding module in a display system based on augmented reality technology according to an embodiment of the present application. Figure 3 As shown, in an embodiment of the present application, the scene semantic joint encoding module 140 includes: a scene image semantic-level feature extraction unit 141, used to extract scene image semantic-level features from the virtual scene image and the real scene image to obtain a virtual scene image feature coding map and a real scene image feature coding map; a virtual-real scene image semantic feature fusion unit 142, used to fuse the virtual scene image feature coding map and the real scene image feature coding map to obtain a virtual-real scene image semantic joint coding map as the virtual-real scene image semantic joint coding representation.

[0027] More specifically, the scene image semantic-level feature extraction unit 141 is configured to pass the virtual scene image and the real scene image through a pyramid network-based scene semantic feature extractor to obtain a virtual scene image feature encoding map and a real scene image feature encoding map. It should be understood that since user interaction intent often involves both virtual object attributes (e.g., color and shape) and real-world constraints (e.g., spatial layout and object position), in order to address the semantic separation between virtual and real scenes in traditional methods, the technical solution of this application further extracts scene image semantic-level features from the virtual scene image and the real scene image to obtain a virtual scene image feature encoding map and a real scene image feature encoding map. By independently extracting semantic features from the virtual and real scene images, the structural features of the virtual model and the spatial topological relationship of the real scene can be captured. For example, for a virtual sofa model, attributes such as its material and size need to be extracted, while for the location of a window in the real scene, physical constraints such as lighting direction and occlusion relationships need to be identified. This separate feature extraction method avoids the interference of multimodal information at the raw data level, enabling the system to explicitly model the potential associations between virtual elements and the real environment, laying the foundation for subsequent cross-modal fusion. In particular, by adopting a hierarchical feature extraction architecture such as a pyramid network, scene semantics can be parsed at multiple scales: shallow networks capture local details (such as the surface texture of a sofa), while deep networks extract global context (such as the overall layout of a room). This multi-granularity feature encoding not only retains the geometric and semantic information of the scene, but also implies the spatial compatibility rules of virtual and real elements. For example, when the user specifies a "window position", the high-dimensional representation of the window area in the real scene feature encoding map will form a potential match with the placeable area vector in the virtual scene feature.

[0028] Specifically, the virtual-real scene image semantic feature fusion unit 142 is configured to fuse the virtual scene image feature encoding map and the real scene image feature encoding map to obtain a virtual-real scene image joint semantic encoding map as the virtual-real scene image joint semantic encoding representation. It should be understood that since the essence of user interaction intent is to establish a spatial semantic relationship between virtual objects and the real environment (e.g., "a red sofa placed by the window" requires both the sofa color attribute and the window position constraint), simply processing the two scene features separately will result in a semantic gap. Therefore, to overcome the bottleneck of the lack of cross-scene semantic association in traditional methods, the technical solution of this application further fuses the virtual scene image feature encoding map and the real scene image feature encoding map to obtain a virtual-real scene image joint semantic encoding map. During the fusion process, various mathematical operations can be employed, such as feature concatenation, position-based dot multiplication, position-based subtraction, and weighted fusion, to promote interaction between data from different sources and enhance the effectiveness and consistency of the overall representation. These operations not only help establish a stable association structure between virtual objects and the real background, but also promote deep semantic coupling between the two.

[0029] In an embodiment of the present application, the interaction intention semantic understanding module 150 is used to perform semantic understanding on the natural language description of the user interaction intention to obtain a semantic embedded coding representation of the user interaction intention. Specifically, the interaction intention semantic understanding module 150 is used to: perform semantic understanding based on embedded coding on the natural language description of the user interaction intention to obtain a user interaction intention semantic embedded coding vector as the semantic embedded coding representation of the user interaction intention. It should be understood that when the user uses natural instructions such as "put the red sofa by the window", the system needs to deconstruct the multi-level semantics implicit in the language: color attributes (red), object categories (sofa), spatial relationships (by the window) and action intentions (placement). Traditional rule-based keyword extraction methods have difficulty capturing the association logic between these semantic elements. For example, they cannot distinguish between the subordinate relationship between "red sofa" as a whole concept and "red" modifying "sofa", and they cannot understand the spatial topological constraints of the vertical wall surface and the floor plane implied by "by the window". Therefore, to address the intent parsing challenges presented by the ambiguity of language interaction and the dynamic nature of scenarios, the technical solution of this application further semantically understands the natural language description of the user interaction intent to obtain a semantic embedding encoding vector for the user interaction intent. Specifically, the input data is first segmented to identify key elements such as "red," "sofa," and "by the window." Subsequently, a pre-trained language model (such as one based on the Transformer architecture) is used to convert these words into corresponding word embedding vectors. This word embedding approach not only considers the meaning of the words themselves but also captures the distinct semantic characteristics exhibited by words in different contexts. For example, the combination of "red" and "sofa" is interpreted as a holistic concept, rather than simply concatenating two separate words. Furthermore, the sentence structure is parsed to understand the grammatical and logical relationships between its components. This includes identifying subject-verb-object structures, modifier relationships, and the directionality of demonstrative pronouns. This approach enables a deeper understanding of the sentence's semantics, beyond superficial word matching. For example, in the example above, "by the window" isn't just a simple directional description; it also implies specific location information relative to the room layout, which is crucial for correctly executing the user's command. In addition, given the ambiguity and diversity of natural language, a certain level of reasoning ability is also required to fill in information gaps or resolve ambiguities. This means that in some cases, it is necessary to combine context or other external knowledge to infer the explanation that best matches the user's intention. For example, if the user only says "move the sofa away," it is necessary to determine which sofa should be moved and where it should be moved based on the current scenario to meet the user's expectations. Ultimately, after the above series of processing steps, a semantic embedding encoding vector will be obtained that comprehensively reflects the user's interaction intention. As a compact and rich representation, this vector not only contains the specific content expressed by the user, but also covers the intentions and expectations behind it.Semantic embedding encoding maps discrete word sequences into a continuous, high-dimensional vector space, where semantically similar phrases (such as "near the window" and "placed by the window") form compact clusters. This allows for the construction of a language representation aligned with the semantic space of the virtual and real scene, establishing a reference coordinate system for subsequent cross-modal fusion. Specifically, by employing multi-granular semantic embedding, the system builds hierarchical semantic representations at the word level ("red"), phrase level ("window position"), and sentence level (overall intent). For example, during semantic understanding, the Transformer architecture of word embeddings captures the modifier relationship between "red" and "sofa" through a self-attention mechanism, while preserving the relative position of "by the window" in the spatial layout through positional encoding. This structured embedding decouples elements such as color, material, and spatial orientation in the language description into interpretable feature dimensions, which are then computably aligned with the material parameters of the virtual scene and the depth map of the real scene.

[0030] Figure 4 FIG is a block diagram of an interactive intention semantic enhancement module in a display system based on augmented reality technology according to an embodiment of the present application. Figure 4As shown, in an embodiment of the present application, the interaction intention semantic enhancement module 160 includes: a virtual-real scene image semantic local feature compression unit 161, configured to perform information compression based on feature dissociation analysis on the virtual-real scene image semantic joint coding map to obtain a set of virtual-real scene image semantic local feature compression coding vectors; a user interaction intention semantic feature excitation unit 162, configured to perform information excitation based on deconvolution coding on the user interaction intention semantic embedding coding vector to obtain a user interaction intention semantic feature excitation coding vector; and a user interaction feature-level semantic interaction recursive analysis unit 163, configured to perform feature-level semantic interaction recursive analysis on the set of virtual-real scene image semantic local feature compression coding vectors and the user interaction intention semantic feature excitation coding vector to obtain a user interaction intention semantic enhancement coding vector as the user interaction intention semantic enhancement coding representation. It should be understood that when the user issues the command "Place the red sofa by the window," although the semantic embedding coding has parsed the color attributes and spatial orientation, traditional methods cannot dynamically associate the geometric coordinates of the real-scene window with the available placement area of the virtual sofa model. Therefore, in order to overcome the interaction dilemma of the separation between natural language intent and scene context, the technical solution of this application further uses the virtual-real scene image semantic joint coding map as auxiliary information to perform semantic enhancement on the user interaction intention semantic embedding coding vector based on cross-modal information interaction to obtain the user interaction intention semantic enhancement coding vector as the user interaction intention semantic enhancement coding representation. By introducing the virtual-real scene image semantic joint coding map as auxiliary information, the system can establish a mapping bridge between language symbols and physical space, achieve deep coupling between user interaction language intent and virtual-real scene semantics, and construct an interactive logical reasoning framework with physical constraints. Specifically, the process of semantic enhancement based on cross-modal information interaction uses deconvolution to excite the user interaction intention semantic embedding coding vector, mapping it to a high-dimensional space aligned with scene features, so that the "red" attribute vector dimension resonates with the material parameter space of the virtual sofa. At the same time, the virtual-real scene image semantic joint coding map is adaptively compressed using void convolution, preserving the depth information and lighting parameters of the window area, and capturing the spatial topological relationship between the vertical wall surface and the floor plane through dynamic adjustment of the receptive field. Furthermore, in the first-level feature interaction unit, the stimulated intent vector "red sofa" interacts locally with the compressed scene feature "window area," generating an initial matching matrix through a dot product operation in the feature space. For example, when there are multiple windows in a scene, the system calculates the spatial compatibility of each window feature with the "window-side" semantic vector to select the ergonomically effective area.A second-level recursive interaction unit driven by an LSTM then models the correlations between these local matching results, simulating the human decision-making process of gradually eliminating distracting factors (such as blocked windows or poorly lit areas). Ultimately, it generates a semantically reinforced encoding vector of the user's interaction intent, taking into account the constraints of virtuality and reality. This hierarchical interaction mechanism enables the system to handle complex instructions such as "move the coffee table before placing the rug," preserving the changes to the scene state caused by the preceding and following actions through memory gating.

[0031] In an embodiment of the present application, the virtual-real scene image semantic local feature compression unit 161 is used to: perform feature decomposition on the virtual-real scene image semantic joint coding map along the channel dimension to obtain a set of virtual-real scene image semantic local feature coding matrices; combine the feature distribution spatial characteristics of each virtual-real scene image semantic local feature coding matrix in the set of virtual-real scene image semantic local feature coding matrices, and perform information compression based on void convolution coding on each virtual-real scene image semantic local feature coding matrix to obtain a set of virtual-real scene image semantic local feature compression coding vectors.

[0032] Specifically, feature dissociation is performed on the virtual-real scene image semantic joint coding map along the channel dimension to obtain a set of virtual-real scene image semantic local feature coding matrices, which is expressed as the virtual-real scene feature dissociation formula: in, is the virtual-real scene image semantic joint coding map, For feature dissociation processing, They are the first, second, and third in the set of semantic local feature encoding matrices of virtual-real scene images. and The virtual-real scene image semantic local feature encoding matrix is constructed. It is understandable that, given that the virtual-real scene image semantic joint encoding map carries the fusion features of multi-source heterogeneous data in the channel dimension, these features may have potential information coupling and redundant interference during the initial fusion process. Because the material properties and geometric structure of the virtual scene and the spatial topology and lighting conditions of the real scene are represented differently at the channel level, directly processing undisentangled composite features will weaken the model's ability to resolve cross-modal semantic associations. Through structured feature deconstruction, the intertwined virtual and real scene semantic features in the channel dimension can be decomposed into multiple virtual-real scene image semantic local feature encoding matrices with independent representation capabilities, allowing each virtual-real scene image semantic local feature encoding matrix to focus on key information at a specific modality or semantic level. This channel-dimensional disentanglement operation essentially constructs a decoupled mapping of the feature space, allowing the color and texture features of the virtual object and the depth and layout features of the real scene to maintain their inherent physical constraints in independent channels. In this way, the ability to express refined features can be improved. By eliminating cross-channel feature interference and enhancing local semantic independence, a clear semantic foundation is provided for subsequent cross-modal interactions, enabling the system to accurately identify the correspondence between key elements such as "red sofa material characteristics" and "window area spatial coordinates", thereby supporting the dynamic matching of user intentions and physical constraints in virtual and real fusion scenarios.

[0033] Specifically, in combination with the feature distribution space characteristics of each virtual-real scene image semantic local feature coding matrix in the set of virtual-real scene image semantic local feature coding matrices, each virtual-real scene image semantic local feature coding matrix is subjected to information compression based on void convolution coding to obtain a set of virtual-real scene image semantic local feature compression coding vectors, which is expressed as a virtual-real scene image semantic local feature compression coding formula: in, is the square of the matrix F norm, is the dilated convolutional coding, for The corresponding virtual-real scene image semantic local feature compression encoding vectors. It should be understood that the dissociated virtual-real scene image semantic local feature encoding matrices exhibit differential distribution characteristics in spatial dimensions. For example, the material texture features of virtual objects differ significantly from the depth and topology features of the real scene in terms of spatial correlation. By introducing an adaptive dilation rate mechanism for dilated convolution, a virtual-real scene image semantic local feature compression encoding vector with dynamic receptive field adjustment is constructed for different spatial distribution patterns, such as the geometric features of the window frame and the surface features of the sofa model. Its core purpose is to achieve semantic distillation and spatial topology preservation of cross-modal local features. This method compresses redundant background information while maintaining depth continuity in the window region, and performs multi-scale contextual modeling on the surface features of the virtual sofa. Ultimately, a set of virtual-real scene image semantic local feature compression encoding vectors is formed that preserves key semantics while eliminating noise interference. This spatial feature adaptation-based dilated convolution compression ensures precise semantic alignment between the material parameter space of the virtual model and the physical layout space of the real scene, establishing a feature foundation with dimensionally matched and information-density optimized features for subsequent cross-modal recursive interaction.

[0034] Specifically, the user interaction intention semantic feature excitation unit 162 is used to perform information excitation based on deconvolution coding on the user interaction intention semantic embedding coding vector to obtain a user interaction intention semantic feature excitation coding vector, which is expressed as the user interaction intention semantic feature excitation formula: in, is deconvolution coding, and are the trainable deconvolution weight matrix and deconvolution bias vector, respectively. The semantic feature excitation encoding vector for user interaction intention is understandably a dimensionality gap between the discrete symbolic features of natural language descriptions and the high-dimensional continuous scene feature space. When a user issues the command "place the red sofa by the window," the semantic embedding encoding vector for user interaction intention already encodes the color attributes and spatial orientation. However, traditional linear embedding cannot effectively activate the implicit association between the "red" material parameters and the "by the window" wall coordinates. Through the nonlinear mapping effect of the deconvolutional network, the one-dimensional intention vector is expanded into a three-dimensional feature tensor, constructing an excitation feature space encompassing multidimensional parameters such as material reflectance coefficient and spatial coordinate offset. Under the action of the deconvolution kernel, the "red" attribute generates excitation ripples that couple with the material parameter space of the virtual sofa surface. In this way, the originally flattened semantic embedding encoding vector for user interaction intention is reconstructed into a feature cube containing gradient direction information. Different channels correspond to physical parameters such as color perception intensity and spatial orientation weight. This allows the "by the window" orientation description to not only activate the coordinate features of the window area but also capture the orthogonal constraint between the vertical wall plane and the floor plane through the rotational properties of the deconvolution kernel. This dimensionality increase of feature space enables natural language intent to leap from the symbolic level to a parameter space with geometric interpretability, providing a computable feature anchor for the fusion of virtual and real scenes.

[0035] Notably, during cross-modal interaction, differentiated processing is employed for the semantic embedding features of user intent and the semantic features of scene images. Deconvolution is used to expand the dimensionality of intent features and enhance key information, while dilated convolution is used to spatially compress scene features and filter redundant information. This differentiated feature optimization strategy stems from the different functional roles of different modalities in the interaction process: user intent, as a guiding signal for interactive behavior, requires enhanced semantic representation, while scene features, as environmental constraints, focus on core spatial information. Specifically, the deconvolution network overcomes the dimensionality limitations of the original semantic vector through nonlinear mapping, transforming discrete semantics such as "red sofa by the window" into high-dimensional excitation features containing material parameters and coordinate offsets. Scene feature compression adaptively adjusts the receptive field to preserve key geometric constraints such as window boundaries and wall verticality. This targeted strategy, while not favoring either over the other, leverages the complementary relationship between intent guidance and environmental constraints to achieve cross-modal synergy through differentiated adjustments in feature dimensionality and information density, ultimately improving the coupling accuracy between user intent and scene semantics during virtual-reality fusion.

[0036] In an embodiment of the present application, the user interaction feature-level semantic interaction recursive analysis unit 163 is used to: perform structural information optimization expression on each of the virtual-real scene image semantic local feature compression coding vectors in the set of virtual-real scene image semantic local feature compression coding vectors to obtain a set of optimized expression virtual-real scene image semantic local feature compression coding vectors; input the user interaction intention semantic feature excitation coding vector and each of the optimized expression virtual-real scene image semantic local feature compression coding vectors in the set of optimized expression virtual-real scene image semantic local feature compression coding vectors into the first-level feature interaction unit respectively to obtain a set of scene-user interaction intention semantic local interaction coding vectors; input the set of scene-user interaction intention semantic local interaction coding vectors into the second-level feature interaction recursive unit based on the LSTM model to obtain the user interaction intention semantic enhancement coding vector.

[0037] Specifically, each of the virtual-real scene image semantic local feature compression coding vectors in the set of virtual-real scene image semantic local feature compression coding vectors is subjected to structural information optimization expression to obtain a set of optimized expression virtual-real scene image semantic local feature compression coding vectors. The specific processing process of this process is as follows: The individual vectors in the set of the virtual-real scene image semantic local feature compression coding vectors are analyzed in depth, aiming to reveal their inherent topological characteristics through the optimized expression of structural information. As a structural representation of the feature space distribution, the constraint terms It can be regarded as a compact orientation of the entire structure manifold, which not only helps to simplify the data representation, but also preserves the key spatial structure information. And further based on structural invariants To encode the semantic local feature matrix of the virtual-real scene image From composite topological representation to specific information compression characterization representation, that is, Mapping is performed to compress information while retaining as much of the original fine structure feature information as possible: Therefore, the virtual-real scene image semantic local feature encoding matrix With a compact directional representation for manifolds and structural invariants It is expected that the additional topological feature information of the virtual-real scene image can be better expressed while including the compression information in the mapping process. Therefore, the semantic local feature encoding matrix of the virtual-real scene image is firstly The corresponding normalized information compression feature representation vector , calculate its manifold-based compact directional representation Additional field orientation factor for: is the eigenvector of The first eigenvector after the function is activated eigenvalues.

[0038] Then, by the structural invariant Incorporating additional field orientation factors To refine the representation of higher dimensional information based on curvature form integration: Furthermore, the virtual-real scene image semantic local feature compression coding vector is optimized based on the topological subspace additive representation as the compressed information representation under the composite topology: In this way, the virtual-real scene image semantic local feature encoding matrix can be better expressed in the information compression representation process based on additional topological features. The structure information is then extracted, thereby improving the information compression expression effect of the compressed encoding vectors of the semantic local features of the virtual-real scene images. In this way, the resulting set of optimized compressed encoding vectors expressing the semantic local features of the virtual-real scene images can more accurately reflect the actual needs of users and can be presented in a highly consistent manner in the virtual environment.

[0039] Specifically, the user interaction intention semantic feature excitation coding vector and each optimized expression virtual-real scene image semantic local feature compression coding vector in the set of optimized expression virtual-real scene image semantic local feature compression coding vectors are respectively input into the first-level feature interaction unit to obtain a set of scene-user interaction intention semantic local interaction coding vectors, with scene-user interaction intention as the starting point. Figure 1 The level feature interaction formula is expressed as: in, It is the first in the set of compressed coding vectors that optimizes the expression of semantic local features of virtual-real scene images. A compressed encoding vector that optimizes the expression of the semantic local features of the virtual-real scene image, is the feature cascade, is the point product by position, For positional subtraction, and are the trainable interaction weight matrix and the trainable interaction bias vector, respectively. is the Sigmoid activation function, are the first, second, and third in the set of local interaction encoding vectors between scene-user interaction intention semantics. and The local interaction encoding vector between scene-user interaction intention semantics, The set of local interaction encoding vectors between the scene and user interaction intention semantics should be understood. Considering that when the user interaction intention semantic feature excitation encoding vector carries the global semantics of "a red sofa placed by the window," each optimized set of local feature compression encoding vectors for expressing virtual-real scene semantics corresponds to independent physical constraints such as the spatial coordinates and lighting parameters of the window area. Directly fusion of overall features will lead to semantic mismatching due to the heterogeneity of local scene features. By constructing a parallel first-level feature interaction channel and performing independent dot product operations and feature concatenation on the user interaction intention semantic feature excitation encoding vector and each optimized local feature compression encoding vector for expressing virtual-real scene semantics, a correlation strength matrix between the "red material attribute" and the illumination reflectance coefficient of a specific window area can be established. This also allows for the calculation of spatial compatibility parameters between the "window orientation" semantics and the boundary coordinates of each window. In this way, an independent local interaction encoding vector between the scene and the user interaction intention semantics is generated for each window area, in which the high-dimensional feature value reflects the comprehensive matching degree of the area to meet the sofa material reflection requirements and the spatial layout constraints. For example, the southwest-facing window forms a high matching value due to the sunlight angle and the spectral reflection characteristics of the red material, while the north-facing small window produces a low matching feature due to the insufficient spatial size, thereby providing a local interaction data base with physical interpretability for subsequent recursive screening.

[0040] Specifically, the set of local interaction encoding vectors between the scene and user interaction intention semantics is input into the secondary feature interaction recursive unit based on the LSTM model to obtain the user interaction intention semantic enhancement encoding vector, which is expressed as the secondary feature interaction recursive formula: in, for Model processing, The semantically enhanced encoding vector for the user interaction intention is represented by the user interaction intention semantics. It should be understood that after the first-level interaction unit generates a set of local interaction encoding vectors between scene-user interaction intention semantics, considering the non-independent spatial distribution of high-matching features for southwest-facing windows and low-matching parameters for north-facing small windows, sequential modeling is required to capture the decision logic chain for "preferring the best-lit area." The LSTM's hidden state transfer mechanism dynamically updates the window selection weights as each local interaction encoding vector between scene-user interaction intention semantics is iteratively processed. This allows for a cross-modal parameter association model for "sofa material reflectivity - sunlight angle - room dimensions" to be established, eliminating the influence of local optimal solutions on the global decision. In this way, the input gate controls the influence of current window parameters on the final decision, while the output gate integrates historical information to generate a semantically enhanced encoding vector for user interaction intention that includes the spatial layout optimization level. This ultimately maps the "window placement" intention to the south-facing bay window area, while eliminating pseudo-window matches obscured by curtains, achieving a virtual-reality fusion decision that is ergonomically and optically responsive.

[0041] Figure 5 FIG is a block diagram of a virtual interactive display module in a display system based on augmented reality technology according to an embodiment of the present application. Figure 5 As shown, in an embodiment of the present application, the virtual interaction display module 170 includes: a user interaction intention type label determination unit 171, which is used to determine the type label of the user interaction intention based on the user interaction intention semantic enhancement coding vector; and a virtual interaction display screen generation unit 172, which is used to generate a virtual interaction display screen based on the type label of the user interaction intention.

[0042] Specifically, the user interaction intention type label determination unit 171 is configured to determine the type label of the user interaction intention based on the semantically enhanced encoding vector of the user interaction intention. It should be understood that when the semantically enhanced encoding vector of the user interaction intention carries the complex information "place the red sofa by the window," the system needs to map this high-dimensional semantic information into actionable interaction instruction types (such as object placement, attribute modification, or spatial adjustment), while also ensuring that the virtual rendering strictly adheres to the physical constraints of the scene. Therefore, the type label of the user interaction intention is further determined based on the semantically enhanced encoding vector of the user interaction intention. In particular, the type label of the user interaction intention can be obtained using a classifier-based user interaction intention type determinist. Specifically, after inputting the semantically embedded encoding vector of the user interaction intention, a trained classifier is used to analyze and classify these vectors. Classifier design typically relies on machine learning or deep learning techniques, such as support vector machines (SVMs), random forests, or neural networks. Deep learning methods, due to their powerful feature extraction and nonlinear mapping capabilities, excel in handling complex and variable user intent recognition tasks. In particular, the use of convolutional neural networks (CNNs) or recurrent neural networks (RNNs), as well as their variants such as long short-term memory (LSTM) and Transformer architectures, can effectively capture long-term dependencies in input sequences, thereby improving classification accuracy. Training a classifier requires a large, labeled dataset containing various types of user interaction intent and their corresponding correct type labels. Using these datasets for supervised learning, the classifier learns the mapping between semantic embeddings and specific type labels. During training, the model continuously adjusts its internal parameters to minimize the error between predicted results and actual labels. This typically involves complex optimization algorithms, such as gradient descent and its variants, to update weights to minimize the loss function. Once the classifier is trained, it can be applied to user interaction intents. Based on its previously learned knowledge, the classifier evaluates the semantic embeddings of the input user interaction intent and calculates a probability distribution for each possible type label. Ultimately, the type with the highest probability is selected as the predicted label for that user interaction intent.

[0043] Specifically, the virtual interactive display screen generation unit 172 is configured to generate a virtual interactive display screen based on the type label of the user's interactive intent. It should be understood that generating a virtual interactive display screen based on the type label of the user's interactive intent enhances the realism and interactivity of the user experience. First, this process ensures that virtual objects accurately reflect the user's intent, allowing the system to not only understand the user's expressed content but also visually represent these intentions. For example, when a user requests to "place a red sofa by the window," the system not only recognizes this as a furniture placement operation but also accurately places the virtual sofa that meets the description in the specified location through detailed spatial analysis and object matching. This accuracy allows users to feel that their instructions are fully understood and executed, thereby enhancing their trust in the system. Furthermore, the process of generating the virtual interactive display screen includes fine-tuning of virtual objects, such as adapting attributes such as size, proportion, color, and material. This not only ensures harmony between the virtual objects and the real environment but also enhances the realism and immersion of the overall scene. For example, when placing a sofa in a spacious living room, the system automatically adjusts the sofa's proportions based on the actual size of the room, ensuring a harmonious and natural appearance. Furthermore, by choosing the right materials and colors, the virtual sofa appears to exist in the environment. Its surface reflects light in a way that matches its surroundings, further enhancing visual continuity. Furthermore, the virtual interactive display dynamically responds to the user's continuous actions and environmental changes. As the user's perspective changes or the ambient lighting conditions change, the system updates the virtual object's representation in real time, maintaining a consistent and realistic visual experience. For example, when sunlight shines through a window, the virtual sofa casts corresponding shadows and moves or deforms according to the changing angle of the light source, as if it were actually there. This instant feedback mechanism not only increases the fun of interaction but also makes the entire experience more fluid and natural. Finally, by combining advanced rendering technology and animation effects, the virtual interactive display offers a richer and more diverse visual experience. From realistic lighting and shadow effects to detailed textures, the virtual objects appear to be lifelike. Combined with carefully designed animation sequences, such as opening a drawer or rotating a chair, users feel as if they are interacting with real objects, rather than simply viewing static images. This detailed approach not only enhances visual quality but also creates a deeper and more immersive experience for users.

[0044] In summary, the display system 100 based on augmented reality technology according to the embodiment of the present application is explained. It first obtains virtual scene images and real scene images and performs semantic-level feature fusion of the scene images, and simultaneously obtains the natural language description of the user's interaction intention for semantic understanding. Then, by extracting the semantic-level feature encoding representation of the virtual and real scenes, combined with the multi-granularity semantic embedding of the natural language description, the recursive cross-modal attention mechanism is used to strengthen the user intention representation layer by layer, and finally generates an interactive screen that is highly adapted to the scene context. In this way, through the cross-modal hierarchical interaction recursive mechanism, the limitations of traditional single-modal intent recognition are broken through, and the deep coupling of dynamic scene semantics and user language intention is achieved, providing a new technical path for highly immersive AR interaction and enhancing user participation.

[0045] As described above, the augmented reality technology-based display system 100 according to the embodiments of the present application can be implemented in various terminal devices. In one example, the augmented reality technology-based display system 100 can be integrated into the terminal device as a software module and / or hardware module. For example, the augmented reality technology-based display system 100 can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the augmented reality technology-based display system 100 can also be one of the many hardware modules of the terminal device.

[0046] Alternatively, in another example, the display system 100 based on augmented reality technology and the terminal device may also be separate devices, and the display system 100 based on augmented reality technology may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0047] Figure 6 FIG. 1 is a flow chart of a display method based on augmented reality technology according to an embodiment of the present application. Figure 6 As shown, according to the display method based on augmented reality technology of an embodiment of the present application, it includes: S110, obtaining a virtual scene image; S120, obtaining a real scene image; S130, obtaining a natural language description of the user interaction intention; S140, performing scene image semantic-level feature fusion on the virtual scene image and the real scene image to obtain a virtual-real scene image semantic joint coding representation; S150, performing semantic understanding on the natural language description of the user interaction intention to obtain a user interaction intention semantic embedded coding representation; S160, using the virtual-real scene image semantic joint coding representation as auxiliary information, performing semantic enhancement on the user interaction intention semantic embedded coding representation based on cross-modal information interaction to obtain a user interaction intention semantic enhanced coding representation; S170, generating a virtual interaction display screen based on the user interaction intention semantic enhanced coding representation.

[0048] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned display method based on augmented reality technology have been referred to above. Figures 1 to 5 The description of the display system based on augmented reality technology has been introduced in detail, and therefore, its repeated description will be omitted.

[0049] In summary, the display method based on augmented reality technology according to the embodiment of the present application is explained. It first obtains virtual scene images and real scene images and performs semantic-level feature fusion of scene images, and simultaneously obtains the natural language description of the user's interaction intention for semantic understanding. Then, by extracting the semantic-level feature encoding representation of the virtual and real scenes, combined with the multi-granularity semantic embedding of the natural language description, the recursive cross-modal attention mechanism is used to strengthen the user intention representation layer by layer, and finally generates an interactive screen that is highly adapted to the scene context. In this way, through the cross-modal hierarchical interaction recursive mechanism, the limitations of traditional single-modal intent recognition are broken through, and the deep coupling of dynamic scene semantics and user language intention is achieved, which provides a new technical path for highly immersive AR interaction and enhances the user's sense of participation.

[0050] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0051] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0052] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0053] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0054] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0055] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Second-order terms are used to indicate names and do not imply any particular order.

Claims

1. A display system based on augmented reality technology, characterized in that: include: A virtual scene image acquisition module, used for acquiring a virtual scene image; A real scene image acquisition module, used to acquire real scene images; Interaction intention text collection module, used to obtain the natural language description of user interaction intention; A scene semantic joint encoding module, configured to perform scene image semantic level feature fusion on the virtual scene image and the real scene image to obtain a virtual-real scene image semantic joint encoding representation; An interaction intention semantic understanding module is used to perform semantic understanding on the natural language description of the user interaction intention to obtain a semantic embedding coding representation of the user interaction intention; An interaction intention semantic enhancement module is configured to use the virtual-real scene image semantic joint coding representation as auxiliary information to perform semantic enhancement on the user interaction intention semantic embedding coding representation based on cross-modal information interaction to obtain a user interaction intention semantic enhancement coding representation; The virtual interaction display module is used to generate a virtual interaction display screen based on the semantic enhancement coding representation of the user interaction intention.

2. The display system based on augmented reality technology according to claim 1, characterized in that: The scene semantic joint encoding module includes: A scene image semantic level feature extraction unit, configured to extract scene image semantic level features from the virtual scene image and the real scene image to obtain a virtual scene image feature coding map and a real scene image feature coding map; A virtual-real scene image semantic feature fusion unit is used to fuse the virtual scene image feature coding map and the real scene image feature coding map to obtain a virtual-real scene image semantic joint coding map as the virtual-real scene image semantic joint coding representation.

3. The display system based on augmented reality technology according to claim 2, characterized in that: The scene image semantic-level feature extraction unit is used to: respectively pass the virtual scene image and the real scene image through a scene semantic feature extractor based on a pyramid network to obtain the virtual scene image feature coding map and the real scene image feature coding map.

4. The display system based on augmented reality technology according to claim 3, characterized in that: The interaction intention semantic understanding module is used to: perform semantic understanding based on embedded coding on the natural language description of the user interaction intention to obtain a user interaction intention semantic embedded coding vector as the user interaction intention semantic embedded coding representation.

5. The display system based on augmented reality technology according to claim 4, characterized in that: The interaction intention semantic enhancement module includes: a virtual-real scene image semantic local feature compression unit, configured to perform information compression based on feature dissociation analysis on the virtual-real scene image semantic joint coding map to obtain a set of virtual-real scene image semantic local feature compression coding vectors; A user interaction intention semantic feature excitation unit is used to perform information excitation based on deconvolution coding on the user interaction intention semantic embedding coding vector to obtain a user interaction intention semantic feature excitation coding vector; The user interaction feature-level semantic interaction recursive analysis unit is used to perform feature-level semantic interaction recursive analysis on the set of semantic local feature compression coding vectors of the virtual-real scene image and the user interaction intention semantic feature excitation coding vector to obtain a user interaction intention semantic enhancement coding vector as the user interaction intention semantic enhancement coding representation.

6. The display system based on augmented reality technology according to claim 5, characterized in that: The virtual-real scene image semantic local feature compression unit is used to: Performing feature dissociation on the virtual-real scene image semantic joint coding map along the channel dimension to obtain a set of virtual-real scene image semantic local feature coding matrices; In combination with the feature distribution spatial characteristics of each virtual-real scene image semantic local feature coding matrix in the set of virtual-real scene image semantic local feature coding matrices, information compression based on void convolution coding is performed on each virtual-real scene image semantic local feature coding matrix to obtain a set of virtual-real scene image semantic local feature compression coding vectors.

7. The display system based on augmented reality technology according to claim 6, characterized in that: The user interaction feature-level semantic interaction recursive analysis unit is used to: Performing structural information optimization expression on each of the virtual-real scene image semantic local feature compression coding vectors in the set of virtual-real scene image semantic local feature compression coding vectors to obtain a set of optimized expression virtual-real scene image semantic local feature compression coding vectors; Inputting the user interaction intention semantic feature excitation coding vector and each optimized expression virtual-real scene image semantic local feature compression coding vector in the set of optimized expression virtual-real scene image semantic local feature compression coding vectors into the first-level feature interaction unit respectively to obtain a set of scene-user interaction intention semantic local interaction coding vectors; The set of local interaction encoding vectors between the scene-user interaction intention semantics is input into the secondary feature interaction recursive unit based on the LSTM model to obtain the user interaction intention semantic enhancement encoding vector.

8. The display system based on augmented reality technology according to claim 7, characterized in that: The virtual interactive display module includes: A user interaction intention type label determination unit, configured to determine a type label of the user interaction intention based on the user interaction intention semantic enhancement encoding vector; A virtual interaction display screen generating unit is configured to generate a virtual interaction display screen based on the type label of the user interaction intention.

9. The display system based on augmented reality technology according to claim 8, characterized in that: The user interaction intention type label determination unit includes: passing the user interaction intention semantic enhancement encoding vector through a user interaction intention type determiner based on a classifier to obtain a type label of the user interaction intention.

10. A display method based on augmented reality technology, characterized in that: include: Acquire a virtual scene image; Acquire real scene images; Obtain natural language description of user interaction intentions; Performing scene image semantic-level feature fusion on the virtual scene image and the real scene image to obtain a virtual-real scene image semantic joint coding representation; Performing semantic understanding on the natural language description of the user interaction intention to obtain a semantic embedding coding representation of the user interaction intention; Using the virtual-real scene image semantic joint coding representation as auxiliary information, the semantic embedded coding representation of the user interaction intention is semantically enhanced based on cross-modal information interaction to obtain a semantically enhanced coding representation of the user interaction intention; Based on the semantically enhanced coded representation of the user interaction intention, a virtual interaction display screen is generated.

Images