Panoramic interactive display method and system based on AI multi-modal fusion

Abstract

According to the panoramic interaction display method and system based on AI multi-modal fusion provided by the invention, efficient and accurate multi-modal fusion is realized by integrating the visual and audio information of the current panoramic material and combining multiple pieces of panoramic interaction hotspot information, a panoramic interaction model with rich interactivity is generated, and the interaction effect of the panoramic interaction model is improved. And hotspot interaction space features are further extracted, so that panoramic display can automatically adjust a visual angle according to a user demand or a preset situation and trigger a corresponding commentary audio or interaction animation, immersion and interactivity of panoramic display are remarkably improved, user experience is optimized, seamless connection of audio playing and user interaction operation is realized, and user experience is improved. Therefore, the dynamic display and interaction of the panoramic content can be realized conveniently.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a panoramic interactive display method and system based on AI multimodal fusion. Background Art

[0002] With the rapid development of virtual reality (VR), augmented reality (AR) and panoramic shooting technology, panoramic content has been widely used in entertainment, education, tourism and other fields. Traditional panoramic display methods often focus on providing a 360-degree visual experience, however, there are still many deficiencies in interactivity and immersion. Users can usually only passively browse panoramic images and cannot deeply interact with the elements in them, which greatly limits the expressiveness of panoramic content and user experience.

[0003] In order to improve the interactivity and immersion of panoramic displays, related technologies have begun to try to integrate information from multiple modes (such as vision, audio, etc.) into panoramic displays. However, existing multimodal fusion technologies often have problems such as poor fusion effect, insensitive interactive response, and unsmooth user experience. For example, some methods simply overlay audio information with panoramic images, but fail to achieve accurate matching and dynamic interaction between audio and visual elements; other methods may lead to high computational costs due to complex processing procedures, making them difficult to promote in practical applications. Summary of the invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a panoramic interactive display method based on AI multimodal fusion, the method comprising: Acquire first visual and audio combination data of the current panoramic material, and acquire multiple panoramic interaction hotspot information of the current panoramic material; Performing multimodal fusion according to the first visual and audio combination data and the plurality of panoramic interaction hotspot information to generate a panoramic interaction model of the current panoramic material; Extracting interactive spatial features from the panoramic interaction model of the current panoramic material to generate hotspot interactive spatial features of the current panoramic material; Matching the hotspot interactive spatial features of the current panoramic material with corresponding commentary audio or interactive animation to generate a triggerable interactive hot zone of the current panoramic material; Based on the hotspot interactive space characteristics of the current panoramic material and the triggerable interactive hot zones of the current panoramic material, according to the audio playback progress or user interactive operations, the viewing angle is automatically adjusted to the corresponding panoramic interactive hotspot, and the first interactive processing result of the corresponding commentary audio or interactive animation is triggered.

[0005] On the other hand, an embodiment of the present invention also provides a panoramic interactive display system based on AI multimodal fusion, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0006] Based on the above aspects, the embodiment of the present application realizes efficient and accurate multimodal fusion by integrating the visual and audio information of the current panoramic material and combining multiple panoramic interactive hotspot information. It not only generates a panoramic interaction model with rich interactivity, but also further extracts the hotspot interaction space characteristics, so that the panoramic display can automatically adjust the viewing angle according to user needs or preset scenarios and trigger corresponding commentary audio or interactive animation, which significantly improves the immersion and interactivity of the panoramic display, optimizes the user experience, and realizes seamless connection between audio playback and user interactive operations, thereby facilitating the dynamic display and interaction of panoramic content. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a schematic diagram of the execution flow of the panoramic interactive display method based on AI multimodal fusion provided in an embodiment of the present invention.

[0008] Figure 2 It is a schematic diagram of the hardware architecture of a panoramic interactive display system based on AI multimodal fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 It is a flow chart of a panoramic interactive display method based on AI multimodal fusion provided by an embodiment of the present invention. The panoramic interactive display method based on AI multimodal fusion is introduced in detail below.

[0010] Step S110, obtaining first visual and audio combination data of the current panoramic material, and obtaining multiple panoramic interactive hotspot information of the current panoramic material.

[0011] In this embodiment, it is assumed that the server is processing a panoramic material about a historical monument, such as a panoramic display of an old castle. This panoramic material contains visual information, such as the architectural appearance, internal structure, and surrounding environment of the castle. These picture information are stored in a specific format (such as a high-definition image format) in the storage system of the server. This is the first visual data part. At the same time, the audio data related thereto may include environmental sound effects, such as wind, birdsong, and some narrations describing the history and architectural features of the castle. These audio data exist in a suitable audio coding format, and the visual data and audio data are combined to form the first visual audio combined data.

[0012] In terms of obtaining panoramic interactive hotspot information, there may be multiple panoramic interactive hotspot information for this castle panoramic material. For example, the castle gate may be a hotspot. When the user pays attention to this area, it can trigger the audio or interactive animation related to the construction time and craftsmanship characteristics of the gate; the tower in the castle is also a hotspot, and the interactive triggers may be related to the military defense function of the tower, the stories that happened here in history, etc.; the garden in the castle is also a hotspot, and the corresponding interactive content may be the types of plants planted in the garden, the role of the garden in the life of the castle at that time, etc. The server identifies these possible panoramic interactive hotspot information by pre-marking the panoramic material or analyzing the algorithm, and extracts them from the panoramic material.

[0013] Step S120, performing multimodal fusion based on the first visual and audio combination data and the plurality of panoramic interaction hotspot information to generate a panoramic interaction model of the current panoramic material.

[0014] In this embodiment, the first visual audio combination data and the plurality of panoramic interactive hotspot information obtained previously are used as model building elements for building a panoramic interactive model. For the example of a castle, the castle appearance image and the related narration audio in the visual audio combination data, as well as the panoramic interactive hotspot information such as the gate, tower, garden, etc. are all model building elements.

[0015] When determining the interaction and connection between model building elements, take the castle gate as an example. There is a correlation between the visual image of the gate and the audio describing the gate. For example, when the audio mentions the material of the gate, the material texture of the gate in the visual image should be focused on at this time. This is an interaction between vision and audio. This relationship is determined as a connection relationship in the panoramic interaction model. Similarly, there is a connection between the gate hotspot and the overall visual image of the castle, because the gate is part of the castle, and the visual relationship from the whole to the part needs to be reflected in the model.

[0016] In terms of feature extraction and generation of feature vectors for model building elements, if the index tag of the model building element is the first visual audio combination data, such as a visual picture of the internal structure of the castle hall and the corresponding audio commentary in the castle panoramic material. The server performs reciprocating cyclic temporal coding on this interactive path. Assuming that the duration of this video is 30 seconds, the server encodes the changes in the visual picture and the rhythm of the audio within this time period at a certain time interval (such as 1 second as a time unit) to generate the cyclic temporal features of the interactive path. Then determine the index positioning features of this interactive path, such as the starting position of this picture and audio in the entire panoramic material, the scene type (castle internal structure scene), etc., merge the cyclic temporal features and index positioning features, generate path features, and then allocate focus weights to this path feature. For example, a higher weight is given to the important building structure in the center of the hall in the picture and the part of the audio that focuses on describing this structure, and finally generate the feature vector of the element of this interactive path.

[0017] If the index label of the model building element is a visual picture corresponding to a priori panoramic material, such as a visual picture in a panoramic material similar to a castle that has been processed before, the server can use artificial intelligence (AI) algorithms to encode the semantic features of the visual picture of the material, such as identifying semantic information such as architectural style and architectural elements in the picture, generating semantic features of the visual picture, and then converting these semantic features, such as converting the semantic features into vector form, thereby generating an element feature vector of the visual picture of the material.

[0018] For the interactive hot zones corresponding to the prior panoramic material, such as the interactive hot zones of a specific room in the previous castle material, the server searches the interactive hot zone parameter index library based on this interactive hot zone. This index library contains the mapping relationship between each interactive hot zone and each interactive hot zone feature. For example, the interactive hot zone of this room corresponds to the interactive hot zone features such as interior decoration style and historical use. The server uses these searched interactive hot zone features as the element feature vectors of the interactive hot zone corresponding to the prior panoramic material.

[0019] When the index tag of the model construction element is the narrative object corresponding to the prior panoramic material, such as a historical figure narrated in the previous castle material, the server searches the narrative object parameter index library based on this narrative object. This library contains the mapping relationship between each narrative object and each narrative object feature, such as the life story of this historical figure, his relationship with the castle, etc. The searched narrative object features are used as the element feature vector of the narrative object corresponding to the prior panoramic material.

[0020] Step S130, extracting interactive space features from the panoramic interaction model of the current panoramic material to generate hotspot interactive space features of the current panoramic material.

[0021] In this embodiment, the server first performs an architecture analysis on the panoramic interaction model of the castle panoramic material to identify its various components. The relevant part of the visual and audio combination data includes the visual images of each scene of the castle and the corresponding audio, and the relevant part of the panoramic interactive hotspot information includes the relevant information of hotspots such as gates, towers, and gardens. The connection structure between them is, for example, the association between the hotspot position in the visual image and the corresponding audio commentary content.

[0022] Based on these components, the server generates a structural decomposition strategy for presenting the internal structure of the model. According to the function, for example, the parts of the visual and audio related to the description of the building structure are classified into one category, and the parts related to the historical story are classified into another category; according to the data flow, the layers are layered from the original panoramic material data input to the final interactive hotspot related data output; according to the type of construction elements, the visual elements, audio elements, hotspot information elements, etc. are divided into different areas.

[0023] According to this structural decomposition strategy, the interaction paths between the various elements in the model structure after the panoramic interaction model is decomposed are marked. The connection path from the first visual audio combination data to the panoramic interactive hotspot information, such as the connection path from the visual picture of the overall appearance of the castle and the related audio to the hotspot of the gate, marks the location of the gate in the visual picture and the time point when the gate is mentioned in the audio; the association path between different panoramic interactive hotspot information, such as the possible architectural layout association path between the gate hotspot and the tower hotspot; other logical connection paths between model construction elements, such as the logical connection path between a certain audio describing the overall defense system of the castle and multiple hotspots such as the gate and the tower.

[0024] Then, the elements in the panoramic interaction model are grouped according to the marked interaction paths. For each element group, the interaction relationship between the elements in the group and the interaction boundary with the elements in other groups are analyzed. Taking the element group related to the gate hotspot as an example, the elements in the group include the visual image of the gate, the audio describing the gate, and the historical events related to the gate. The interaction relationship between them is the audio's interpretation of the visual image and the association between the historical events and the gate. The interaction boundary between this element group and the element group related to the tower hotspot lies in the connection relationship in the architectural structure.

[0025] Within each interactive space, feature extraction is performed on the model elements. For visual elements, features are extracted from the visual composition of the castle gate, such as the shape, color, and material texture of the gate, as well as the distribution pattern of visual elements, such as the arrangement of decorative elements on the gate. For audio elements, features are extracted from the frequency characteristics of the audio related to the gate, such as the low bell sound that may represent an ancient and solemn feeling, rhythm characteristics, such as the rhythm changes in the audio when the gate is opened, and the synchronization relationship with the visual elements, such as when the audio mentions a certain part of the gate, the visual picture just focuses on that part. For hotspot information elements, features are obtained from the triggering conditions of the gate hotspot, such as the user's line of sight focusing on the gate area or the mouse clicking on the gate area, and the interactive actions associated with the hotspot, such as triggering the audio of the detailed introduction of the gate or the interactive animation showing the construction process of the gate.

[0026] Based on the predefined feature evaluation criteria, each feature in the initial feature set within each interactive space is evaluated. This feature evaluation criterion is determined based on the importance of the feature to the hotspot interaction function of the panoramic interactive model, the correlation between features, and the representativeness of the feature in the overall interaction logic. For example, the material texture feature of the gate is important for the hotspot interaction function of showing the construction process of the gate, while some insignificant stain textures on the gate may have a lower correlation and will be evaluated as lower importance. Screening and optimization are performed based on the evaluation results to generate a screened and optimized target feature set.

[0027] The target feature set within each interactive space is spatially mapped according to the predefined spatial mapping rules, based on the distance relationship between the feature and the key elements in the interactive space, and the spatial association between the feature and other features. For example, the construction age feature of the gate and the material feature of the gate may have a closer relationship in the spatial mapping because they are both related to the overall properties of the gate. Finally, the feature representations after spatial mapping within each interactive space are integrated to construct the hotspot interactive space features that describe the current castle panoramic material.

[0028] Step S140, matching the hotspot interactive spatial features of the current panoramic material with corresponding commentary audio or interactive animation to generate a triggerable interactive hot zone of the current panoramic material.

[0029] In this embodiment, the server identifies different feature dimension information in the hotspot interactive space features of the castle panoramic material, and builds a matching relationship analysis framework for analyzing the hotspot interactive space features and the commentary audio or interactive animation based on this information.

[0030] For audio commentary, for example, there is an audio commentary about the historical changes of a castle. In terms of content structure, the hierarchical structure of the audio commentary narrative is analyzed, which may start with the origin of the castle's construction, and then the development in different historical periods. The distribution of key content may be in the parts corresponding to important historical events; in terms of semantic information, the concepts and descriptions related to the panoramic materials of the castle contained in the audio commentary are excavated, such as the change of the castle's owner, the role of the castle in the war, etc.; in terms of emotional tendency, the overall emotional atmosphere conveyed by the audio is judged, such as describing the glorious period of the castle is a positive emotion, and describing the destruction of the war is a heavy emotion; in terms of rhythm and rhyme, the rhythm speed and rhythm rules of the audio commentary are analyzed, for example, the rhythm is accelerated when describing the tense war scene.

[0031] For interactive animation, suppose there is an interactive animation showing the process of castle building. In terms of the composition of the visual elements of the animation, determine the graphics contained in the interactive animation, such as the shape of the building, the color, such as the color of the stone, and the objects, such as the building tools; in terms of action logic, determine the movement rules and logical relationships of the various elements in the interactive animation, such as the route of the workers carrying the stone, the order of stone stacking, etc.; in terms of the interactive response mechanism, determine the interactive operations that the interactive animation responds to, such as the user clicking on a certain construction stage to view more detailed instructions; in terms of visual rhythm, determine the speed of the interactive animation's screen switching and the rhythm of the element movement, such as the slow rhythm when stacking stones.

[0032] Based on the matching relationship analysis framework and the feature set of the narration audio and interactive animation obtained by analysis, a set of matching association index system is established, including content matching index, interaction logic matching index, and visual and audio coordination index. The content matching index is used to measure the matching degree between the semantic information in the hotspot interactive space features and the content expressed by the narration audio and the interactive animation, such as whether the semantic information of the castle construction age in the hotspot interactive space features of the castle panorama material matches the construction age mentioned in the narration audio and the timeline of the construction process in the interactive animation; the interaction logic matching index is used to evaluate the degree of fit between the interaction trigger conditions in the hotspot interactive space features and the interactive response mechanism of the interactive animation and the guiding logic of the narration audio in the interactive process, such as whether the trigger conditions of the gate hotspot in the hotspot interactive space features are consistent with the trigger logic of the gate construction process in the interactive animation; the visual and audio coordination index is used to determine the coordination between the visual and audio association features in the hotspot interactive space features and the visual and audio in the narration audio and interactive animation, such as whether the light and dark changes in the visual picture of a scene in the castle panorama material are coordinated with the atmosphere of the narration audio.

[0033] Based on the matching association index system, the hotspot interactive space features are matched and evaluated with the commentary audio and interactive animation. For each index, the matching score between the hotspot interactive space features and the commentary audio or interactive animation on the index is calculated according to the evaluation method defined for the index, and a preliminary matching result set between the hotspot interactive space features and the commentary audio and interactive animation is obtained. This result set contains the scores of each matching pair on each index and the comprehensive score. The preliminary matching result set is filtered according to the set filtering criteria, for example, matching pairs with a comprehensive score higher than a certain threshold are filtered out. For the filtered matching pairs, they are sorted according to the comprehensive score to obtain a sorted matching result set.

[0034] Based on the sorted matching result set, the relationship between the hotspot interactive space features and the commentary audio or interactive animation in each matching pair is analyzed to determine the scope definition rule set for the triggerable interactive hot zone. For example, if the hotspot interactive space features of the castle gate in a matching pair have a high degree of match with the commentary audio and interactive animation about the gate, then the determined scope definition rule may be that the related commentary audio or interactive animation is triggered when the user's line of sight approaches the gate area or clicks the gate area. Based on the determined scope definition rule set for the triggerable interactive hot zone, the triggerable interactive hot zone is marked in the hotspot interactive space features of the current castle panoramic material, thereby generating the triggerable interactive hot zone of the current panoramic material.

[0035] Step S150, based on the hotspot interactive space characteristics of the current panoramic material and the triggerable interactive hot zone of the current panoramic material, according to the audio playback progress or user interaction operation, automatically adjust the viewing angle to the corresponding panoramic interaction hotspot, and trigger the first interaction processing result of the corresponding commentary audio or interactive animation.

[0036] In this embodiment, the server uses a pre-built real-time interactive monitoring module to continuously track the current progress information of audio playback and the real-time status information of user interactive operations. For the audio playback of the castle panoramic material, the current progress information of the audio playback includes the current playback timestamp, such as 3 minutes have been played, the current playback paragraph identifier, such as the audio of the castle's internal structure is being played, and the current playback content overview, such as describing the decoration of the castle hall; the real-time status information of the user interactive operation includes the user operation type, such as the user's mouse movement and click operation, the operation trigger timestamp, such as the click operation occurred in the 5th minute, the operation target identifier, such as the castle garden area is clicked, and the operation context description, such as the user just viewed the tower part of the castle.

[0037] The pre-built hotspot positioning and prediction unit receives the audio playback progress information and user interaction operation status information from the real-time interaction monitoring module as input information. The user's next point of interest or interaction intention is predicted by analyzing these input information. The hotspot positioning and prediction unit maintains a user behavior pattern library, which learns the behavior preferences and interaction patterns of different users in various situations based on historical interaction data. For example, for a castle panoramic material, if historical data shows that many users are interested in the garden after viewing the tower, when the current user views the tower, the hotspot positioning and prediction unit uses this user behavior pattern library to match the current user's behavior characteristics and predicts that the panoramic interaction hotspot that the current user is concerned about may be the garden. At the same time, combined with the audio playback progress, the hotspot area related to the current playback content is inferred. For example, the current audio is describing the outdoor part of the castle, which also increases the possibility that the garden will become the user's focus. The output is a set of predicted hotspot areas (such as gardens, castle walls, etc.) and the corresponding prediction confidence scores (the garden has a higher confidence score).

[0038] The pre-built interactive decision engine receives the predicted hotspot area set and confidence score output by the hotspot positioning and prediction unit, as well as the hotspot interaction spatial characteristics and triggerable interactive hot zone information of the current castle panoramic material, and outputs the interactive instruction set after the decision. The interactive instruction set includes the target hotspot identification, such as determining the target hotspot as the garden, the interaction type, such as triggering the audio commentary about the plant species in the garden, the perspective adjustment parameters, such as the parameters required to smoothly adjust the perspective from the current tower perspective to the garden perspective, and the execution time, such as before the audio plays the next description about the garden. This interactive decision engine defines a decision rule set, based on which the next panoramic interactive hotspot to be displayed is determined, as well as the interaction type that triggers the commentary audio and / or interactive animation, and the corresponding perspective adjustment scheme is calculated. The perspective adjustment scheme includes the target perspective coordinates (the coordinate position of the garden in the panoramic material), the adjustment speed (such as how many degrees to rotate per second), and the smoothness parameter (to ensure smooth perspective switching).

[0039] A pre-built automatic perspective adjustment module is used to receive the perspective adjustment parameters and target hotspot identifiers in the interactive instruction set output by the interactive decision engine, and a smooth transition path from the current perspective to the target perspective is calculated based on the perspective adjustment parameters and the target hotspot identifier. At the same time, the perspective adjustment mechanism of the panoramic display system is controlled to perform perspective switching according to the calculated smooth transition path and perspective adjustment parameters, so that the user's perspective is aligned with the target castle panoramic interactive hotspot after the decision, that is, the garden area.

[0040] The pre-built content trigger and synchronization control unit receives the target hotspot identifier, interaction type, and execution timing in the interactive instruction set output by the interactive decision engine, and receives the view adjustment completion timestamp output by the view automatic adjustment module for analysis, and outputs the content display status information after the trigger. For the audio commentary, the content trigger and synchronization control unit controls the audio playback system to start playing from the specified position, such as starting from the audio part about the types of garden plants; for interactive animations, the content trigger and synchronization control unit starts the animation rendering and display process, such as displaying the interactive animation of plant growth in the garden.

[0041] After the content triggering and synchronization control unit executes the content display, the pre-built user feedback collection and analysis module collects the user's immediate feedback data, uses the machine learning algorithm to analyze the feedback data, evaluates the user's satisfaction and interest in the current interactive processing results, and outputs a user feedback analysis report. The immediate feedback data includes the user's reaction time, such as how long it takes for the user to perform the next operation after the content is displayed, interactive operations, such as whether the user continues to view other parts of the garden or switches to other hot spots, emotional expressions, such as whether the user has positive or negative feedback actions on the interface, and comments or ratings. The user feedback analysis report contains user satisfaction scores, interest point distribution, and improvement suggestions. For example, if the user satisfaction score is low, the interest point distribution shows that the user is particularly interested in a plant in the garden but the display content is insufficient. The improvement suggestion may be to add detailed explanation audio or interactive animation about this plant.

[0042] The pre-built adaptive optimization loop mechanism is used to receive the user feedback analysis report output by the user feedback collection and analysis module, combined with the interaction data continuously collected by the real-time interaction monitoring module, and the reinforcement learning algorithm is used to optimize the prediction model of the hotspot positioning and prediction unit, the decision rule set of the interactive decision engine, and the synchronization control logic of the content triggering and synchronization control unit. For example, if it is found that many users have a poor interactive experience with the garden, the adaptive optimization loop mechanism can adjust the prediction strategy of the hotspot positioning and prediction unit for the garden hotspot, optimize the decision rules of the interactive decision engine for garden-related interactions, and improve the synchronization control logic of the content triggering and synchronization control unit for the display of garden content, thereby improving the overall interactive experience.

[0043] Based on the above steps, the embodiment of the present application realizes efficient and accurate multimodal fusion by integrating the visual and audio information of the current panoramic material and combining multiple panoramic interactive hotspot information. It not only generates a panoramic interaction model with rich interactivity, but also further extracts the hotspot interaction space features, so that the panoramic display can automatically adjust the viewing angle according to user needs or preset scenarios and trigger corresponding commentary audio or interactive animation, significantly improving the immersion and interactivity of the panoramic display, optimizing the user experience, and realizing seamless connection between audio playback and user interactive operations, thereby facilitating the dynamic display and interaction of panoramic content.

[0044] In a possible implementation, step S120 includes: Step S121: Use the first visual and audio combination data and the plurality of panoramic interaction hotspot information as model building elements of the panoramic interaction model.

[0045] Step S122, determining the connection relationship between the model construction elements of the panoramic interaction model according to the interaction relationship between the model construction elements.

[0046] Step S123, performing feature extraction on the model construction elements to generate feature feature vectors of the model construction elements of the panoramic interaction model.

[0047] In this embodiment, the panoramic visual data of the castle, such as the castle exterior, interior rooms, battlements on the castle wall, etc., and the accompanying audio data, such as wind sound, bird song, and narration about the castle history, together constitute the first visual audio combination data. The panoramic interactive hotspot information includes the castle gate, tower, inner garden and other areas. These elements are the key parts of building a panoramic interactive model.

[0048] Next, the visual image of the castle's exterior is associated with the audio describing the castle's overall structure. For example, when the audio mentions the castle's height, the overall tall image of the castle in the visual image echoes it, which is a connection relationship. Let's look at the panoramic interactive hotspot of the castle gate. Its visual image is associated with the audio describing the construction characteristics of the gate. When the audio mentions the material of the gate, the material texture of the gate in the visual image should be paid attention to, which establishes a connection between the visual and audio of the gate. Moreover, as part of the castle, the hotspot information of the gate also has a logical connection with the overall visual image and audio of the castle. For example, the audio commentary of the castle's overall defense system can be linked to the role of the gate in defense. This logical relationship is also a connection between the elements of model construction.

[0049] After that, the model building elements are feature extracted to generate feature vectors. Taking the interactive path of the visual image and related audio of the castle tower as an example, if it is used as an index tag of the model building element, the server will perform reciprocating cyclic timing encoding on this interactive path. For example, the display video of the tower is 20 seconds long, and the changes in the tower image and the rhythm of the audio are encoded according to a time unit of 2 seconds to generate cyclic timing features. Then determine the index positioning features of the interactive path, such as the starting position of the image and audio of this tower in the entire panoramic material, the scene type (castle defense facility scene), etc., and fuse the cyclic timing features and index positioning features to generate path features. Then, focus weights are allocated to the path features, and higher weights are given to the unique architectural structure of the tower and the part of the audio that focuses on describing the defense function of the tower, and finally generate the feature vector of the interactive path. For the visual picture corresponding to the a priori panoramic material of the castle inner courtyard garden, the server will encode its semantic features, identify the semantic information such as plant species and garden layout in the picture, generate semantic features of the visual picture, and then convert these semantic features into vector form, thereby generating the element feature vector of the visual picture of this material. If it involves the interactive hot zone corresponding to the a priori panoramic material, such as the interactive hot zone of a treasure room in the castle, the server searches the interactive hot zone parameter index library based on this interactive hot zone, and the index library contains the mapping relationship between the interactive hot zone and the interactive hot zone features. For example, the interactive hot zone of the treasure room corresponds to the interactive hot zone features such as treasure legends and secret room mechanisms, and the searched interactive hot zone features are used as the element feature vector of the interactive hot zone corresponding to this a priori panoramic material. When the index label of the model construction element is the narrative object corresponding to the a priori panoramic material, such as a lord in the history of the castle, the server searches the narrative object parameter index library based on this narrative object, and the library contains the mapping relationship between the narrative object and the narrative object features, such as the lord's family history, the development of the castle during the reign, etc., and the searched narrative object features are used as the element feature vector of the narrative object corresponding to this a priori panoramic material. Through the above operations, the server successfully constructs the element feature vector of the panoramic interaction model, thereby completing the construction of the panoramic interaction model.

[0050] In a possible implementation, when the index tag of the model construction element is any one interaction path among the first visual and audio combination data, the second visual and audio combination data in the prior panoramic material, and the second interaction processing result in the prior panoramic material, step S123 includes: The interaction path is subjected to reciprocating cyclic timing coding to generate a cyclic timing feature of the interaction path.

[0051] Encoding is performed according to the cyclic time series features of the interaction path to generate an element feature vector of the interaction path.

[0052] In a possible implementation manner, encoding and representing the cyclic temporal features of the interaction path to generate the element feature vector of the interaction path includes: An index positioning feature of the interaction path is determined.

[0053] The cyclic timing features of the interaction path and the index positioning features of the interaction path are fused to generate path features of the interaction path.

[0054] Focus weights are allocated to the path features of the interaction path to generate an element feature vector of the interaction path.

[0055] In this embodiment, taking the castle panoramic material as an example, if there is an interactive path from the visual picture and related audio of the castle hall (belonging to the first visual audio combination data) to the second visual audio combination data related to the castle architectural structure in the prior panoramic material. The server first performs a reciprocating cyclic timing encoding on this interactive path. Assuming that the display video of the castle hall is 30 seconds long, the server encodes the dynamic changes of the visual picture of the hall within 30 seconds, such as people walking, light and shadow changes, etc., as well as the rhythm changes of the audio, volume fluctuations, etc., according to a specific time interval, such as every 3 seconds as a unit. This encoding method records the change pattern of the visual and audio over time in this interactive path, thereby generating a cyclic timing feature of the interactive path.

[0056] Then, the element feature vector is generated by encoding the cyclic timing features of this interactive path. In this process, the server needs to determine the index positioning features of the interactive path. For the interactive path of the castle hall, the index positioning features include the starting and ending positions of the interactive path in the entire panoramic material system, such as its specific time period in the entire castle panoramic display, and the specific scene type to which it belongs, here is the hall scene inside the castle. After determining the index positioning features, the server fuses the cyclic timing features and the index positioning features. The fusion process is to comprehensively consider the visual and audio changes in the time dimension and its positioning information in the overall material to generate the path features of the interactive path. For example, in the visual picture of the hall, the light and shadow changes in a specific time period (cyclic timing features) are combined with the specific position of the light and shadow changes in the hall and the order in the entire display (index positioning features).

[0057] Finally, the path features of the interaction path are focused and weighted to generate a feature feature vector. In the example of the castle hall, higher weights are given to those parts that are more important in showing the castle's architectural features or historical and cultural significance. For example, the visual part of the hall showing the castle's family emblem or the place where important historical events took place, as well as the part of the audio that explains these important elements in detail, will be assigned a higher weight. Some relatively minor visual elements, such as the decorative details in the corners of the hall, are assigned lower weights if they have little impact on the overall interaction path. Through such weight allocation, the path features are converted into a feature feature vector that can accurately reflect the importance and characteristics of this interaction path in the panoramic interaction model. This feature feature vector can accurately represent this specific interaction path in the panoramic interaction model, whether it is a path related to the first visual audio combination data, or a path related to the visual audio combination data or the interaction processing results in the prior panoramic material, thereby helping to build a complete and accurate panoramic interaction model.

[0058] For the interaction path related to the second interaction processing result in the prior panoramic material, it is assumed that it is the interaction path from the result obtained by the user's previous interaction operation on another similar castle panoramic material (such as the area frequently viewed by the user or the feedback information after a specific interaction, etc.) to the visual and audio data of a specific area in the current castle panoramic material. The server also performs the above operation. First, reciprocating cyclic timing encoding is performed, for example, the operation sequence, operation frequency, etc. in the previous interaction processing result and the changes in the visual and audio of the current specific area are encoded according to a time unit of 5 seconds to generate cyclic timing features. When determining the index positioning feature, the relative position of this interaction path in the two materials (prior material and current material) and the relationship with the overall interaction logic are considered. Then the cyclic timing feature and the index positioning feature are fused to obtain the path feature, and finally the focus weight is allocated. For the part that is closely related to the previous user interaction behavior, such as the corresponding part of the building structure that the user frequently viewed before in the current material, a higher weight is given, thereby generating the element feature vector of this interaction path, providing an accurate element representation for the construction of the panoramic interaction model.

[0059] In a possible implementation manner, when the index tag of the model construction element is any one of the visual pictures corresponding to the prior panoramic material and the visual pictures corresponding to the current panoramic material, step S123 may further include: The material visual picture is encoded with semantic features to generate visual picture semantic features of the material visual picture.

[0060] The visual picture semantic features of the material visual picture are converted to generate an element feature vector of the material visual picture.

[0061] Continuing with the castle panoramic material as an example, for the visual picture corresponding to the prior panoramic material, it is assumed that it is a panoramic visual picture similar to a castle that has been processed before. The server first encodes the semantic features of this material visual picture. The server will identify various elements in the picture and their semantic information. For example, the architectural style of the castle. If it is a Gothic building, then the building's spires, stained glass windows and other features will be identified and encoded; the defensive facilities in the castle, such as the arrow battlements and watchtowers on the city wall, their position, shape, quantity and other information will be converted into semantic features; and the layout inside the castle, such as the distribution relationship of the hall, bedroom, storage room, etc. will also be encoded. For the environmental elements around the castle, such as the surrounding forests and rivers, their relative position relationship with the castle and their own characteristics, such as the vegetation type of the forest and the flow direction of the river, will also be included in the category of semantic feature encoding. In this way, the visual picture semantic features of this prior material visual picture are generated.

[0062] Next, the semantic features of the visual image of the prior material are converted to generate an element feature vector. This conversion process involves converting the semantic features into a vector form that can be processed by the computer. For example, for the characteristics of the Gothic architectural style, the height of the spire, the shape and proportion of the stained glass windows, and other information may be converted into a specific numerical vector according to predefined coding rules. The characteristics of defensive facilities such as the number of arrow battlements on the city wall and the height of the watchtower will also be converted into corresponding vector representations. The area, connection relationship and other information of each room in the internal layout of the castle will also be mapped into vector elements. The type of forest vegetation in the surrounding environment of the castle may be converted into a specific coding vector according to the vegetation classification system, and the direction and width of the river will also be quantified into vector form. These converted vector elements are combined together to generate the element feature vector of the visual image corresponding to the prior panoramic material, which can accurately represent the characteristics of the visual image of the prior material in the panoramic interaction model.

[0063] For the visual picture corresponding to the current panoramic material, such as the panoramic picture of the castle being processed. The server also performs semantic feature encoding operations. Identify the semantic features of the main building of the castle, such as the building material of the castle. If it is stone, the color, texture and other information of the stone will be encoded; the gate structure of the castle, including the material of the door, the way the door opens, etc.; the decorative elements of the courtyard in the castle, such as the type and position relationship of fountains and sculptures, etc. will be identified and encoded as semantic features. At the same time, the activities of people in the castle, such as the patrol routes of the guards and the work areas of the servants, will also be included in the semantic feature encoding if they are reflected in the visual picture.

[0064] Then the semantic features of the visual picture are converted to generate an element feature vector. The semantic features of the castle building materials such as color and texture are converted into vector elements according to the predetermined conversion rules. The detailed features of the gate structure are also converted into vector representations. The type and location of the courtyard decoration elements are quantified into vectors. The activities of personnel will also be mapped into specific vector elements. These vector elements are combined to form the element feature vector of the visual picture corresponding to the current panoramic material. This element feature vector can accurately represent the characteristics of the current material visual picture in the panoramic interaction model, providing effective data support for interactive analysis with other model construction elements and determining the role and influence of this visual picture in the construction process of the panoramic interaction model.

[0065] In a possible implementation manner, when the index tag of the model construction element is an interactive hotspot corresponding to the prior panoramic material, step S123 may further include: An interactive hot zone parameter index library is searched according to the interactive hot zone corresponding to the prior panoramic material, wherein the interactive hot zone parameter index library includes mapping relationships between each of the interactive hot zones and each of the interactive hot zone features.

[0066] The searched interaction hot zone features are used as element feature vectors of the interaction hot zone corresponding to the priori panoramic material.

[0067] When the index tag of the model construction element is the narrative object corresponding to the priori panoramic material, step S123 may further include: A narration object parameter index library is searched according to the narration object corresponding to the priori panoramic material, wherein the narration object parameter index library includes mapping relationships between each of the narration objects and each of the narration object features.

[0068] The searched narration object features are used as element feature vectors of the narration object corresponding to the priori panoramic material.

[0069] In this embodiment, the a priori panoramic material of the castle is continued as an example, for the interactive hot zone therein, such as the interactive hot zone of the treasure room in the castle. The server searches the interactive hot zone parameter index library according to the interactive hot zone corresponding to the a priori panoramic material. This interactive hot zone parameter index library contains the mapping relationship between each interactive hot zone and each interactive hot zone feature. In the case of the interactive hot zone of the treasure room, the index library may store the mapping relationship between the treasure room and its specific interactive hot zone features. For example, the location information of the treasure room is related to the interactive hot zone feature, and its specific location in the castle, whether it is underground or in a tower of the castle, etc., has a corresponding mapping. The security protection mechanism of the treasure room is also one of the interactive hot zone features. Protection measures such as heavy stone doors and complex locks are mapped with the interactive hot zone of the treasure room in the index library. In addition, the treasure legend that may exist in the treasure room also belongs to the interactive hot zone feature, such as the legendary treasure type, the source of the treasure, etc., which is mapped with the interactive hot zone of the treasure room.

[0070] The server searches the interactive hot zone parameter index library to obtain the features related to the interactive hot zone of the treasure room. These interactive hot zone features are then used as the element feature vector of the interactive hot zone corresponding to the prior panoramic material. This element feature vector contains the location information vector of the treasure room, such as its three-dimensional position in the castle in the form of coordinates; the security protection mechanism vector, such as the thickness of the stone door and the complexity of the lock with a specific value; the treasure legend vector, which may use a predefined code to represent the type and source of the treasure. This element feature vector can accurately represent the characteristics of the interactive hot zone of the treasure room in the panoramic interaction model, and provides an important data basis for model construction.

[0071] When the index tag of the model construction element is the narrative object corresponding to the prior panoramic material, the operation of the server is as follows. Assume that the narrative object in the prior panoramic material is the former lord of the castle. The server searches the narrative object parameter index library based on the narrative object corresponding to this prior panoramic material. This narrative object parameter index library contains the mapping relationship between each narrative object and each narrative object feature. For the narrative object of the castle lord, in the narrative object parameter index library, his family information and the narrative object feature are mapped. For example, family information such as the name of the family to which the lord belongs and the heraldic logo of the family are part of the narrative object feature. The lord's reign is also an important narrative object feature, including the start and end years of his reign, the development of the castle under his rule, and other information, which are mapped to the lord's narrative object. The lord's military achievements also belong to the narrative object features, such as the castle defense battles he led and the military victories he achieved, which are mapped to the lord's narrative object.

[0072] The server retrieves these features related to the lord's narration object by searching the narration object parameter index library. These narration object features found are then used as the element feature vector of the narration object corresponding to the prior panoramic material. This element feature vector contains a family information vector, such as a family name and heraldic logo represented by a specific code; a reign period vector, represented by a quantitative representation of the time interval and castle development; a military achievement vector, which may use predefined values ​​to represent the scale of military victory and other information. This element feature vector can accurately represent the characteristics of the castle lord narration object in the panoramic interaction model, which helps to build a complete panoramic interaction model.

[0073] In a possible implementation, step S130 may include: Step S131, performing an architecture analysis on the panoramic interaction model to identify various components in the panoramic interaction model, wherein the various components include a portion related to the visual and audio combination data, a portion related to the panoramic interaction hotspot information, and a connection structure between the portion related to the visual and audio combination data and the portion related to the panoramic interaction hotspot information.

[0074] Step S132, generating a structural decomposition strategy for presenting the internal structure of the model according to the various components in the panoramic interaction model, wherein the structural decomposition strategy is used to layer or partition the panoramic interaction model according to functions, data flows or building element types.

[0075] Step S133, according to the structural decomposition strategy, the interaction paths between the various elements in the model structure after decomposing the panoramic interaction model are marked, and the interaction paths include the connection path from the first visual and audio combination data to the panoramic interaction hotspot information, the association path between different panoramic interaction hotspot information and other logical connection paths between model construction elements.

[0076] Step S134, grouping the elements in the panoramic interaction model according to the marked interaction paths, and analyzing the interaction relationship between the elements in each group and the interaction boundaries with other groups of elements for each element group, thereby determining the interaction space range involved in the group of elements.

[0077] Step S135, within each determined interactive space range, feature extraction is performed on the model elements therein to obtain an initial feature set within each interactive space range. The basis for feature extraction includes the type of element, the functional role of the element in the interactive space, and the interactive relationship between the element and other elements, etc. Among them, for visual elements, features are extracted from the dimensions of the composition of the visual picture and the distribution law of the visual elements. For audio elements, extraction is performed from the dimensions of the frequency features, rhythm features, and synchronization relationship with the visual elements of the audio. For hotspot information elements, features are obtained from the dimensions of the triggering conditions of the hotspot and the interactive actions associated with the hotspot.

[0078] Step S136, based on the predefined feature evaluation criteria, each feature in the initial feature set within each interactive space is evaluated, and screening and optimization are performed according to the evaluation results to generate a screened and optimized target feature set. The feature evaluation criteria are determined based on the importance of the feature to the hotspot interaction function of the panoramic interaction model, the correlation between the features, and the representativeness of the feature in the overall interaction logic.

[0079] Step S137, spatially mapping the target feature set within each interactive space to obtain a feature representation after spatial mapping, wherein the spatial mapping is performed based on a predefined spatial mapping rule, wherein the spatial mapping rule includes mapping based on a distance relationship between the feature and a key element in the interactive space, and a spatial association between the feature and other features.

[0080] Step S138, integrating the feature representations after spatial mapping within each interactive space range to construct a hotspot interactive space feature that describes the current panoramic material.

[0081] In this embodiment, first, the panoramic interaction model is structurally analyzed to identify its various components. The relevant part of the visual and audio combination data contains image data of the castle panoramic picture, such as different angle views of the castle's exterior, layout pictures of each room inside, etc., and corresponding audio data, such as narration about the history of the castle, the sound of wind and rain in the castle environment, etc. The relevant part of the panoramic interactive hotspot information covers various hotspots in the castle, such as gates, towers, treasure rooms and other hotspot areas, including the location of the hotspot, interactive prompts related to the hotspot, etc. The connection structure between the relevant part of the visual and audio combination data and the relevant part of the panoramic interactive hotspot information is also an important component. For example, there is a connection between the visual picture of the castle gate and the audio about the history of the gate's construction, and the connection between the gate hotspot and the overall visual audio of the castle.

[0082] Next, based on the various components in the panoramic interaction model, a structural decomposition strategy for presenting the internal structure of the model is generated. According to the functional layering, the visual and audio data related to the display of the castle's architectural structure can be classified into one layer, the data related to the castle's historical and cultural narrative can be classified into another layer, and the hotspot information related to the user interaction function can be classified into one layer. According to the data flow partition, starting from the input of the original panoramic material data, through data processing, fusion with hotspot information, to the final output of the interactive results, each link is partitioned separately. When partitioning according to the construction element type, the visual elements, audio elements, and hotspot information elements are divided into different areas.

[0083] Then, according to the structural decomposition strategy, the interaction paths between the various elements in the model structure after the panoramic interaction model is decomposed are marked. The connection path from the first visual audio combination data to the panoramic interactive hotspot information, such as the path from the visual picture of the castle's exterior and the overall environmental audio to the tower hotspot, marks the position of the tower in the visual picture and the time point when the tower is mentioned in the audio. The association path between different panoramic interactive hotspot information, such as the possible castle internal channel association path between the gate hotspot and the treasure room hotspot, marks the connection logic of the channel between the two hotspots. Other logical connection paths between model construction elements, such as the logical association path between an audio clip describing the castle's defense system and multiple hotspots such as gates and towers, marks the defense logic described in the audio and the role relationship between each hotspot in the defense system.

[0084] According to the marked interaction paths, the elements in the panoramic interaction model are grouped. For each element group, the interaction relationship between the elements in the group and the interaction boundary with other groups of elements are analyzed. Taking the element group related to the tower hotspot as an example, the elements in the group include the visual picture of the tower, the audio describing the function of the tower, and the historical events related to the tower. The interaction relationship within the group is the audio's interpretation of the visual picture, the association between historical events and the tower, etc. The interaction boundary with other groups of elements, such as the boundary with the gate hotspot element group, lies in the relative position relationship between the tower and the gate in the castle building structure, etc., thereby determining the interactive space range involved in the group of elements.

[0085] Within each determined interactive space, feature extraction is performed on the model elements therein to obtain the initial feature set within each interactive space. For visual elements, taking the visual image of the tower as an example, from the aspect of the composition of the visual image, the features of the tower's shape, color, and texture of the building materials are extracted. From the aspect of the distribution law of visual elements, the layout relationship between the upper defense facilities of the tower and the lower living space is extracted. For audio elements, such as audio related to the tower, from the aspect of the frequency characteristics of the audio, the high and low changes in the audio frequency when describing the tower's function are extracted. From the aspect of the rhythm characteristics, the fast and slow changes in the audio rhythm when describing the tower's construction process are extracted. In terms of the synchronization relationship with the visual elements, the features of whether the visual image switches to a certain structure of the tower in time when the audio mentions it are extracted. For hot spot information elements, such as tower hot spots, from the aspect of the triggering conditions of the hot spots, whether the interaction is triggered by clicking the mouse or focusing the line of sight on the tower area is extracted. From the aspect of the interactive actions associated with the hot spots, the features of whether the interactive animation of the internal structure of the tower is displayed or more detailed audio of the tower's history is played after the trigger are extracted.

[0086] Based on the predefined feature evaluation criteria, each feature in the initial feature set within each interactive space is evaluated. This feature evaluation criterion is determined based on the importance of the feature to the hotspot interaction function of the panoramic interaction model, the correlation between features, and the representativeness of the feature in the overall interaction logic. For example, the architectural structure features of the tower are very important for the hotspot interaction function of showing the function and history of the tower, while some tiny decorative details in the tower picture are less important if they are not closely related to the hotspot interaction function. Screening and optimization are performed based on the evaluation results, and those features with low importance, weak correlation, or insufficient representation in the overall interaction logic are screened out to generate a screened and optimized target feature set.

[0087] The target feature set within each interactive space is spatially mapped. According to the pre-defined spatial mapping rules, the mapping is performed based on the distance relationship between the feature and the key elements in the interactive space and the spatial association between the feature and other features. For example, the building height feature of the tower and the defensive function feature of the tower may have a close mapping relationship in the spatial mapping because they are closely related to the overall properties of the tower. Finally, the feature representations after spatial mapping within each interactive space are integrated to construct a hotspot interactive space feature that describes the current castle panorama material. This hotspot interactive space feature will fully reflect the interactive characteristics of each hotspot in the castle panorama material, providing an important basis for subsequent matching operations.

[0088] In other areas of the castle, such as the hall element group inside the castle, the visual elements include the hall's architectural decoration, character activities and other images, the audio elements include the hall's ambient sound and the audio of the explanation of the hall's purpose, and the hotspot information element may be a specific display area in the hall. The same process is followed to perform feature extraction, evaluation, screening, spatial mapping and other operations. The hall's decorative style, character costumes and other features are extracted from the composition of the visual picture, and the relationship between personnel activities and the use of the hall space is extracted from the distribution law; the synchronization relationship between the frequency and rhythm characteristics of the audio and the visual elements is reflected in the changes in the audio and the coordination of the visual picture when explaining important activities in the hall; the hotspot trigger conditions and associated interactive actions involve the user's operation on a specific area of ​​the hall and the subsequent interactive response. After a series of operations, the feature representations of these areas are also integrated into the overall hotspot interactive space features, so that the hotspot interactive space features can accurately cover the interactive characteristics of the castle panoramic material.

[0089] For the castle's external environment element group, the visual elements include the terrain and vegetation around the castle, the audio elements include natural environment sound effects, and the hotspot information elements may be the fortifications around the castle. In the feature extraction process, the composition of the visual picture extracts the topographic features of the terrain, the types of vegetation, etc., and the distribution pattern considers the layout relationship between vegetation and fortifications. The frequency and rhythm characteristics of the audio are related to the natural environment, and the synchronization relationship with the visual elements is reflected in the coordination of the environmental sound effects and the picture. The hotspot trigger conditions and associated interactive actions involve interactive operations on the surrounding areas of the castle. After evaluation, screening, and spatial mapping, the characteristics of these elements are also integrated into the overall hotspot interactive space characteristics, thereby constructing a complete description of the hotspot interactive space characteristics of the current castle panoramic material.

[0090] In a possible implementation, step S140 includes: Step S141, identifying different feature dimension information in the hotspot interaction space feature, and constructing a matching relationship analysis framework for analyzing the hotspot interaction space feature and the commentary audio or interactive animation based on the different feature dimension information.

[0091] In this embodiment, the hotspot interaction spatial features of the castle panoramic material include multiple feature dimension information related to the visual, audio, and hotspot interactions of different areas of the castle (such as towers, gates, halls, etc.). For example, the architectural structure features of the tower area, the audio features related to it (such as the features of the audio clips that tell the history of the tower), and the features of the hotspot triggered interaction (such as clicking to view the internal structure of the tower). The matching relationship analysis framework is constructed using this information of different dimensions, which will serve as the basis for subsequent analysis of the matching relationship with the commentary audio or interactive animation.

[0092] Step S142, for the commentary audio, feature analysis is performed from the aspects of content structure, semantic information, emotional tendency, and rhythm and rhyme of the commentary audio. In terms of content structure, the hierarchical structure of the commentary audio narration and the distribution of key contents are analyzed; in terms of semantic information, concepts and descriptions related to the panoramic material contained in the commentary audio are mined; in terms of emotional tendency, the emotional atmosphere conveyed by the audio as a whole is judged; in terms of rhythm and rhyme, the speed of the rhythm, the regularity of the rhythm, etc. of the commentary audio are analyzed.

[0093] For the audio commentary, feature analysis is performed from multiple aspects. Taking an audio commentary about a castle as an example, in terms of content structure, its narrative hierarchy is analyzed. Assuming that the audio commentary starts with the overall appearance and historical background of the castle, and then gradually goes into various areas inside the castle, such as first introducing the significance of the construction of the gate, and then describing the functions and uses of the hall, this is a hierarchical structure. In terms of the distribution of key content, for example, the content related to the castle's defense system may be the focus, which will occupy a large amount of space in the audio and be described in detail. In terms of semantic information, the concepts and descriptions related to the panoramic material of the castle contained in the audio commentary are mined. For example, the audio mentions that the architectural style of the castle is Gothic, the material of the castle gate is oak, and the hall is decorated with murals from a specific period, etc. These are all semantic information closely related to the panoramic material of the castle. In terms of emotional tendency, the overall emotional atmosphere conveyed by the audio is judged. If the audio is telling the glorious historical period of the castle, it may have an emotional atmosphere of pride and reverence; while when describing the castle being destroyed by war, it may convey regret and heavy emotions. In terms of rhythm and rhyme, the rhythm speed and rhythmic rules of the audio commentary are analyzed. For example, when introducing the grand scale of the castle, the rhythm may be slower to reflect its solemnity; when telling some interesting stories in the castle, the rhythm may be faster to increase the fun.

[0094] Step S143, for the interactive animation, extract features from the visual element composition, action logic, interactive response mechanism, and visual rhythm of the animation. In terms of visual element composition, determine the graphics, colors, and objects included in the interactive animation. In terms of action logic, determine the movement rules and logical relationships of each element in the interactive animation. In terms of interactive response mechanism, determine the interactive operations to which the interactive animation responds. In terms of visual rhythm, determine the screen switching speed of the interactive animation and the rhythm of element movement.

[0095] For interactive animation, features are also extracted from multiple aspects. Taking an interactive animation related to a castle as an example, in terms of the composition of visual elements, the graphics, colors, objects, etc. contained in the interactive animation are determined. For example, the graphics of the castle building in the animation present a realistic three-dimensional effect, and the colors follow the actual stone color and architectural decoration color of the castle. The objects include the characters, weapons and equipment in the castle. In terms of action logic, the movement rules and logical relationships of each element in the interactive animation are determined. For example, the guards in the castle move according to the patrol route, the opening and closing of the gate follows certain physical laws, and the launching action of the trebuchet has corresponding operation logic. In terms of interactive response mechanism, the interactive operation that the interactive animation responds to is determined. For example, when the user clicks on a room in the castle, the animation will show the detailed layout of the room; when the mouse hovers over a weapon, the detailed information of the equipment will be displayed. In terms of visual rhythm, the speed of screen switching and the rhythm of element movement of the interactive animation are determined. For example, the switching speed of the screen when showing different areas of the castle is moderate, neither too fast to cause the user to not have time to see clearly, nor too slow to make the user feel dragged; the movement rhythm of the characters and objects in the castle conforms to the physical laws and normal visual perception in reality.

[0096] Step S144, based on the matching relationship analysis framework and the feature set of the narration audio and interactive animation obtained by analysis, a set of matching association index system is established. The matching association index system should include indicators of multiple dimensions, including content matching index, interaction logic matching index, and visual and audio coordination index. The content matching index is used to measure the degree of matching between the semantic information in the hotspot interactive space features and the narration audio content and the content expressed by the interactive animation. The interaction logic matching index is used to evaluate the degree of fit between the interaction triggering conditions in the hotspot interactive space features and the interactive response mechanism of the interactive animation and the guiding logic of the narration audio in the interactive process. The visual and audio coordination index is used to determine the coordination between the visual and audio association features in the hotspot interactive space features and the visual and audio in the narration audio and interactive animation.

[0097] For example, whether the tower building structure features in the hotspot interactive space feature of the castle panorama material are consistent with the description of the tower in the commentary audio and the display content of the tower in the interactive animation. If the hotspot interactive space feature indicates that the tower has five floors, the commentary audio mentions the number of floors of the tower and the interactive animation also accurately displays the five-story tower, then the content matching will have a higher score. The interactive logic matching index is used to evaluate the degree of fit between the interactive trigger conditions in the hotspot interactive space feature and the interactive response mechanism of the interactive animation and the guiding logic of the commentary audio in the interactive process. For example, the hotspot interactive space feature sets the interaction of clicking on the castle hall to trigger the display of the interior decoration of the hall, and the interactive animation sets the interactive mechanism of clicking on the hall area to respond to display the interior decoration, and the commentary audio guides the user to click to view the interior decoration when mentioning the hall, then the interactive logic matching will be better evaluated. The visual and audio coordination index is used to determine the coordination between the visual and audio association features in the hotspot interactive space feature and the visual and audio in the commentary audio and interactive animation. For example, in the castle gate area, when the visual picture in the hotspot interactive space feature shows the gate slowly opening, the commentary audio is accompanied by the sound effect of the gate opening. The gate opening action in the interactive animation matches the sound effect in the audio in terms of rhythm and timing, so there will be a better performance in the visual and audio coordination index.

[0098] Step S145, based on the matching association indicator system, the hotspot interactive space features and the commentary audio and interactive animation are matched and evaluated respectively. For each indicator, the matching score of the hotspot interactive space features and the commentary audio or interactive animation on the indicator is calculated according to the evaluation method defined corresponding to the indicator, and a preliminary matching result set between the hotspot interactive space features and the commentary audio and interactive animation is obtained. The preliminary matching result set includes the scores of each matching pair on each indicator and the comprehensive score. The matching pairs include the hotspot interactive space features and the commentary audio, and the hotspot interactive space features and the interactive animation.

[0099] For example, for the content matching index, the score is quantitatively scored according to the matching degree of semantic information. If it is a complete match, it may get a full score, and if it is a partial match, it is scored according to the matching ratio; for the interactive logic matching index, it is scored according to the fit of the interactive trigger and response mechanism and the guiding logic; for the visual and audio synergy index, it is scored according to the synergy effect of vision and audio. In this way, the preliminary matching result set between the hotspot interactive space features and the commentary audio and interactive animation is obtained. This result set contains the scores of each matching pair (hotspot interactive space features and commentary audio, hotspot interactive space features and interactive animation) on each indicator and the comprehensive score.

[0100] Step S146, screening the preliminary matching result set according to the set screening criteria, sorting the screened matching pairs according to the comprehensive scores to obtain a sorted matching result set.

[0101] Assume that the screening criteria are that the comprehensive score is higher than a certain threshold, and the scores of each indicator cannot be lower than the minimum requirement. The matching pairs that do not meet this criterion are screened out, and the screened matching pairs are sorted according to the comprehensive score to obtain a sorted matching result set.

[0102] Step S147, based on the sorted matching result set, analyzing the relationship between the hotspot interaction space feature and the commentary audio or interactive animation in each matching pair to determine a range definition rule set that can trigger the interactive hot zone.

[0103] For example, if the hotspot interactive spatial features of the castle gate area in a match pair are highly matched with the audio commentary and interactive animation about the gate, then the scope definition rule determined may be that when the user's line of sight approaches the gate area or the mouse clicks the gate area, the relevant audio commentary or interactive animation is triggered. The determination of this rule is based on an in-depth analysis of the relationship between the various elements in the match pair, including the visual features of the gate, the description focus in the audio commentary, and the interactive response logic in the interactive animation.

[0104] Step S148, based on the determined range definition rule set of the triggerable interaction hot zone, the triggerable interaction hot zone is marked in the hotspot interaction space feature of the current panoramic material, wherein, for each range definition rule, the area that meets the range definition rule is searched in the hotspot interaction space feature and marked, thereby identifying the triggerable interaction hot zone in the hotspot interaction space feature, thereby generating the triggerable interaction hot zone of the current panoramic material.

[0105] For example, for the triggering rules of the gate area, the visual features, related audio features, hotspot triggering features and other parts that meet the rules are found in the hotspot interactive space features and marked, thereby identifying the triggerable interactive hotspot in the hotspot interactive space features, thereby generating the triggerable interactive hotspot of the current castle panoramic material. In other areas of the castle, such as towers, halls, etc., the same process is followed to finally determine the triggerable interactive hotspot of the entire castle panoramic material. This helps to accurately trigger the commentary audio or interactive animation related to a specific area when the user interacts with the castle panoramic material, thereby improving the user's interactive experience.

[0106] In a possible implementation, step S150 includes: Step S151, using a pre-built real-time interaction monitoring module to continuously track the current progress information of audio playback and the real-time status information of user interaction operations, wherein the current progress information of audio playback includes the current playback timestamp, the current playback segment identifier, and the current playback content overview, and the real-time status information of user interaction operations includes the user operation type, the operation trigger timestamp, the operation target identifier, and the operation context environment description.

[0107] In this embodiment, for the audio playback of the castle panoramic material, its current playback timestamp accurately records the specific time of playback, for example, it has been played for 10 minutes and 30 seconds. The current playback paragraph identifier clarifies the paragraph to which the currently playing content belongs, such as the audio paragraph about the castle defense facilities being played. The current playback content overview briefly summarizes the audio content being played, such as the function of the arrow stacks on the castle wall. As for the real-time status information of the user's interactive operation, if the user operates in the panoramic display interface, the user's operation type will be accurately recorded, such as a mouse click operation or a touch sliding operation. The operation trigger timestamp records the exact moment when the operation occurs, assuming that the user operates at 12 minutes and 15 seconds. The operation target identifier determines the target pointed to by the user's operation, such as the user clicks on the tower area in the castle. The operation context description covers the panoramic picture state when the operation occurs, the previous interaction history and other information, such as the overall appearance of the castle is being displayed when the operation is performed, and the user has just viewed the castle gate area and other information.

[0108] Step S152, using a pre-built hotspot positioning and prediction unit to receive the audio playback progress information and user interaction operation status information from the real-time interaction monitoring module as input information, and predicting the user's next point of interest or interaction intention by analyzing the input information, wherein the hotspot positioning and prediction unit maintains a user behavior pattern library, and the user behavior pattern library learns the behavior preferences and interaction patterns of different users in various situations based on historical interaction data. The hotspot positioning and prediction unit uses the user behavior pattern library to match the behavior characteristics of the current user, predicts the panoramic interaction hotspot that the current user is concerned about, and at the same time, combines the audio playback progress to infer the hotspot area related to the current playback content, and outputs a predicted hotspot area set and a corresponding prediction confidence score.

[0109] For example, it is found from historical data that after viewing the castle gate, a high proportion of users will be interested in the nearby towers; when the audio plays the content related to the castle defense facilities, many users will tend to view specific defense buildings. The hotspot positioning and prediction unit uses this user behavior pattern library to match the behavior characteristics of the current user. Assuming that the current user has performed some related operations after viewing the castle gate, combined with the audio playing the content related to the castle defense facilities, through matching with the user behavior pattern library, it is predicted that the panoramic interactive hotspot that the current user is concerned about may be the nearby tower. At the same time, combined with the audio playback progress, the hotspot area related to the current playback content is inferred. For example, when the audio mentions the arrow stack, it may be related to the defense function of the tower, so the tower is regarded as one of the hotspot areas related to the current playback content. The final output is a set of predicted hotspot areas, which may include towers, nearby small fortifications, etc., as well as the corresponding prediction confidence scores. The tower may have a confidence score of 0.8, indicating that it is highly likely to be the next hotspot that the user will pay attention to.

[0110] Step S153, using a pre-built interactive decision engine to receive the predicted hotspot area set and confidence score output by the hotspot positioning and prediction unit, as well as the hotspot interactive spatial characteristics and triggerable interactive hot zone information of the current panoramic material, and output a post-decision interactive instruction set, wherein the interactive instruction set includes a target hotspot identifier, an interaction type, a viewing angle adjustment parameter, and an execution timing. The interactive decision engine defines a decision rule set, thereby determining the next panoramic interactive hotspot to be displayed based on the decision rule set and input information, as well as the interaction type that triggers the commentary audio and / or interactive animation, and calculating the corresponding viewing angle adjustment scheme, wherein the viewing angle adjustment scheme includes a target viewing angle coordinate, an adjustment speed, and a smoothness parameter.

[0111] For example, based on the fact that the tower in the predicted hotspot area set has a high confidence score, as well as the relevant features of the tower area in the hotspot interaction space features and the triggerable interaction hot zone information, the target hotspot is identified as the tower. The interaction type may be to click to view the internal structure of the tower. The perspective adjustment parameters are calculated based on the positional relationship between the current perspective and the tower in the panoramic material. The target perspective coordinates are determined to be the coordinate position that can best display the overall appearance and entrance of the tower. The adjustment speed is set to a moderate speed, such as 10 degrees per second. The smoothness parameter ensures that the perspective switching process is smooth and natural to avoid screen jitter or jump. The execution timing is set to execute within a short time after the audio is played to the content related to the tower or the user operates.

[0112] Step S154, using a pre-built automatic perspective adjustment module to receive the perspective adjustment parameters and target hotspot identifiers in the interactive instruction set output by the interactive decision engine, and calculating a smooth transition path from the current perspective to the target perspective based on the perspective adjustment parameters and the target hotspot identifier, while controlling the perspective adjustment mechanism of the panoramic display system, executing perspective switching according to the calculated smooth transition path and perspective adjustment parameters, so that the user's perspective is aligned with the target panoramic interactive hotspot after the decision.

[0113] Assuming that the current perspective is in the direction of the castle gate and needs to be switched to the tower direction, the perspective automatic adjustment module calculates a smooth transition path based on the target perspective coordinates (the coordinate position of the tower), adjustment speed (10 degrees per second) and smoothness parameters, which may be a path of first turning upward at a certain angle, then turning horizontally to the tower direction, etc. At the same time, the perspective adjustment mechanism of the panoramic display system is controlled to perform perspective switching according to the calculated smooth transition path and perspective adjustment parameters, so that the user's perspective is aligned with the target panoramic interactive hotspot after the decision, that is, the tower area.

[0114] Step S155, using a pre-built content trigger and synchronization control unit to receive the target hotspot identifier, interaction type and execution timing in the interaction instruction set output by the interaction decision engine, and at the same time receive the perspective adjustment completion timestamp output by the perspective automatic adjustment module for analysis, and output the content display status information after triggering. The content trigger and synchronization control unit maintains an accurate timer and synchronization control logic. For the commentary audio, the content trigger and synchronization control unit controls the audio playback system to start playing from the specified position. For the interactive animation, the content trigger and synchronization control unit starts the animation rendering and display process.

[0115] Afterwards, if the interaction type is to click to view the internal structure of the tower, when the perspective adjustment is completed, the audio playback system is controlled to start playing from the specified position according to the execution timing and the perspective adjustment completion timestamp, such as starting from the audio part describing the internal structure characteristics of the tower. For interactive animations, if there is an interactive animation showing the internal structure of the tower, the content triggering and synchronization control unit starts the animation rendering and display process, and renders and displays it on the user interface according to the predefined settings of the animation.

[0116] Step S156, after the content triggering and synchronization control unit executes the content display, the user's instant feedback data is collected by using the pre-built user feedback collection and analysis module, and the feedback data is analyzed using a machine learning algorithm to evaluate the user's satisfaction and interest in the current interaction processing results, and output a user feedback analysis report, wherein the user feedback analysis report includes a user satisfaction score, a distribution of points of interest, and improvement suggestions, and the instant feedback data includes the user's reaction time, interaction operation, emotional expression, and comments or scores.

[0117] Instant feedback data includes user response time, such as how long it takes for the user to perform the next operation after the content is displayed. If the user performs the operation again soon, it may indicate that the user is not very interested in the current displayed content or wants to quickly view more content; interactive operations, such as whether the user has further viewed the internal structure of the tower, such as clicking to view a room or a specific item; emotional expression, which can be judged by the user's operation behavior on the interface or facial expression recognition (if there is relevant equipment support), such as whether the user has actively explored behavior or shows boredom; comments or ratings, if the user provides an evaluation of the tower content display, whether it is a text comment or a rating. Use machine learning algorithms to analyze these feedback data, evaluate the user's satisfaction and interest in the current interactive processing results, and output a user feedback analysis report. This report includes user satisfaction scores. If the score is high, it means that the user is relatively satisfied with the current tower content display; distribution of points of interest, such as finding that the user is more interested in a specific room or item in the tower; improvement suggestions, such as if the user is less satisfied, it may be recommended to add audio content about the history of the tower or optimize the display effect of the interactive animation.

[0118] Step S157, using a pre-built adaptive optimization loop mechanism to receive the user feedback analysis report output by the user feedback collection and analysis module, combined with the interaction data continuously collected by the real-time interaction monitoring module, and using a reinforcement learning algorithm to optimize the prediction model of the hotspot positioning and prediction unit, the decision rule set of the interaction decision engine, and the synchronization control logic of the content triggering and synchronization control unit.

[0119] For example, if the user feedback analysis report shows that many users are less satisfied with the display of tower content, the adaptive optimization loop mechanism will analyze whether it is because the hotspot positioning and prediction unit predicts the user's points of interest inaccurately, or there is a problem with the decision rules of the interactive decision engine, or the synchronous control logic of the content triggering and synchronization control unit is defective. If it is a problem with the hotspot positioning and prediction unit, it may be necessary to adjust the relevant weights in the user behavior pattern library or add new behavior patterns; if it is a problem with the interactive decision engine, modify the rules for tower-related interactions in the decision rule set; if it is a problem with the content triggering and synchronization control unit, optimize its synchronization control logic for tower content display, thereby continuously improving the accuracy and effectiveness of the entire system to automatically adjust the viewing angle and trigger the corresponding content according to the audio playback progress or user interaction operations, and enhance the user experience in the interaction of the castle panoramic material.

[0120] In other scenarios of the castle panoramic material, for example, when the audio is played to the content related to the castle's internal hall, the user operates in different areas, and the entire process is also carried out in the above manner. The real-time interactive monitoring module continuously tracks the progress information of the audio playing to the relevant part of the hall and the user's operation status information in the hall area or other areas. Based on this information and the user behavior pattern library, the hotspot positioning and prediction unit predicts the hotspot areas inside the hall that the user may pay attention to, such as murals in the hall or specific display stands. The interactive decision engine makes decisions based on the prediction results, the hotspot interactive space characteristics of the hall, and the triggerable interactive hot zone information. The perspective automatic adjustment module performs perspective switching, the content triggering and synchronization control unit triggers the corresponding audio or animation content, the user feedback collection and analysis module collects feedback data, and the adaptive optimization loop mechanism is optimized to ensure that a good interactive experience can be provided to users in each interactive scenario of the castle panoramic material.

[0121] Figure 2 The hardware structure of the panoramic interactive display system 100 based on AI multimodal fusion for implementing the panoramic interactive display method based on AI multimodal fusion provided by the embodiment of the present invention is shown as follows: Figure 2 As shown, the panoramic interactive display system 100 based on AI multimodal fusion may include a processor 110, a machine-readable storage medium 120, a bus 130 and a communication unit 140.

[0122] The machine-readable storage medium 120 may store data and / or instructions. In some embodiments, the machine-readable storage medium 120 may store data obtained from an external terminal. In some embodiments, the machine-readable storage medium 120 may store data and / or instructions that the panoramic interactive display system 100 based on AI multimodal fusion uses to execute or use to complete the exemplary method described in the present invention.

[0123] During the specific implementation process, one or more processors 110 execute computer executable instructions stored in the machine-readable storage medium 120, so that the processor 110 can execute the panoramic interactive display method based on AI multimodal fusion in the above method embodiment. The processor 110, the machine-readable storage medium 120 and the communication unit 140 are connected through the bus 130, and the processor 110 can be used to control the sending and receiving actions of the communication unit 140.

[0124] The specific implementation process of the processor 110 can refer to the various method embodiments executed by the panoramic interactive display system 100 based on AI multimodal fusion mentioned above. The implementation principles and technical effects are similar, and this embodiment will not be repeated here.

[0125] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer executable instructions are preset. When the processor executes the computer executable instructions, the panoramic interactive display method based on AI multimodal fusion as above is implemented.

[0126] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, various features are sometimes combined into one embodiment, drawing or description thereof.

Claims

1. A panoramic interactive display method based on AI multimodal fusion, characterized in that: The method comprises: Acquire first visual and audio combination data of the current panoramic material, and acquire multiple panoramic interaction hotspot information of the current panoramic material; Performing multimodal fusion according to the first visual and audio combination data and the plurality of panoramic interaction hotspot information to generate a panoramic interaction model of the current panoramic material; Extracting interactive spatial features from the panoramic interaction model of the current panoramic material to generate hotspot interactive spatial features of the current panoramic material; Matching the hotspot interactive spatial features of the current panoramic material with corresponding commentary audio or interactive animation to generate a triggerable interactive hot zone of the current panoramic material; Based on the hotspot interactive space characteristics of the current panoramic material and the triggerable interactive hot zones of the current panoramic material, according to the audio playback progress or user interactive operations, the viewing angle is automatically adjusted to the corresponding panoramic interactive hotspot, and the first interactive processing result of the corresponding commentary audio or interactive animation is triggered.

2. The panoramic interactive display method based on AI multimodal fusion according to claim 1 is characterized in that: The performing multimodal fusion according to the first visual and audio combination data and the plurality of panoramic interaction hotspot information to generate a panoramic interaction model of the current panoramic material includes: Using the first visual and audio combination data and the plurality of panoramic interaction hotspot information as model building elements of the panoramic interaction model; Determining the connection relationship between the model building elements of the panoramic interactive model according to the interaction relationship between the model building elements; Feature extraction is performed on the model construction elements to generate element feature vectors of the model construction elements of the panoramic interaction model.

3. The panoramic interactive display method based on AI multimodal fusion according to claim 2 is characterized in that: When the index tag of the model construction element is any one of the first visual and audio combination data, the second visual and audio combination data in the prior panoramic material, and the second interaction processing result in the prior panoramic material, the feature extraction representation of the model construction element to generate the element feature vector of the model construction element of the panoramic interaction model includes: Performing reciprocating cyclic timing coding on the interaction path to generate cyclic timing features of the interaction path; Encoding is performed according to the cyclic time series features of the interaction path to generate an element feature vector of the interaction path.

4. The panoramic interactive display method based on AI multimodal fusion according to claim 3 is characterized in that: The encoding and representing according to the cyclical time series features of the interaction path to generate the element feature vector of the interaction path includes: Determining index positioning features of the interaction path; fusing the cyclic timing features of the interaction path and the index positioning features of the interaction path to generate path features of the interaction path; Focus weights are allocated to the path features of the interaction path to generate an element feature vector of the interaction path.

5. The panoramic interactive display method based on AI multimodal fusion according to claim 2 is characterized in that: When the index tag of the model construction element is any one of the visual pictures corresponding to the prior panoramic material and the visual pictures corresponding to the current panoramic material, the feature extraction and representation of the model construction element to generate the feature vector of the model construction element of the panoramic interaction model includes: Performing semantic feature encoding on the material visual picture to generate visual picture semantic features of the material visual picture; The visual picture semantic features of the material visual picture are converted to generate an element feature vector of the material visual picture.

6. The panoramic interactive display method based on AI multimodal fusion according to claim 2 is characterized in that: When the index tag of the model construction element is an interactive hotspot corresponding to a priori panoramic material, the feature extraction representation of the model construction element to generate an element feature vector of the model construction element of the panoramic interaction model includes: Searching an interactive hot zone parameter index library according to the interactive hot zone corresponding to the prior panoramic material, wherein the interactive hot zone parameter index library includes mapping relationships between each of the interactive hot zones and each of the interactive hot zone features; The searched interactive hot zone feature is used as an element feature vector of the interactive hot zone corresponding to the prior panoramic material; When the index tag of the model construction element is the narration object corresponding to the prior panoramic material, the feature extraction and representation of the model construction element to generate the feature vector of the model construction element of the panoramic interaction model includes: Searching a narration object parameter index library according to the narration object corresponding to the prior panoramic material, wherein the narration object parameter index library includes mapping relationships between each of the narration objects and each of the narration object features; The searched narration object features are used as element feature vectors of the narration object corresponding to the priori panoramic material.

7. The panoramic interactive display method based on AI multimodal fusion according to claim 1 is characterized in that: The step of extracting interactive space features from the panoramic interaction model of the current panoramic material to generate hotspot interactive space features of the current panoramic material includes: Performing an architecture analysis on the panoramic interaction model to identify various components in the panoramic interaction model, wherein the various components include a portion related to the visual and audio combination data, a portion related to the panoramic interaction hotspot information, and a connection structure between the portion related to the visual and audio combination data and the portion related to the panoramic interaction hotspot information; According to each component of the panoramic interaction model, a structural decomposition strategy for presenting the internal structure of the model is generated, wherein the structural decomposition strategy is used to layer or partition the panoramic interaction model according to function, data flow or construction element type; According to the structural decomposition strategy, the interaction paths between the elements in the model structure after decomposing the panoramic interaction model are marked, wherein the interaction paths include the connection path from the first visual and audio combination data to the panoramic interaction hotspot information, the association path between different panoramic interaction hotspot information, and other logical connection paths between model construction elements; According to the marked interaction paths, the elements in the panoramic interaction model are grouped, and for each element group, the interaction relationship between the elements in the group and the interaction boundary with the elements in other groups are analyzed, thereby determining the interaction space scope involved in the group of elements; Within each determined interactive space, feature extraction is performed on the model elements therein to obtain an initial feature set within each interactive space. The basis for feature extraction includes the type of element, the functional role of the element in the interactive space, and the interactive relationship between the element and other elements. For visual elements, features are extracted from the dimensions of the composition of the visual picture and the distribution law of the visual elements; for audio elements, features are extracted from the dimensions of the frequency features, rhythm features, and synchronization relationship with the visual elements; for hotspot information elements, features are obtained from the dimensions of the triggering conditions of the hotspot and the interactive actions associated with the hotspot; Based on the predefined feature evaluation criteria, each feature in the initial feature set within each interactive space is evaluated, and screening and optimization are performed according to the evaluation results to generate a screened and optimized target feature set; the feature evaluation criteria are determined based on the importance of the feature to the hotspot interaction function of the panoramic interaction model, the correlation between the features, and the representativeness of the feature in the overall interaction logic; Performing spatial mapping on the target feature set within each interactive space to obtain a feature representation after spatial mapping, wherein the spatial mapping is performed based on a predefined spatial mapping rule, wherein the spatial mapping rule includes mapping based on a distance relationship between the feature and a key element in the interactive space and a spatial association between the feature and other features; The feature representations after spatial mapping within each interactive space are integrated to construct the hotspot interactive space features that describe the current panoramic material.

8. The panoramic interactive display method based on AI multimodal fusion according to claim 1 is characterized in that: The step of matching the hotspot interactive spatial features of the current panoramic material with corresponding commentary audio or interactive animation to generate a triggerable interactive hot zone of the current panoramic material includes: Identify different feature dimension information in the hotspot interactive space feature, and construct a matching relationship analysis framework for analyzing the hotspot interactive space feature and the commentary audio or interactive animation based on the different feature dimension information; For the audio commentary, feature analysis is performed from the aspects of content structure, semantic information, emotional tendency, and rhythm and rhyme of the audio commentary. In terms of content structure, the hierarchical structure of the audio commentary and the distribution of key contents are analyzed. In terms of semantic information, the concepts and descriptions related to the panoramic material contained in the audio commentary are mined. In terms of emotional tendency, the emotional atmosphere conveyed by the audio as a whole is judged. In terms of rhythm and rhyme, the speed of the rhythm and the regularity of the rhythm of the audio commentary are analyzed. For interactive animation, features are extracted from the visual element composition, action logic, interactive response mechanism, and visual rhythm of the animation. In terms of visual element composition, the graphics, colors, and objects included in the interactive animation are determined. In terms of action logic, the movement rules and logical relationships of the elements in the interactive animation are determined. In terms of interactive response mechanism, the interactive operations to which the interactive animation responds are determined. In terms of visual rhythm, the screen switching speed and the rhythm of element movement of the interactive animation are determined. According to the matching relationship analysis framework and the feature set of the narration audio and interactive animation obtained by analysis, a matching association index system is established. The matching association index system should include indicators of multiple dimensions, including content matching index, interaction logic matching index, and visual and audio coordination index. The content matching index is used to measure the matching degree between the semantic information in the hotspot interactive space features and the narration audio content and the content expressed by the interactive animation; the interaction logic matching index is used to evaluate the interaction triggering conditions in the hotspot interactive space features and the interactive response mechanism of the interactive animation and the guiding logic of the narration audio in the interactive process; the visual and audio coordination index is used to determine the coordination between the visual and audio association features in the hotspot interactive space features and the visual and audio in the narration audio and interactive animation; Based on the matching association index system, the hotspot interactive space features and the commentary audio and the interactive animation are matched and evaluated respectively. For each index, according to the evaluation method defined corresponding to the index, the matching score of the hotspot interactive space features and the commentary audio or the interactive animation on the index is calculated to obtain a preliminary matching result set between the hotspot interactive space features and the commentary audio and the interactive animation. The preliminary matching result set includes the scores of each matching pair on each index and the comprehensive score. The matching pairs include the hotspot interactive space features and the commentary audio, and the hotspot interactive space features and the interactive animation. The preliminary matching result set is screened according to the set screening criteria, and the screened matching pairs are sorted according to the comprehensive scores to obtain a sorted matching result set; Based on the sorted matching result set, the relationship between the hotspot interaction spatial features and the commentary audio or interactive animation in each matching pair is analyzed to determine a scoping rule set that can trigger the interactive hot zone; Based on the determined scoping rule set for the triggerable interaction hot zone, the triggerable interaction hot zone is marked in the hotspot interaction space feature of the current panoramic material, wherein, for each scoping rule, the area that meets the scoping rule is searched in the hotspot interaction space feature and marked, thereby identifying the triggerable interaction hot zone in the hotspot interaction space feature, thereby generating the triggerable interaction hot zone of the current panoramic material.

9. The panoramic interactive display method based on AI multimodal fusion according to claim 1 is characterized in that: The step of automatically adjusting the viewing angle to the corresponding panoramic interaction hotspot according to the hotspot interaction space characteristics of the current panoramic material and the triggerable interaction hot zone of the current panoramic material according to the audio playback progress or the user interaction operation, and triggering the first interaction processing result of the corresponding commentary audio or interactive animation includes: Use the pre-built real-time interaction monitoring module to continuously track the current progress information of audio playback and the real-time status information of user interaction operations, where the current progress information of audio playback includes the current playback timestamp, the current playback segment identifier, and the current playback content overview, and the real-time status information of user interaction operations includes the user operation type, the operation triggering timestamp, the operation target identifier, and the operation context description; The pre-built hotspot positioning and prediction unit receives the audio playback progress information and user interaction operation status information from the real-time interaction monitoring module as input information, and predicts the user's next point of interest or interaction intention by analyzing the input information, wherein the hotspot positioning and prediction unit maintains a user behavior pattern library, and the user behavior pattern library learns the behavior preferences and interaction patterns of different users in various situations based on historical interaction data. The hotspot positioning and prediction unit uses the user behavior pattern library to match the behavior characteristics of the current user, predicts the panoramic interaction hotspot that the current user is concerned about, and at the same time, infers the hotspot area related to the current playback content in combination with the audio playback progress, and outputs a predicted hotspot area set and a corresponding prediction confidence score; A pre-built interactive decision engine is used to receive the predicted hotspot area set and confidence score output by the hotspot positioning and prediction unit, as well as the hotspot interaction space characteristics and triggerable interactive hot zone information of the current panoramic material, and output a decision-making interactive instruction set, wherein the interactive instruction set includes a target hotspot identifier, an interaction type, a viewing angle adjustment parameter, and an execution timing. The interactive decision engine defines a decision rule set, thereby determining the next panoramic interactive hotspot to be displayed based on the decision rule set and input information, and the interaction type that triggers the commentary audio and / or interactive animation, and calculating the corresponding viewing angle adjustment scheme, wherein the viewing angle adjustment scheme includes a target viewing angle coordinate, an adjustment speed, and a smoothness parameter; The pre-built automatic viewing angle adjustment module is used to receive the viewing angle adjustment parameters and the target hotspot identifier in the interactive instruction set output by the interactive decision engine, and a smooth transition path from the current viewing angle to the target viewing angle is calculated according to the viewing angle adjustment parameters and the target hotspot identifier. At the same time, the viewing angle adjustment mechanism of the panoramic display system is controlled to perform viewing angle switching according to the calculated smooth transition path and viewing angle adjustment parameters, so that the user's viewing angle is aligned with the target panoramic interactive hotspot after the decision; The pre-built content trigger and synchronization control unit receives the target hotspot identifier, interaction type and execution timing in the interactive instruction set output by the interactive decision engine, and receives the view adjustment completion timestamp output by the view automatic adjustment module for analysis, and outputs the content display status information after the trigger. The content trigger and synchronization control unit maintains an accurate timer and synchronization control logic. For the commentary audio, the content trigger and synchronization control unit controls the audio playback system to start playing from the specified position. For the interactive animation, the content trigger and synchronization control unit starts the animation rendering and display process. After the content triggering and synchronization control unit executes content display, the pre-built user feedback collection and analysis module collects the user's immediate feedback data, uses the machine learning algorithm to analyze the feedback data, evaluates the user's satisfaction and interest in the current interactive processing results, and outputs a user feedback analysis report, which includes the user satisfaction score, interest point distribution, and improvement suggestions. The immediate feedback data includes the user's reaction time, interactive operation, emotional expression, and comments or scores; A pre-built adaptive optimization loop mechanism is used to receive the user feedback analysis report output by the user feedback collection and analysis module, combined with the interaction data continuously collected by the real-time interaction monitoring module, and a reinforcement learning algorithm is used to optimize the prediction model of the hotspot positioning and prediction unit, the decision rule set of the interaction decision engine, and the synchronization control logic of the content triggering and synchronization control unit.

10. A panoramic interactive display system based on AI multimodal fusion, characterized in that: The panoramic interactive display system based on AI multimodal fusion includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the panoramic interactive display method based on AI multimodal fusion as described in any one of claims 1 to 9 above.

Images