AR Glass Jingwen Annotation Method and Annotation System for Blogger Live Video

Abstract

The present invention discloses a method and a marking system for scenic text annotation of bloggers' live video AR glasses. The method specifically includes: dynamically adjusting the knowledge graph retrieval strategy of the tourism live broadcast scene through the hot topic set and the sentiment classification result to obtain a dynamic retrieval result; obtaining the blogger's head pose data based on the IMU unit of the AR glasses, combining the geographical positioning information, and constructing a 3D spatial topology map of the tourism live broadcast scene through the SLAM algorithm; performing spatial registration on the dynamic retrieval result and the 3D spatial topology map, and combining to generate a scene enhancement annotation layer through the NeRF algorithm; superimposing and displaying the video stream and the scene enhancement annotation layer, and dynamically adjusting the annotation visibility by using an adaptive transparency algorithm. The present invention realizes the dynamic annotation of the scenic text content of the AR glasses by collecting the video stream and the interactive data stream in real time, can timely respond to the changes in the live broadcast scene and the interactive needs of the audience, and improves the real-time performance and accuracy of the live broadcast annotation.

Description

Technical Field

[0001] The present invention relates to the field of AR glasses technology, and in particular to a method and system for annotating scenes and texts in AR glasses used in blogger live broadcast videos. Background Art

[0002] In today's digital age, the live streaming industry is booming. Travel live streaming, as an emerging form, has attracted a large audience with its uniquely immersive experience. Bloggers using AR glasses to broadcast travel videos can provide viewers with a more intuitive and vivid representation of travel scenes. However, while existing AR glasses landscape annotation systems for blogger live streams achieve a certain degree of multimodal interaction and dynamic scene reconstruction, they still face several pressing issues that limit the quality of the live streams and the viewing experience.

[0003] First, currently, interactive data such as comments and likes in live broadcasts contain rich real-time information about viewers' interests. However, existing technologies fail to fully tap into the value of this interactive data, simply displaying it as simple information without in-depth analysis of the changing trends in audience interests reflected behind it. For example, during a travel live broadcast, viewers may ask detailed questions about a specific attraction through comments or express strong interest in a particular attraction. However, the system is unable to capture this information in a timely manner and make corresponding adjustments. As a result, the scenery text annotations on the AR glasses do not match the actual needs of the audience, and the information that the audience is interested in cannot be accurately delivered.

[0004] Secondly, existing technologies have a significant lag in updating the annotation content. The system cannot dynamically adjust the annotation content based on the real-time interactive data in the live broadcast room, and often annotates according to preset rules and content. This results in the inability of the annotation content to keep up with the audience's interests or new hot topics during the live broadcast, making the information push out of touch with the audience's actual needs. For example, when a suddenly popular tourist attraction or a topic that sparks heated discussion among the audience appears in the live broadcast, the system cannot quickly integrate this information into the annotation content, affecting the audience's viewing experience.

[0005] Finally, existing technologies primarily focus on analyzing the preferences of AR glasses wearers, while ignoring the real-time behavioral patterns of audience groups in live tourism broadcasts. In live broadcast scenarios, audience group behavior often exhibits group and trend characteristics, such as the emergence of hot topics and shifts in collective interests. Existing technologies lack the ability to monitor and analyze audience group behavior, and are unable to capture changes in these group behaviors in a timely manner, making it impossible to optimize and adjust annotation content based on group behavior. For example, when most viewers are interested in a specific type of attraction, the system cannot automatically adjust the annotation focus to highlight this type of attraction, resulting in the live broadcast content failing to meet the needs of the group.

[0006] In addition, the emotional tendencies of the audience during the live broadcast, such as excitement, confusion, and affection, are crucial for enhancing the interactivity and attractiveness of the live broadcast. However, the existing technologies lack the effective ability to recognize the emotional tendencies of the audience and cannot provide corresponding emotional responses based on the emotional feedback of the audience. For example, when the audience shows excitement and affection for a certain scenic spot, the system cannot further enhance this emotional experience by adjusting the annotation content or display method; when the audience has doubts about a certain content, the system cannot give a clear explanation and annotation in a timely manner, affecting the viewing satisfaction of the audience. Summary of the Invention

[0007] The purpose of the present invention is to provide a method and a system for annotating the scene text of a blogger's live video with an AR glasses, which can dynamically annotate the scene text content of the AR glasses by collecting the video stream and the interactive data stream in real time, and can respond to the changes in the live broadcast scene and the interactive needs of the audience in a timely manner, improving the real-time performance and accuracy of the live broadcast annotation, so as to solve at least one of the above-mentioned problems of the existing technologies.

[0008] In a first aspect, the present invention provides a method for annotating the scene text of a blogger's live video with an AR glasses, and the method specifically includes:

[0009] Collect the video stream of the tourism live broadcast scene in real time through the built-in camera and the built-in sensor of the AR glasses, and obtain the interactive data stream in the live broadcast room in real time through the Apache Flink stream processing engine;

[0010] Based on the interactive data stream in the live broadcast room, obtain the set of hot topics and the emotional classification result by analyzing the bullet screen word frequency, the live broadcast time period, and the bullet screen text;

[0011] Based on the video stream of the live broadcast scene, dynamically adjust the knowledge graph retrieval strategy of the tourism live broadcast scene through the set of hot topics and the emotional classification result, and obtain the dynamic retrieval result;

[0012] Obtain the blogger's head pose data through the IMU unit of the AR glasses, and combine the geographic location information to construct a 3D spatial topology map of the tourism live broadcast scene through the SLAM algorithm;

[0013] Perform spatial registration on the dynamic retrieval result and the 3D spatial topology map, and combine to generate a scene enhancement annotation layer through the NeRF algorithm;

[0014] Overlay and display the video stream and the scene enhancement annotation layer, and dynamically adjust the annotation visibility by using the adaptive transparency algorithm.

[0015] In a second aspect, the present invention provides a system for annotating the scene text of a blogger's live video with an AR glasses, and the system specifically includes:

[0016] The first annotation module is used to collect the video stream of the tourism live broadcast scene in real time through the built-in camera and built-in sensors of the AR glasses, and obtain the interactive data stream of the live broadcast room in real time through the Apache Flink stream processing engine;

[0017] The second annotation module is used to obtain the set of hot topics and the sentiment classification result based on the interactive data stream of the live broadcast room by analyzing the barrage word frequency, live broadcast time period, and barrage text;

[0018] The third annotation module is used to dynamically adjust the knowledge graph retrieval strategy of the tourism live broadcast scene based on the video stream of the live broadcast scene through the set of hot topics and the sentiment classification result, and obtain the dynamic retrieval result;

[0019] The fourth annotation module is used to obtain the blogger's head pose data based on the IMU unit of the AR glasses, and combine the geolocation information to construct a 3D spatial topology map of the tourism live broadcast scene through the SLAM algorithm;

[0020] The fifth annotation module is used to perform spatial registration on the dynamic retrieval result and the 3D spatial topology map, and combine to generate a scene enhancement annotation layer through the NeRF algorithm;

[0021] The sixth annotation module is used to superimpose and display the video stream and the scene enhancement annotation layer, and dynamically adjust the annotation visibility using the adaptive transparency algorithm.

[0022] In a third aspect, the present invention provides a computer device, including: a memory, a processor, and a computer program stored on the memory. When the computer program is executed on the processor, it implements the blogger live video AR glasses scene text annotation method as described in any one of the above methods.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it implements the blogger live video AR glasses scene text annotation method as described in any one of the above methods.

[0024] Compared with the prior art, the present invention has at least one of the following technical effects:

[0025] 1. By collecting the video stream and interactive data stream in real time, dynamic annotation of the scene text of the AR glasses is realized, which can respond to the changes of the live broadcast scene and the interactive needs of the audience in a timely manner, and improve the real-time performance and accuracy of live broadcast annotation.

[0026] 2. Deeply analyze the interactive data such as barrages and likes in the live broadcast room, accurately capture the real-time interest changes of the audience, provide a strong basis for the optimization of the annotation content, and make the annotation content more in line with the needs of the audience.

[0027] 3. Dynamically adjust the annotation content according to the interaction data, ensuring that the live broadcast can timely reflect the audience's interests and hot topics during the live broadcast, avoiding the disconnection between information push and the audience's needs, and improving the timeliness and pertinence of the live broadcast.

[0028] 4. Pay attention to the real-time behavior patterns of the audience group, timely capture group behaviors such as the emergence of hot topics, and optimize the annotation content according to the group behavior, meeting the common needs of the group and enhancing the attractiveness and influence of the live broadcast.

[0029] 5. Identify the emotional tendencies of the audience and provide emotional responses according to the emotional feedback, enhancing the emotional interaction between the audience and the travel live broadcast and improving the audience's viewing satisfaction and loyalty.

[0030] 6. Segment the bullet screen data by time window and analyze the word frequency and text sentiment, which can accurately capture the hot topics and the audience's emotional tendencies in different live broadcast periods, providing a reliable basis for subsequent annotation optimization.

[0031] 7. Dynamically adjust the knowledge graph retrieval strategy by combining the hot topics and the results of emotional classification, which can screen out the scene entities that better meet the audience's interests and emotional needs, making the annotation content more targeted and attractive.

[0032] 8. Construct a 3D spatial topology map based on the IMU unit data and geographical location information, which can accurately restore the spatial structure of the travel live broadcast scene and provide a more intuitive spatial positioning basis for subsequent annotation.

[0033] 9. Generate a scene enhancement annotation layer through spatial registration and the NeRF algorithm, which can fuse the dynamic retrieval results with the 3D spatial information, realizing the deep combination of the annotation and the scene and enhancing the visual effect of the annotation.

[0034] 10. Adopt an adaptive transparency algorithm to dynamically adjust the annotation visibility, and comprehensively determine the transparency according to the blogger's fixation point, head movement and the importance of the annotation content, making the annotation display more in line with the blogger's viewing habits and improving the viewing experience.

[0035] 11. Combine the audience portrait data and the real-time feedback data, calculate the display weights of the annotation elements through the semantic similarity function and generate a rendering strategy matrix, which can realize personalized annotation display, meet the needs of different audiences, and improve the interactivity and audience satisfaction of the live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 It is a schematic flowchart of a method for annotating scene text of a blogger's live video AR glasses provided by an embodiment of the present invention;

[0038] Figure 2 It is a schematic structural diagram of a system for annotating scene text of a blogger's live video AR glasses provided by an embodiment of the present invention;

[0039] Figure 3 It is a schematic structural diagram of a computer device provided by an embodiment of the present invention. Detailed implementation manners

[0040] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented in order to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0041] It should be understood that when used in the specification and appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0042] It should also be understood that the term "and / or" as used in the specification and appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0043] As used in the specification and appended claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0044] In addition, in the description of the specification and appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0045] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that specific features, structures, or characteristics described in connection with that embodiment are included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized.

[0046] In an embodiment of this application, the execution subject of the process includes a terminal device. The terminal device includes but is not limited to: devices such as servers, computers, smartphones, and tablet computers that can execute the methods disclosed in this application. Figure 1 The flowchart of the blogger live video AR glasses scene text annotation method disclosed in an embodiment of the present invention is shown as follows and is described in detail:

[0047] S101, collect the video stream of the travel live scene in real time through the built-in camera and built-in sensor of the AR glasses, and obtain the interactive data stream in the live room in real time through the Apache Flink stream processing engine.

[0048] In this embodiment, through standardization processing and timestamp synchronization, the consistency and accuracy of the video stream and the interactive data stream are ensured, and the reliability of the analysis results is improved. The Apache Flink stream processing engine has good scalability and can easily handle the processing requirements of high-concurrency data streams, providing strong support for the dynamic annotation and personalized recommendation of travel live broadcasts.

[0049] S102, based on the interactive data stream in the live room, obtain the set of hot topics and the sentiment classification result by analyzing the barrage word frequency, live broadcast period, and barrage text.

[0050] In this embodiment, the interactive data is sliced according to the live broadcast period for subsequent analysis of the audience interaction in different time periods. The barrage text in each data slice is segmented, and the continuous text is cut into meaningful lexical units. A dictionary-based segmentation method or a statistics-based segmentation method can be used. The frequency of each lexical unit appearing in the barrage text is counted to form a word frequency statistical table. Focus on the words with higher frequencies, which may represent the current hot topics of concern to the audience.

[0051] Based on the word frequency statistics results, identify the keywords that may represent hot topics. Keywords can be screened out by setting a word frequency threshold or using topic extraction algorithms (such as TF-IDF algorithm). Aggregate the relevant keywords together to form a set of hot topics. Analyze the heat changes of hot topics in different live broadcast periods, and associate the hot topics with the corresponding live broadcast periods for storage, so as to dynamically adjust the annotation content according to the live broadcast period later.

[0052] Collect existing basic sentiment lexicons, including positive sentiment words, negative sentiment words and neutral sentiment words. These lexicons can be used as the basic basis for sentiment classification. Expand the sentiment lexicon for the field of travel live broadcast. Match the sentiment words in the bullet screen text of each data slice, and count the occurrence times of positive sentiment words, negative sentiment words and neutral sentiment words. Judge the overall sentiment tendency of the bullet screen text according to the occurrence times and sentiment tendency of the sentiment words. Count the number of bullet screen texts with different sentiment tendencies in each data slice to form the sentiment classification statistical results. Analyze the change trend of the sentiment classification results over time to understand the emotional fluctuations of the audience during the live broadcast.

[0053] In this embodiment, it is possible to analyze the interaction data in the live broadcast room in real time, capture hot topics and changes in the audience's emotions in a timely manner, and provide support for dynamically adjusting the annotation content.

[0054] S103, based on the video stream of the live broadcast scene, dynamically adjust the knowledge graph retrieval strategy of the travel live broadcast scene through the set of hot topics and the sentiment classification results to obtain dynamic retrieval results.

[0055] In this embodiment, image recognition technology (such as object detection algorithm based on deep learning) is used to identify the scene elements in the video frame, including scenic spot buildings, natural landscapes, people, signboards, etc. Combining natural language processing technology, semantic understanding is carried out on the identified scene elements and corresponding semantic labels are assigned. Associate the set of hot topics obtained by analyzing the interactive data stream in the live broadcast room with the scene element recognition results. Analyze the occurrence frequency and heat changes of hot topics in different live broadcast periods, and combine the appearance time and position information of scene elements in the video stream to further refine the association degree between hot topics and scene elements.

[0056] According to the sentiment classification results, assign corresponding sentiment weights to the scene elements. For example, if the audience shows positive emotions (such as excitement, love) towards a certain scene element (such as beautiful seaside scenery), a higher positive sentiment weight is assigned to this scene element; if the audience shows negative emotions (such as dissatisfaction, boredom) towards a certain scene element (such as crowded people in the scenic area), a lower sentiment weight or negative sentiment weight is assigned to this scene element.

[0057] Consider the change of sentiment tendency over time and dynamically adjust the sentiment weights of scene elements. For example, as the live broadcast progresses, the audience's interest in a scene element that was originally positive gradually decreases, and the sentiment weight also decreases accordingly; conversely, if a scene element suddenly triggers enthusiastic discussions and positive sentiment feedback from the audience, its sentiment weight is quickly increased.

[0058] Before the live broadcast starts, set the initial knowledge graph retrieval strategy according to the theme and common scenes of the travel live broadcast. The knowledge graph contains rich travel-related information, such as scenic spot introductions, history and culture, travel guides, surrounding food, etc. The initial retrieval strategy can be based on common travel information needs, such as sorting the retrieval according to the popularity and heat of scenic spots.

[0059] Dynamically adjust the retrieval direction of the knowledge graph according to the set of hot topics. For example, when the hot topic focuses on the history and culture of a specific scenic spot, place the retrieval focus on the knowledge graph nodes related to the history and culture of that scenic spot, such as the construction history of the scenic spot, important historical events, and relevant historical figures.

[0060] Combine the sentiment weights of scene elements to optimize the sorting of the knowledge graph retrieval results. For scene elements with higher sentiment weights, preferentially retrieve detailed information related to them and increase their display priority in the retrieval results.

[0061] As the live broadcast progresses, continuously adjust the knowledge graph retrieval strategy in real time according to new hot topics and sentiment classification results. For example, when a new hot topic appears in the live broadcast or the audience's sentiment tendency changes significantly, immediately re-evaluate the association relationships and sentiment weights of scene elements and update the retrieval strategy accordingly to ensure that the retrieval results always match the real-time interests and needs of the audience.

[0062] According to the dynamically adjusted knowledge graph retrieval strategy, retrieve information from the knowledge graph that is relevant to the current live broadcast scene, hot topics, and audience sentiment tendency. This information includes but is not limited to detailed introductions of scenic spots, historical and cultural backgrounds, travel guide suggestions, surrounding food recommendations, relevant stories and legends, etc. Integrate and screen the retrieved information, remove duplicate, irrelevant, or low-quality information, and retain the most valuable content that best meets the needs of the audience. Organize the integrated and screened information according to a certain logic and structure to generate dynamic retrieval results. The dynamic retrieval results can be designed according to different display forms, such as text descriptions, picture displays, video clips, audio explanations, etc., to meet the diverse viewing needs of the audience.

[0063] In this embodiment, by dynamically adjusting the knowledge graph retrieval strategy, a dynamic retrieval result is obtained, which provides strong support for the scene text annotation and content display of the travel live broadcast, and helps to improve the quality of the travel live broadcast and the viewing satisfaction of the audience. The dynamic retrieval result can be combined with the live broadcast scene and the display function of the AR glasses to provide a more immersive travel live broadcast experience for the audience.

[0064] S104. Based on the IMU unit of the AR glasses, obtain the head pose data of the blogger, and combine the geolocation information to construct a 3D spatial topology map of the travel live broadcast scene through the SLAM algorithm.

[0065] In this embodiment, the IMU (Inertial Measurement Unit) built into the AR glasses includes sensors such as an accelerometer, a gyroscope, and a magnetometer. The accelerometer is used to measure the acceleration of an object in three axial directions, the gyroscope is used to measure the angular velocity of an object around three axes, and the magnetometer is used to measure the earth's magnetic field strength to assist in determining the direction. Set the IMU unit to collect data at a relatively high frequency (for example, 100Hz) to ensure that the minute movements of the blogger's head can be captured in real time. The collected data is recorded in a specific format, such as including fields such as timestamps, acceleration values (in the X, Y, and Z directions), angular velocity values (in the X, Y, and Z directions), and magnetic field strength values (in the X, Y, and Z directions). Preprocess the collected head pose data, including operations such as denoising and filtering.

[0066] Adopt high-precision geolocation technologies, such as the Global Positioning System (GPS) combined with the Assisted Global Positioning System (A - GPS), or use indoor positioning technologies such as Wi - Fi positioning and Bluetooth beacon positioning (select the appropriate positioning method according to the different travel live broadcast scenarios). Through the positioning module integrated on the AR glasses, obtain the geographical location information of the blogger in real time, including longitude, latitude, altitude, etc. The positioning data also records timestamps for time synchronization with the head pose data. To improve the positioning accuracy, a fusion scheme of multiple positioning technologies can be adopted.

[0067] Since there may be a deviation in the acquisition clocks of the head pose data and the geolocation data, time synchronization is required. Adopt the method of hardware clock synchronization or software timestamp alignment to ensure that the head pose data and the geolocation data accurately correspond on the time axis. Control the time synchronization error within a small range (for example, millisecond level) to ensure the accuracy and consistency of the data during the subsequent processing of the SLAM algorithm.

[0068] Fuse the pre - processed head pose data and geolocation data. Adopt a loose - coupling or tight - coupling fusion strategy. The loose - coupling method is to use the head pose data and geolocation data respectively for independent pose estimation, and then fuse the results; the tight - coupling method is to directly input the head pose data and geolocation data into the same state estimator for joint optimization. Through data fusion, the high - frequency dynamic characteristics of the head pose data and the absolute position information of the geolocation data can be fully utilized to improve the accuracy and robustness of pose estimation.

[0069] According to the characteristics of the tourism live - broadcast scenario, select a suitable SLAM algorithm, such as a vision - based SLAM algorithm (VSLAM) combined with head pose data, or use a laser SLAM algorithm (if the AR glasses are equipped with a lidar sensor). At the beginning of the live broadcast, initialize the SLAM system. The initialization process includes selecting a suitable reference frame (usually the first - frame image at the start of the live broadcast), estimating the initial camera pose (combining geolocation information as the initial reference), and initializing the coordinate system and key parameters of the map.

[0070] Extract feature points from the real - time video frames captured by the built - in camera of the AR glasses. Commonly used feature extraction algorithms include SIFT (Scale - Invariant Feature Transform), SURF (Speeded - Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF), etc. The ORB algorithm has the advantages of fast calculation speed and good real - time performance, and is suitable for scenarios with high real - time requirements such as tourism live - broadcasts. Match the feature points extracted from the current frame with those of the previous frame to determine the relative motion relationship of the camera at different times. Use brute - force matching or fast approximate nearest - neighbor (FLANN) matching algorithms for feature matching, and remove incorrect matching point pairs through the random sample consensus (RANSAC) algorithm to improve the accuracy of matching.

[0071] Combine the head pose data and the results of feature matching to estimate the current pose of the camera. The head pose data provides the incremental motion information of the camera, and feature matching provides the relative pose constraints between cameras. Optimally estimate the pose of the camera through an extended Kalman filter (EKF) or a non - linear optimization algorithm (such as the graph optimization algorithm g2o) to obtain more accurate pose information. According to the estimated camera pose, add the extracted feature points to the map to construct a 3D spatial point - cloud map. At the same time, filter and optimize the point cloud in the map to remove redundant point - cloud data and improve the quality and storage efficiency of the map. For example, use a voxel grid filtering algorithm to downsample the point cloud to reduce the number of point clouds.

[0072] During the live broadcast, continuously detect whether the camera has returned to a previously passed position, that is, perform loop detection. Loop detection can correct the pose errors accumulated during the SLAM process and improve the global consistency of the map. The bag-of-words model or deep learning method is used for loop detection, and it is judged whether a loop occurs by comparing the feature descriptors of the current frame and historical frames. When a loop is detected, the loop information is used to optimize the global map and the camera pose. Through the graph optimization algorithm, the pose constraints and loop constraints during the entire SLAM process are jointly optimized to further reduce the pose error and obtain a more accurate 3D spatial topology map.

[0073] In this embodiment, by combining the IMU unit data and the geolocation information and adopting an advanced SLAM algorithm, a high-precision 3D spatial topology map can be constructed, which accurately reflects the spatial structure and layout of the tourist live broadcast scene. The constructed 3D spatial topology map provides a basis for scenic text annotation and virtual content fusion, and can provide a more intuitive, vivid, and immersive tourist live broadcast experience for the audience, meeting the audience's needs for obtaining tourist information and visual enjoyment.

[0074] S105, perform spatial registration on the dynamic retrieval result and the 3D spatial topology map, and combine to generate a scene enhancement annotation layer through the NeRF algorithm.

[0075] In this embodiment, the dynamic retrieval result obtained after dynamically adjusting the tourist live broadcast scene knowledge graph retrieval strategy based on the live broadcast scene video stream, the set of hot topics, and the sentiment classification result is parsed in detail. The dynamic retrieval result usually contains various information related to the current live broadcast scene, such as detailed introductions of scenic spots, historical and cultural backgrounds, content related to real-time hot topics, etc., and this information exists in a structured or semi-structured form. The parsed information is classified and sorted according to certain rules, for example, divided according to dimensions such as scenic spots and topics, so as to be registered with the 3D spatial topology map later.

[0076] Extract features from the 3D spatial topology map of the tourist live broadcast scene constructed by the SLAM algorithm. The 3D spatial topology map contains the spatial structure information of the live broadcast scene, such as the positions and shapes of buildings, the distribution of scenic spots, etc. Key feature points, feature surfaces, or feature regions can be extracted from the map, such as the vertices of buildings, the central points of scenic spots, etc. as reference features for registration. At the same time, the map is gridded, and the map is divided into multiple small grid units to facilitate subsequent matching and registration operations.

[0077] Adopt a feature-based registration strategy to match the information in the dynamic retrieval results with the features in the 3D spatial topological map. For example, for a specific scenic spot mentioned in the dynamic retrieval results, search for the corresponding feature area in the 3D spatial topological map. The matching can be carried out by calculating the similarity between features, such as the distance between feature points, the shape similarity of feature surfaces, etc. When the matching features are found, determine the spatial position of the scenic spot information in the dynamic retrieval results in the 3D spatial topological map.

[0078] In addition to feature-based registration, semantic association is also considered for registration. Analyze the semantic information in the dynamic retrieval results, such as the type of scenic spot, historical and cultural background, etc., and associate it with the semantic information in different regions of the 3D spatial topological map. For example, if an ancient building scenic spot with a profound historical and cultural heritage is mentioned in the dynamic retrieval results, then search for the region related to history and culture in the 3D spatial topological map and register the information of this scenic spot to the corresponding position. By combining feature matching and semantic association, the accuracy and reliability of registration are improved.

[0079] Then, perform initial registration to quickly match some obvious feature information in the dynamic retrieval results with the corresponding features in the 3D spatial topological map and determine a rough registration position. For example, match the position of the landmark scenic spot mentioned in the dynamic retrieval results with the center point of this scenic spot in the 3D spatial topological map as the benchmark for initial registration. On the basis of initial registration, perform fine registration. Adopt the idea of the Iterative Closest Point (ICP) algorithm to continuously adjust the relative position and pose between the dynamic retrieval results and the 3D spatial topological map to minimize the feature matching error between the two. Through multiple iterations, gradually optimize the registration results until a satisfactory registration accuracy is achieved.

[0080] Visualize the registered dynamic retrieval results and the 3D spatial topological map, and verify the accuracy of registration through manual observation. Check whether the information in the dynamic retrieval results is accurately located at the corresponding position in the 3D spatial topological map and whether there are obvious deviations or errors. If errors are found in the registration results, analyze the errors. It may be caused by inaccurate feature extraction, unreasonable parameter settings of the registration algorithm, etc. According to the results of error analysis, optimize and adjust the feature extraction method, registration algorithm parameters, etc., and perform the registration operation again until the registration results meet the requirements.

[0081] The NeRF (Neural Radiance Fields) algorithm is a neural network-based method for scene representation and rendering. It represents a scene as a continuous 5D function (taking spatial position and viewing direction as inputs and color and density as outputs), and uses a neural network to learn the geometric and appearance information of the scene. During training, a large number of scene images from different viewpoints and corresponding camera parameters are input. The neural network learns these data to predict the color and density at any position and viewpoint in the scene, thus achieving high-quality scene rendering. In the tourism live broadcast scenario, the NeRF algorithm can be used to generate a more realistic and delicate scene enhancement annotation layer. The tourism live broadcast scenario usually contains rich landscapes and details, and traditional annotation methods may not be able to present these details well. The NeRF algorithm can learn the three-dimensional structure and appearance features of the scene based on the video stream data of the live broadcast scenario, generate a scene rendering result with high realism, and provide richer visual information for scene text annotation.

[0082] Through the built-in camera of the AR glasses, video streams of the tourism live broadcast scenario are collected from multiple different viewpoints. These viewpoints should cover as many parts of the live broadcast scenario as possible to provide rich scene information. During the collection process, camera parameters such as the position, orientation, and focal length of the camera are recorded. Operations such as denoising and color correction are performed on the video frames to improve the image quality. Then, the video frames are classified and sorted according to the viewpoints to construct a training dataset. At the same time, the camera parameters are normalized to meet the input requirements of the NeRF algorithm. Design a NeRF model architecture suitable for the tourism live broadcast scenario. Generally, the NeRF model consists of multi-layer perceptrons (MLPs) for learning the 5D function of the scene. Parameters such as the number of model layers and the number of neurons can be adjusted according to the characteristics of the tourism live broadcast scenario to improve the model's performance and training efficiency. Input the preprocessed training dataset into the NeRF model for training. During training, optimization algorithms such as stochastic gradient descent are used to continuously adjust the model's parameters to minimize the error between the color and density predicted by the model and the real scene images. Through multiple iterative trainings until the model converges, a NeRF model that can accurately represent the tourism live broadcast scenario is obtained.

[0083] Fuse the annotation information in the spatially registered dynamic retrieval results with the scene rendering results generated by the NeRF model. The annotation information can include text information such as the name, introduction, historical and cultural background of scenic spots, as well as some simple graphical annotations, such as arrows, icons, etc. Display the text information at the corresponding positions in the scene rendering results with appropriate fonts and colors, and associate the graphical annotations with the objects in the scene to make them blend naturally with the scene. Optimize the visual effect of the enhanced annotation layer of the fused scene. Adjust parameters such as the transparency and color contrast of the annotation information so that it neither affects the realism of the scene nor can be clearly presented to the audience. At the same time, consider the layout and hierarchy of the annotation information to avoid overcrowding or overlapping of the annotation information and improve the readability and aesthetics of the annotation layer.

[0084] In this embodiment, by spatially registering the dynamic retrieval results with the 3D spatial topology map, the scenic text annotation information can be accurately located at specific positions in the live broadcast scene, avoiding the problem of mismatch between the annotation information and the actual scene and improving the accuracy of annotation. Combine the NeRF algorithm to generate an enhanced annotation layer for the scene. Utilize the high-fidelity rendering ability of the NeRF algorithm to provide a richer and more delicate visual background for the scenic text annotation, enabling the audience to obtain a more immersive experience when watching the live broadcast. It can process the data during the live broadcast in real time, dynamically adjust the dynamic retrieval results and the enhanced annotation layer for the scene according to the interaction data in the live broadcast room and the changes in the live broadcast scene, meeting the real-time and dynamic requirements of tourism live broadcasts. The combination of accurate annotation and realistic scene rendering can better meet the audience's needs for tourism information, improve the audience's viewing satisfaction and participation, and bring better interaction effects to tourism live broadcasts.

[0085] S106, superimpose and display the video stream and the enhanced annotation layer for the scene, and dynamically adjust the annotation visibility using an adaptive transparency algorithm.

[0086] In this embodiment, the enhanced annotation layer for the scene contains annotation information generated based on the dynamic retrieval results and the 3D spatial topology map, such as text information like the name, introduction, historical and cultural background of scenic spots, as well as some graphical annotations, such as arrows, icons, etc. Perform format conversion and optimization processing on the enhanced annotation layer for the scene to make it match the format and size of the video frames. Display the video stream and the enhanced annotation layer for the scene in a layered superimposed manner. Use the video frame as the bottom-layer image and the enhanced annotation layer for the scene as the upper-layer image. During the display process, ensure that the upper-layer annotation layer can accurately cover the corresponding positions of the bottom-layer video frame to achieve the precise correspondence between the annotation information and the actual scene.

[0087] Reasonably plan the display area according to the importance and type of the annotation information. Place important annotation information, such as the core introduction of scenic spots, in prominent positions on the video frame, such as the central area of the picture or positions closely related to the scenic spots; for some auxiliary annotation information, such as arrow direction indicators, they can be placed on the edge of the picture or in positions that do not affect the display of the main scene. At the same time, consider the layout and spacing between annotation information to avoid overcrowding or overlapping of annotation information, which may affect the viewing experience of the audience.

[0088] During the overlay display process, ensure the frame synchronization between the video frame and the scene enhancement annotation layer. That is, each video frame should be accurately overlaid with the corresponding annotation layer. Frame synchronization can be achieved through timestamps or other synchronization mechanisms to ensure that during playback, the annotation information can be accurately updated and displayed as the video frame switches. Use graphics rendering technology to render the overlaid image. During the rendering process, consider factors such as color fusion and lighting effects of the image to make the annotation information blend naturally with the video frame and form a unified visual picture. Finally, output the rendered image to the display device for the audience to view.

[0089] By analyzing the interaction data in the live broadcast room, such as the number of bullet screens, the number of likes, and the content of comments, evaluate the audience's interest in the current annotation information. If the audience shows a high level of interest in a certain annotation information, for example, a large number of audiences discuss the scenic spot corresponding to the annotation in the bullet screen, then appropriately reduce the transparency of the annotation information to make it more prominently displayed and attract the audience's attention; on the contrary, if the audience's interest in a certain annotation information is low, then appropriately increase its transparency to reduce the interference to the audience's viewing of the main scene.

[0090] Consider the impact of the complexity of the tourism live broadcast scene on the visibility of annotations. In a relatively simple scene with less information, the transparency of the annotation information can be appropriately reduced to make it more clearly visible; while in a complex scene with a large amount of information, such as a scene with dense scenic spots and a large number of people, the transparency of the annotation information can be appropriately increased to avoid the annotation information being too prominent and covering up the details of the main scene.

[0091] Record the viewing duration of the audience for each annotation information. If the audience has a relatively long viewing duration for a certain annotation information, it indicates that the audience is more concerned about this information. At this time, the transparency can be reduced to further enhance the display effect of the information; if the viewing duration is short, then increase the transparency to reduce the interference to the audience.

[0092] Assign corresponding weights to the above different adjustment factors. For example, the audience interest may be given a relatively high weight because it is a key factor directly affecting the audience's viewing experience; the scene complexity and viewing duration are assigned relatively low weights according to the actual situation. Through reasonable weight assignment, comprehensively consider the influence of various factors on the transparency of annotation information.

[0093] Set a reasonable range for the transparency of the annotation information, for example, from 0% (completely opaque) to 100% (completely transparent). Calculate the target transparency value of each annotation information in the current situation according to different adjustment factors and weights, and ensure that this value is within the set transparency range.

[0094] During the live broadcast, monitor information such as the interaction data in the live broadcast room, the scene complexity, and the viewing duration of the audience in real time. Calculate the target transparency value of each annotation information in real time according to the set adjustment strategy and weights. For example, analyze the interaction data at regular intervals (such as every 1 second), update the evaluation result of the audience interest, and recalculate the transparency of the annotation information.

[0095] To avoid discomfort to the audience caused by sudden changes in the transparency of the annotation information, adjust the transparency in a smooth transition manner. For example, when the target transparency value changes, instead of immediately adjusting the transparency to the target value, gradually transition to the target value within a certain time interval (such as 0.5 seconds), making the change of transparency more natural and smooth.

[0096] Through the interaction functions in the live broadcast room, such as bullet screens and comments, collect the feedback of the audience on the adjustment effect of the transparency of the annotation information. Understand whether the audience thinks that the display of the annotation information is clearer and more reasonable, and whether it has a positive impact on the viewing experience. Optimize and adjust the adaptive transparency algorithm according to the audience feedback and the actual adjustment effect.

[0097] In this embodiment, by accurately superimposing and displaying the video stream and the scene enhancement annotation layer, and using the adaptive transparency algorithm to dynamically adjust the annotation visibility, the annotation information can be more clearly and accurately displayed in front of the audience, improving the efficiency and accuracy of information transmission. Accurate and clear annotation information and reasonable transparency adjustment can attract the attention of the audience, stimulate the enthusiasm of the audience to participate, promote the interaction between the audience and the anchor, and improve the interactivity and attractiveness of the live broadcast. It can automatically adjust the display mode of the annotation information according to different tourism live broadcast scenarios and the characteristics of the audience group, with strong adaptability and flexibility, and can meet the needs of various tourism live broadcast scenarios.

[0098] In some embodiments, in the above step S102, based on the interaction data stream in the live broadcast room, by analyzing the bullet screen word frequency, the live broadcast time period, and the bullet screen text, obtaining the hot topic set and the sentiment classification result specifically includes:

[0099] Extract the bullet screen data set from the interaction data stream in the live broadcast room, perform time window segmentation on the bullet screen data set, and divide it into several bullet screen data subsets according to the live broadcast time period;

[0100] Perform word frequency statistics on the subset of barrage data within each time window, calculate the heat weight value of each word, and obtain the word frequency vector statistics result;

[0101] Based on the word frequency vector statistics, a hot topic set is generated through the improved LDA topic model;

[0102] The sentiment polarity of the barrage text in the barrage data subset within each time window is classified, and the RoBERTa-base model is used to calculate the sentiment score of each barrage to obtain the sentiment classification result.

[0103] In this embodiment, the interactive data stream is parsed to extract the data fields related to the bullet comment and construct a bullet comment dataset. During the parsing process, the data needs to be cleaned to remove invalid data and noise data.

[0104] Set an appropriate time window size based on the characteristics of the live broadcast and business needs. The size of the time window should take into account factors such as the live broadcast's rhythm, the frequency of topic changes, and computing resource limitations. If the time window is set too small, the amount of data within each window may be insufficient, making it difficult to accurately reflect hot topics and sentiment trends. If the time window is set too large, it may not be able to capture changes in topics and sentiment in a timely manner.

[0105] The bullet comment dataset is segmented according to the specified time window size. The bullet comment data within each time window is extracted to form an independent bullet comment data subset. A word segmentation tool suitable for Chinese text, such as Jieba, is used. Word segmentation is performed on the bullet comment data subset within each time window, breaking the continuous bullet comment text into individual terms. During the word segmentation process, stop words are also filtered. Stop words are words that appear frequently in the text but contribute little to the semantic understanding of the text, such as "的," "了," and "是." Removing stop words reduces the data volume and improves the accuracy and efficiency of subsequent word frequency statistics.

[0106] For each subset of barrage data within a time window, count the number of times each term appears. A data structure such as a hash table can be used to record term frequency information, with the term as the key and the number of occurrences as the value. The term frequency statistics for each time window are stored for subsequent analysis and processing. The results can be stored in a database or saved as a file to ensure data traceability and queryability. When calculating the heat weight value of a term, in addition to term frequency, other factors can also be considered, such as the position of the term in the barrage text and whether it is a new term.

[0107] Considering multiple factors, calculate the popularity weight value for each term. The weighted average method can be used. According to the importance of different factors, assign corresponding weights, and then sum the scores of each factor after weighting to obtain the final popularity weight value of the term. Combine all terms and their corresponding popularity weight values within each time window to form the term frequency vector statistical result. This vector can intuitively reflect the importance of each term within this time window and provide a data basis for subsequent hot topic generation.

[0108] The LDA (Latent Dirichlet Allocation) topic model is a commonly used text topic modeling method. It assumes that a document is generated by a mixture of multiple topics, and each topic is composed of the probability distributions of multiple terms. Through learning a large number of documents, the LDA model can discover the latent topic structure in the documents. Traditional LDA models may have some limitations when dealing with live barrage data, such as poor performance in processing short texts and inability to fully consider time factors. Therefore, it is necessary to improve the LDA model. The improvement directions can include introducing time dimension information to enable the model to capture the changes of topics over time, optimizing the method for determining the number of topics to improve the accuracy of topic mining, etc.

[0109] Convert the term frequency vector statistical result within each time window into a format suitable for LDA model training. Usually, the data needs to be converted into the form of a document-term matrix, where each row represents a time window (which can be regarded as a document), each column represents a term, and the element in the matrix represents the popularity weight value of the term within this time window. Determine some key parameters of the LDA model, such as the number of topics K, hyperparameters α and β, etc. The number of topics K can be determined by some heuristic methods or methods based on cross-validation; the hyperparameters α and β can be set according to experience or experiments, and they affect the distributions of topics and terms.

[0110] Use the prepared training data to train the improved LDA topic model. During the training process, the model will continuously adjust the distribution parameters of topics and terms to enable the model to better fit the training data. After training, extract each topic from the model. Each topic consists of a group of terms with relatively high probabilities, and these terms can reflect the core content of the topic. According to the meaning and context information of the terms, interpret and name each topic to obtain the set of hot topics.

[0111] RoBERTa (Robustly Optimized BERT Pretraining Approach) is a pretrained language model based on the Transformer architecture. It incorporates a series of improvements over BERT, such as larger batch sizes and longer training sequences, to enhance model performance and generalization. The RoBERTa-base model is a foundational model in the RoBERTa family, featuring a smaller model size and faster inference speed, making it suitable for use in scenarios with high real-time requirements. Compared to traditional sentiment classification methods, the RoBERTa-base model better understands the semantic information of text and captures the emotional tendencies within it. Pretrained on large-scale text data, it acquires rich linguistic knowledge and semantic representations, enabling it to accurately process a variety of complex emotional expressions.

[0112] Collect a certain amount of live broadcast comment text data and perform sentiment annotation. Sentiment annotations are typically categorized into three categories: positive, negative, and neutral. Manual annotation can be used to annotate comment text, and consistency checks and other methods can be used to ensure the accuracy of the annotation results. Divide the annotated dataset into training, validation, and test sets. The training set is used for model training, the validation set is used to adjust model hyperparameters and prevent overfitting, and the test set is used to evaluate the final model performance.

[0113] Fine-tune the RoBERTa-base model using the prepared training set. During fine-tuning, replace the model's output layer with a classifier suitable for sentiment classification (such as a softmax classifier), and adjust the model's parameters to achieve good performance on the sentiment classification task. For each barrage text in the barrage data subset within each time window, input it into the fine-tuned RoBERTa-base model. The model outputs a probability value for each barrage text belonging to one of three sentiment categories: positive, negative, or neutral. The probability of positive sentiment is used as the sentiment score for the barrage text (ranging from 0 to 1, with higher scores indicating more positive sentiment). Based on the sentiment score of each barrage text, its sentiment polarity is determined. For example, a threshold can be set: if the positive sentiment score exceeds the threshold, the barrage text is classified as positive; if the negative sentiment score exceeds the threshold, the barrage text is classified as negative; otherwise, it is classified as neutral. The sentiment classification results of all barrage texts within each time window are aggregated to obtain the sentiment classification result for that time window.

[0114] This example effectively analyzes hot topics and audience sentiment during live broadcasts, providing strong support for optimizing and personalizing travel live broadcasts. Based on audience sentiment and the hot topics they care about, personalized live content recommendations and annotations are provided.

[0115] In some embodiments, in step S103 above, for the video stream based on the live broadcast scenario, the knowledge graph retrieval strategy for the tourism live broadcast scenario is dynamically adjusted through the hot topic set and the sentiment classification result to obtain a dynamic retrieval result, which specifically includes:

[0116] Extract the shooting direction angle and timestamp in the video stream of the live broadcast scenario, and generate a set of scene entities through the target detection model. The set of scene entities includes several scene entities and their bounding box coordinates and class labels;

[0117] Based on the set of scene entities, combine the hot topic set and the sentiment classification result to construct a dynamic retrieval weight vector;

[0118] Based on the dynamic retrieval weight vector, screen out the first set of scene entities from the pre-set knowledge graph retrieval strategy;

[0119] According to the associated historical events and their association degrees of each first scene entity in the first set of scene entities, and the overlap rate between the bounding box of each first scene entity and the visual attention area of the blogger, calculate the historical relevance score of each first scene entity in the first set of scene entities;

[0120] Based on the historical relevance score, screen out the second set of scene entities from the first set of scene entities to obtain a dynamic retrieval result.

[0121] In this embodiment, in the video stream data, each frame of image is accompanied by corresponding timestamp information. By parsing the encapsulation format of the video stream (such as MP4, FLV, etc.), the timestamp of each frame of image is extracted. The timestamp is used to identify the specific time position of the video frame during the live broadcast, providing a time reference for subsequent scene analysis and retrieval. The shooting direction angle information is obtained by using the sensors built in the AR glasses (such as gyroscopes, accelerometers, etc.). These sensors can monitor the orientation and angle changes of the camera in real time and associate the obtained direction angle data with the video frame.

[0122] Select an object detection model suitable for live streaming scenarios, such as the YOLO (You Only Look Once) series of models or the Faster R-CNN model. These models offer high detection speed and accuracy, enabling rapid identification of various scene entities in live video streams. The extracted video frames are fed into the object detection model for inference. The model analyzes the video frames, detects various objects within them, and generates bounding box coordinates and category labels for each object. For example, in a travel live stream, the model might detect scene entities such as buildings, people, and natural landscapes, annotating their locations (bounding box coordinates) and categories (e.g., "ancient building," "tourist," "mountain," etc.) within the video frame. All scene entities detected by the model, along with their corresponding bounding box coordinates and category labels, are aggregated to form a scene entity collection. Each scene entity is represented as a data entry in the collection, containing information such as the entity's category and the coordinates of its upper-left and lower-right bounding box corners.

[0123] Conduct in-depth analysis of the generated hot topic collection to understand the scene entity categories associated with each hot topic. For example, if the hot topic is "features of a certain ancient building," then related scene entity categories may include "ancient building" and "architectural decoration." At the same time, count the frequency and popularity of each hot topic during the live broadcast to measure its importance to the current live broadcast content.

[0124] Based on the sentiment classification results, analyze the audience's emotional tendencies towards different scene entities. For example, if the audience's comments about a certain attraction are mostly positive, the weight of the scene entities related to that attraction can be appropriately increased in the search. Count the number of comments with different sentiment categories (positive, negative, neutral) and their degree of association with each scene entity.

[0125] Based on the importance of hot topics, assign higher initial weights to the scene entity categories associated with them. For example, the scene entity categories associated with the current most popular hot topic can be given a higher base weight. Also, consider the frequency of hot topics; the higher the frequency, the higher the weight of the related scene entity categories can be appropriately increased.

[0126] Based on the sentiment classification results, the weights of scene entity categories are dynamically adjusted. For scene entity categories with positive audience sentiment, their weights are increased; for scene entity categories with negative sentiment, their weights are decreased. For example, if a scene entity category receives a high positive sentiment score in the comments, its weight can be increased by a certain amount based on the initial weight.

[0127] Based on the comprehensive calculation of the weights of hot topics and sentiment classification, the final weight of each scenario entity category is obtained. The weighted average method can be used. According to the influence degree of hot topics and sentiment classification on the retrieval results, corresponding weight coefficients are assigned to them, and then the weights of the two are weighted and summed to obtain the comprehensive weight of each scenario entity category. Combine each scenario entity category and its corresponding comprehensive weight to construct a dynamic retrieval weight vector. The dimension of this vector is equal to the number of scenario entity categories, and the value on each dimension represents the weight of the corresponding scenario entity category.

[0128] Comprehensively sort out the pre-set knowledge graph retrieval strategy and understand various retrieval rules and conditions it contains. For example, the retrieval strategy may stipulate the retrieval methods based on scenario entity categories, attributes, relationships, etc., as well as the priorities of different retrieval conditions.

[0129] Evaluate the applicability of the pre-set knowledge graph retrieval strategy in the current live broadcast scenario. Considering the characteristics of the live broadcast (such as real-time and dynamic) and the needs of users (such as obtaining information related to hot topics and sentiment tendencies), judge which retrieval rules and conditions need to be adjusted or retained. According to the dynamic retrieval weight vector, adjust the rules in the pre-set knowledge graph retrieval strategy. Use the adjusted knowledge graph retrieval strategy to retrieve and filter the scenario entity set. According to the retrieval rules, find the scenario entities that match the current retrieval conditions from the scenario entity set to form the first scenario entity set. Mine the associated historical events of each first scenario entity in the first scenario entity set through the knowledge graph. The knowledge graph stores rich entity relationships and event information, and can query according to the identifier of the entity in the knowledge graph to obtain the historical events related to this entity.

[0130] Analyze the degree of association between each historical event and the current live hot topic and sentiment tendency. Manual rule - definition or semantic - analysis - based methods can be used to calculate the degree of association. Utilize the blogger's visual attention area detection technology to determine the visual attention area of the blogger during the live broadcast. By analyzing information such as the blogger's eye direction and head posture, and combining with the layout of the live - broadcast screen, the area of the screen that the blogger is currently focusing on can be roughly determined. Calculate the overlap rate between the bounding box of each first - scene entity and the blogger's visual attention area. Consider the bounding box of the scene entity and the visual attention area as rectangular regions on a two - dimensional plane, and obtain the overlap rate by comparing the ratio of their overlapping area to their respective areas. Considering the associated historical events and their degrees of association of each first - scene entity, as well as the overlap rate between the bounding box and the blogger's visual attention area, formulate a historical relevance scoring rule. According to the scoring rule, calculate the historical relevance score for each first - scene entity in the first - scene entity set. According to the distribution of the historical relevance scores and business requirements, set an appropriate scoring threshold. Statistical analysis methods can be used to calculate statistics such as the average value and standard deviation of the historical relevance scores, and then determine the threshold based on these statistics. According to the set scoring threshold, screen out the first - scene entities with historical relevance scores higher than the threshold from the first - scene entity set to form a second - scene entity set. These entities are considered to be scene entities highly relevant to the current live hot topic, sentiment tendency, and the blogger's attention area. Integrate and output the second - scene entity set as the dynamic retrieval result.

[0131] In this embodiment, the dynamic optimization of the knowledge - graph retrieval in the tourism live - broadcast scenario is realized, improving the relevance and practicality of the retrieval results.

[0132] In some embodiments, in the above - mentioned step S104, the IMU unit based on the AR glasses obtains the blogger's head - posture data, and combines the geolocation information to construct a 3D spatial topology map of the tourism live - broadcast scenario through the SLAM algorithm. Specifically, it includes:

[0133] The IMU unit based on the AR glasses obtains the blogger's head - posture data, and the head - posture data includes three - axis angular velocity and three - axis acceleration;

[0134] Based on the head - posture data, perform quaternion multiplication on the three - axis acceleration using the quaternion differential equation, and combine with the three - axis acceleration for gravity - deviation compensation to construct a head - posture matrix;

[0135] Obtain the geolocation information of the tourism live - broadcast scenario, input the geolocation information and the head - posture matrix into a tightly - coupled SLAM framework, and realize multi - source data fusion through an extended Kalman filter to obtain fusion feature data;

[0136] Based on the fused feature data, the ORB-SLAM3 framework is used to construct a 3D scene topological map, and the map structure is optimized through a topological consistency detection algorithm.

[0137] In this embodiment, an IMU (Inertial Measurement Unit) with high precision and low latency characteristics is selected and integrated into the AR glasses. The IMU usually includes a three-axis gyroscope and a three-axis accelerometer, which can measure the three-axis angular velocity and the three-axis acceleration respectively.

[0138] According to the real-time requirements of the tourism live broadcast scenario, the data acquisition frequency of the IMU unit is set. Generally speaking, a higher acquisition frequency can provide more accurate head pose change information, but it will also increase the data processing burden. The three-axis angular velocity and three-axis acceleration data collected by the IMU unit are transmitted to the data processing module through the communication bus (such as I²C, SPI, etc.) inside the AR glasses.

[0139] Quaternion is a mathematical tool used to represent the rotation in three-dimensional space. Compared with Euler angles and rotation matrices, it has the advantages of high calculation efficiency and avoiding gimbal lock. In this embodiment, quaternion is used to describe the pose change of the blogger's head.

[0140] According to the three-axis angular velocity data collected by the IMU unit, the value of the quaternion is updated using the quaternion differential equation. The quaternion differential equation describes the relationship between the rate of change of the quaternion over time and the angular velocity. At each sampling moment, according to the current angular velocity value and the previous quaternion state, the quaternion differential equation is solved by numerical integration methods (such as Euler integration, Runge-Kutta integration, etc.) to obtain a new quaternion representation.

[0141] In the static state, the three-axis acceleration measured by the IMU unit contains the component of the gravitational acceleration. Since the attitude of the AR glasses changes continuously during the movement, the components of the gravitational acceleration on each axis will also change accordingly, thus introducing measurement errors. Therefore, it is necessary to compensate for the gravity bias of the three-axis acceleration. Using the head pose information represented by the constructed quaternion, the measured three-axis acceleration is transformed from the sensor coordinate system to the world coordinate system (usually with the gravitational direction as the reference). In the world coordinate system, the direction of the gravitational acceleration is known. By subtracting the components of the gravitational acceleration vector on each axis from the transformed acceleration vector, the acceleration data after removing the gravity bias is obtained. The acceleration data after gravity bias compensation is combined with the head pose information represented by the quaternion to construct a head pose matrix. The head pose matrix is a 3×3 rotation matrix, which describes the rotation relationship between the sensor coordinate system and the world coordinate system. By converting the quaternion into a rotation matrix, the pose information of the head in three-dimensional space can be obtained.

[0142] Adopt a combination of multiple positioning technologies to obtain the geographic positioning information of the tourism live broadcast scenario, so as to improve the accuracy and reliability of positioning. For example, simultaneously use GPS (Global Positioning System), Beidou satellite navigation system and base station positioning technology. In an open outdoor environment, mainly rely on GPS or Beidou satellite navigation system to provide high-precision position information; in indoor or areas with severe signal occlusion, use base station positioning technology for auxiliary positioning.

[0143] Fuse and process the positioning data obtained by different positioning technologies. Since various positioning technologies have different error sources and accuracy characteristics, they are integrated through data fusion algorithms (such as Kalman filtering algorithm) to obtain more accurate geographic positioning information.

[0144] Build a tightly coupled SLAM (Simultaneous Localization and Mapping) framework, taking the geographic positioning information and the head pose matrix as input data. The characteristic of the tightly coupled SLAM framework is to closely combine the localization and mapping processes, and improve the accuracy and robustness of the system by simultaneously using multiple sensor data. In the tightly coupled SLAM framework, an Extended Kalman Filter (EKF) is used to achieve multi-source data fusion. The Extended Kalman Filter is a non-linear filtering algorithm that can estimate and update the state of the system. Taking the geographic positioning information and the head pose matrix as observation data, combining the state equation and the observation equation of the system, through the prediction and update steps of the Extended Kalman Filter, continuously optimize the state estimation of the system to obtain the fused feature data. The fused feature data contains more accurate blogger position information and head pose information, as well as environmental feature information extracted from these data.

[0145] ORB-SLAM3 is a feature point-based SLAM algorithm framework with advantages such as high efficiency and robustness. It realizes the localization and mapping of the environment by extracting feature points (such as ORB feature points) in the environment and tracking the position changes of these feature points in different image frames. Input the fused feature data into the ORB-SLAM3 framework to start constructing a 3D scene topological map. First, extract ORB feature points in each frame of image, and use the head pose matrix and geographic positioning information to determine the positions of these feature points in three-dimensional space. Then, through feature point matching and motion estimation algorithms, calculate the relative motion of the blogger between different positions, and gradually construct a 3D point cloud map of the environment. At the same time, according to the distribution and connection relationship of the feature points, construct the topological structure of the scene, indicating the connection relationship between different locations.

[0146] The topological consistency detection algorithm is used to detect and correct possible topological errors in the topological map of the 3D scene. By analyzing the topological structure of the nodes (representing different locations) and edges (representing the connection relationships between locations) in the map, the algorithm determines whether it conforms to the logical relationships of the actual scene. During the process of constructing the 3D scene topological map, the topological consistency detection algorithm is run regularly. If topological errors are found, such as unreasonable connection relationships between nodes or the existence of isolated nodes, the algorithm will correct them according to certain rules. For example, by recalculating the distances and connection relationships between nodes, or deleting incorrect connection edges, the topological structure of the map becomes more accurate and reasonable. After multiple iterations and optimizations, a high-quality 3D spatial topological map of the tourism live broadcast scene is obtained.

[0147] In this embodiment, accurate modeling and positioning of the tourism live broadcast scene are achieved, providing richer and more realistic scene information for the tourism live broadcast.

[0148] In some embodiments, in the above step S105, the spatial registration of the dynamic retrieval result and the 3D spatial topological map, combined with the generation of the scene enhancement annotation layer through the NeRF algorithm, specifically includes:

[0149] Calculate the coordinate correspondence between the dynamic retrieval structure and the 3D spatial topological map through a spatial registration algorithm to construct an affine transformation matrix;

[0150] Based on the affine transformation matrix, use the NeRF algorithm to calculate the color and density of each coordinate to generate the scene enhancement annotation layer;

[0151] Dynamically adjust the annotation style of each annotation box in the scene enhancement annotation layer based on semantic labels.

[0152] In this embodiment, for the dynamic retrieval result, which contains a series of entity information related to the tourism live broadcast scene, these entities have certain position attributes in the virtual space (although they may be represented in a two-dimensional or abstract semantic space). First, the entity information needs to be transformed into a form available for feature extraction. For example, if the dynamic retrieval result contains information about a famous scenic spot, the key position points of the scenic spot in the geographical information or virtual scene can be used as candidate feature points. At the same time, collect auxiliary information such as images and text descriptions related to the scenic spot to more comprehensively describe its features.

[0153] In a 3D spatial topological map, feature points are extracted using geometric and semantic features in the map. Geometric features can be obvious inflection points, corner points, etc. in the map, while semantic features can be the center points or boundary points of different regions (such as buildings, roads, landscapes, etc.) in the map. Through feature extraction algorithms (such as SIFT, SURF, etc., although the formula description is not involved, the principle is to use local changes in images or space to identify feature points), a set of representative feature points is extracted from the 3D spatial topological map.

[0154] To establish the connection between the dynamic retrieval results and the 3D spatial topological map, the key entities in the dynamic retrieval results are associated with the corresponding feature points in the 3D spatial topological map, and possible matching feature point pairs are searched for between the two.

[0155] Since the dynamic retrieval results and the 3D spatial topological map may use different coordinate systems, it is first necessary to unify their coordinate systems. For example, if the dynamic retrieval results use a two-dimensional coordinate system based on images, and the 3D spatial topological map uses a three-dimensional spatial coordinate system, the two can be made to be in the same coordinate system framework by mapping the two-dimensional coordinate system to a specific plane in the three-dimensional space, or projecting the three-dimensional coordinate system onto the two-dimensional plane.

[0156] Select a suitable spatial registration algorithm to calculate the coordinate correspondence between the dynamic retrieval results and the 3D spatial topological map. Commonly used registration algorithms include the Iterative Closest Point (ICP) algorithm, etc. The ICP algorithm continuously iterates to find the best matching relationship between two point sets, minimizing the error between them. During the calculation process, the feature point sets of the dynamic retrieval results and the 3D spatial topological map are used as inputs, and through iterative optimization, the corresponding coordinates of each feature point in the coordinate system of the other are obtained.

[0157] According to the calculated coordinate correspondence, an affine transformation matrix is constructed. The affine transformation matrix is a linear transformation matrix that can describe transformation relationships such as translation, rotation, and scaling between two coordinate systems. By analyzing the position change rules between feature points, the parameters in the affine transformation matrix are determined, thus obtaining an affine transformation matrix that can accurately map the dynamic retrieval results to the 3D spatial topological map.

[0158] The NeRF (Neural Radiance Fields) algorithm is a neural network-based scene representation method that can represent a three-dimensional scene as a continuous function, which can output the color and density information at that position according to the coordinates and viewing directions in space. In this embodiment, the NeRF algorithm is used to generate a scene enhancement annotation layer, enabling the annotation to be more realistically integrated into the scene.

[0159] To use the NeRF algorithm, it is necessary to collect multi-view image data related to the tourism live broadcast scenario. These image data can capture the scenario from different angles to cover all aspects of the scenario. At the same time, preprocess the collected image data, including operations such as image correction (such as removing distortion) and color normalization, to improve the data quality. In addition, it is also necessary to associate the image data with the corresponding spatial coordinates and viewing direction information so that the NeRF algorithm can learn the spatial and color information of the scenario.

[0160] Using the previously constructed affine transformation matrix, map the coordinate information in the dynamic retrieval result to the coordinate system of the 3D spatial topology map. Then, sample in the mapped coordinate space and select a series of discrete coordinate points as the input of the NeRF algorithm. These coordinate points cover the distribution area of the dynamic retrieval result in the 3D spatial topology map. Input the sampled coordinate points and the preset viewing direction information into the trained NeRF model. The NeRF model calculates the color and density values of each coordinate point in the given viewing direction according to the scene representation learned internally. The color value determines the display color of this position during rendering, and the density value affects the degree of light occlusion at this position, thus realizing the three-dimensional representation and rendering of the scenario. Generate a scene enhancement annotation layer based on the calculated color and density information of each coordinate point. The annotation layer can be regarded as a transparent layer superimposed on the 3D spatial topology map, where the color and transparency of each pixel point are determined by the color and density of the corresponding coordinate point. In this way, the generated scene enhancement annotation layer can seamlessly fuse with the 3D spatial topology map and present a realistic three-dimensional effect according to different viewing angles.

[0161] The dynamic retrieval result usually contains rich semantic information, which can be used as the source of semantic labels. For example, for entities such as scenic spots, buildings, and people in the tourism live broadcast scenario, the dynamic retrieval result will give their category, name, feature description, and other semantic information. Extract key semantic labels from this information, such as "ancient building", "natural landscape", "tourist", etc. Evaluate the importance of the obtained semantic labels. The importance can be determined according to factors such as the relevance of the semantic label to the current live broadcast theme and user interests, and the frequency of occurrence of the semantic label in the scenario. For example, if the current live broadcast theme is about the history and culture of a certain ancient building, then the semantic label "ancient building" has a higher importance; while some semantic labels with a low frequency of occurrence and weak relevance to the theme have relatively lower importance.

[0162] Define multiple different annotation style styles, such as changes in color, shape, size, transparency, etc. For example, for the semantic label "ancient building", it can be stipulated that the color of its annotation box is a simple brown, the shape is a rectangle, and the size is dynamically adjusted according to the importance of the ancient building in the scene; for the semantic label "tourist", the color of the annotation box can be set to a lively yellow, the shape is a circle, and the size is relatively small. Associate the semantic labels with the corresponding annotation style style rules. Dynamically adjust the annotation style style according to the importance evaluation result of the semantic label. For semantic labels with higher importance, adopt a more prominent and eye-catching annotation style style to attract the user's attention; for semantic labels with lower importance, adopt a relatively low-key and simple annotation style style to avoid causing too much interference to the scene.

[0163] In the process of generating the scene enhancement annotation layer, according to the semantic label corresponding to each annotation box, apply the corresponding annotation style style rules. By querying the pre-defined rule table, obtain the parameters such as color, shape, size, transparency, etc. corresponding to each semantic label, and apply them to the drawing of the annotation box. Integrate and render the scene enhancement annotation layer with the applied annotation style style and the 3D space topology map to generate the final scene display effect. During the tourism live broadcast, the rendered scene is displayed to the user in real time, enabling the user to intuitively see the tourism live broadcast scene with enhanced annotations and enhancing the user's viewing experience.

[0164] In this embodiment, the enhanced display of the tourism live broadcast scene is realized, and the information richness and user experience of the live broadcast content are improved.

[0165] In some embodiments, in the above step S106, the overlay display of the video stream and the scene enhancement annotation layer and the dynamic adjustment of the annotation visibility using the adaptive transparency algorithm specifically include:

[0166] Obtain multiple annotation elements of the scene enhancement annotation layer, embed the annotation elements on the frame sequence of the video stream to form target scene data, and the annotation elements include annotation content, bounding box coordinates, and semantic metadata;

[0167] Obtain the fixation point coordinates of the blogger through the eye movement tracking algorithm, and calculate the fixation focus of each fixation point on the target scene data according to the fixation point coordinates;

[0168] Calculate the head movement speed based on the head pose data to generate a motion attenuation factor;

[0169] Determine the annotation content importance by calculating the semantic correlation between the hot topic set and the semantic metadata;

[0170] Determine the personalized transparency of each annotation element based on the fixation focus, motion attenuation factor, and annotation content importance.

[0171] In this embodiment, a plurality of annotation elements are extracted from the generated scene enhancement annotation layer. These annotation elements contain rich information. The annotation content clarifies the specific information of the annotated object. For example, in a travel live broadcast scenario, it may be annotated as "Ancient building name: XX Pavilion", "Scenic spot feature: Historical and cultural relic", etc.; the bounding box coordinates determine the position range of the annotation in the scene, and define the positions of the four corners of the annotation box in the form of coordinate points; the semantic metadata provides more in-depth semantic information, such as the category of the annotated object (ancient building, natural landscape, etc.), related attributes (construction age, area, etc.).

[0172] The extracted annotation elements are formatted for subsequent embedding into the video stream frame sequence. For example, the annotation content is converted into text format, and its basic styles such as font, font size, and color are determined; the bounding box coordinates are converted into a coordinate system that matches the video stream frame resolution to ensure that the annotation can be accurately placed at the corresponding position in the video frame.

[0173] A continuous video stream frame sequence is obtained from the video source of the travel live broadcast. The video stream frame sequence is a series of image frames arranged in chronological order, and each frame represents the image information of the live broadcast scene at a certain moment. Using a frame-by-frame processing method, the formatted annotation elements are embedded into each frame of the video stream frame sequence. For each frame of the image, according to the bounding box coordinates of the annotation elements, an annotation box is drawn at the corresponding position in the image, and the annotation content is displayed inside the annotation box. At the same time, the semantic metadata is associated and stored with the annotation elements for subsequent use. In this way, the scene enhancement annotation layer is combined with the video stream frame sequence to form the target scene data, so that the video picture contains both the original live broadcast scene and rich annotation information superimposed.

[0174] A suitable eye tracking device, such as an eye tracker, is selected and installed on the AR glasses. The eye tracker can track the blogger's eye movements in real time and obtain information on the position and gaze direction of the eyes. During the installation process, ensure that the position of the eye tracker is accurate, it can stably capture the blogger's eye movement data, and it will not interfere with the blogger's normal live broadcast operation. During the travel live broadcast, the eye tracker continuously collects the blogger's eye movement data. The eye movement data includes information such as the gaze point coordinates, gaze time, and blink frequency of the eyes. Among them, the gaze point coordinates are the key data for calculating the gaze focus, which represents the screen position where the blogger's eyes focus at a certain moment.

[0175] The collected gaze point coordinates are mapped to the target scene data (i.e., the video frame with the annotated elements superimposed). Since the video frame has a specific coordinate system when displayed on the screen, the gaze point coordinates collected by the eye tracker need to be converted to coordinates that match the video frame coordinate system. For each gaze point in the target scene data, its positional relationship with each annotated element is analyzed. If the gaze point falls within the bounding box of an annotated element, the blogger is considered to have paid attention to the annotated element. Based on factors such as the dwell time and gaze frequency of the gaze point within the annotated element, the gaze focus of the annotated element at the current gaze point is calculated. For example, if a blogger gazes at an annotated element for a long time or multiple times, its gaze focus is high; conversely, if it is only briefly gazed at, the gaze focus is low.

[0176] The head's movement speed is calculated based on head posture data collected by the AR glasses' IMU unit. Movement speed can be estimated by analyzing changes in head posture at different times. For example, changes in angular velocity and acceleration over a period of time can be used to determine whether the head is stationary, moving slowly, or rotating rapidly. A motion attenuation factor is generated based on the head's movement speed. When the head moves quickly, it means the blogger may be quickly browsing the scene. In this case, the visibility of the annotations can be appropriately reduced, resulting in a smaller attenuation factor. When the head moves slowly or is stationary, the blogger may be more focused on the current scene, and the visibility of the annotations needs to be increased, resulting in a larger attenuation factor. The motion attenuation factor is used to subsequently adjust the transparency of the annotation elements to accommodate the blogger's different head movements.

[0177] Obtain a collection of hot topics from relevant data sources for travel live streaming. These hot topics can be current hot topics in the tourism sector, collected through social media, travel forums, live streaming platforms, and other channels, such as "Recommended niche attractions in a certain region" and "Specialty food check-in spots." Organize and categorize the acquired hot topics, removing duplicate or irrelevant topics and grouping related topics for subsequent matching with the semantic metadata of the annotated elements. Match the semantic metadata of each annotated element in the scene enhancement annotation layer with the collection of hot topics. Analyze the semantic association between the categories, attributes, and other information contained in the semantic metadata and the hot topics. For example, if the semantic metadata of an annotated element indicates that it belongs to the "niche attractions" category, and the hot topic collection includes the topic "Recommended niche attractions in a certain region," then the annotated element can be considered to have a high semantic relevance to the hot topic. Based on the calculated semantic relevance, determine the importance of the annotated content for each annotated element. Annotated elements with higher semantic relevance have higher importance. Importance can be expressed as different levels, such as high, medium, and low, or as a specific numerical range, for subsequent use in calculating transparency.

[0178] Integrate the calculated gaze focus degree, motion attenuation factor, and annotation content importance. These three factors respectively reflect the visibility requirements of annotation elements from aspects such as the blogger's visual attention, head movement state, and the relevance degree between the annotation content and the hot topic. According to the actual application scenario and requirements, different weights are assigned to the gaze focus degree, motion attenuation factor, and annotation content importance. For example, if it is considered that the blogger's visual attention has the greatest impact on annotation visibility, a higher weight can be assigned to the gaze focus degree; if the head movement state has a smaller impact on annotation visibility, a lower weight can be assigned to the motion attenuation factor.

[0179] According to the assigned weights, perform weighted summation or other comprehensive calculation methods on the gaze focus degree, motion attenuation factor, and annotation content importance to obtain the personalized transparency value of each annotation element. The transparency value is usually between 0 and 1, where 0 represents completely transparent (invisible), and 1 represents completely opaque (completely visible). During the travel live broadcast, adjust the transparency of each annotation element in real time according to the calculated personalized transparency value. When the blogger's gaze point falls on a certain annotation element, if the gaze focus degree of this annotation element is high, the importance is also high, and the head movement speed is slow (the motion attenuation factor is large), then its transparency is increased to make it more prominent; on the contrary, if the gaze focus degree is low, the importance is low, or the head movement speed is fast (the motion attenuation factor is small), then its transparency is decreased to reduce the interference to the blogger's vision. In this way, the dynamic adjustment of annotation visibility is realized, and the user experience of the travel live broadcast is improved.

[0180] In this embodiment, the intelligent display of annotation information in the travel live broadcast scenario is realized, enabling the blogger to obtain relevant information more conveniently, while avoiding the impact of information overload on the live broadcast experience.

[0181] Further, after step S106, the method further includes:

[0182] Obtain the audience portrait data and real-time feedback data of the live broadcast room;

[0183] Based on the audience portrait data, determine the interest degree of each user in different contents, and construct an audience attribute vector;

[0184] Based on the real-time feedback data, determine multiple feedback events to form a feedback event set, where the feedback event includes an event type, an annotation element identifier, and a timestamp;

[0185] Based on the audience attribute vector and the feedback event set, calculate the display weight of each annotation element through a semantic similarity function based on knowledge graph embedding to obtain a priority scoring result;

[0186] Determine the display parameters of each annotation element according to the priority scoring results, and generate a rendering strategy matrix.

[0187] In this embodiment, audience portrait data is collected from multiple data sources of the live streaming platform. On the one hand, through the registration information of users on the live streaming platform, the basic attributes of the audience, such as age, gender, region, etc., are obtained. These information are the basic data for constructing the audience portrait and reflect the basic characteristics of the audience. On the other hand, analyze the historical behavior data of the audience, including the types of live streams watched, the hosts followed, the interactive activities participated in (such as liking, commenting, sharing, etc.), and the duration of staying in the live room. For example, if a certain audience often watches travel-related live streams and stays in the travel live room for a long time, it can be initially judged that the audience has a high interest in travel-related content.

[0188] Utilize the technical architecture of the live streaming platform to capture the feedback data of the audience in real time. When the audience performs operations such as liking, commenting, or selecting an emoji, the system immediately records the relevant information of these feedback events, including the event type (such as liking, commenting, emoji selection, etc.), the annotation element identifier (if the feedback is for a certain annotation element in the live room, such as a scenic spot annotation, a product recommendation annotation, etc., record the unique identifier of the annotation element), and the timestamp (record the specific time when the feedback event occurs).

[0189] According to the business requirements and content characteristics of the live room, divide the live content into multiple categories, such as travel scenic spot introduction, food recommendation, cultural and historical explanation, etc. For each content category, define corresponding interest evaluation indicators. For example, for the travel scenic spot introduction content, the proportion of the duration of the audience watching this type of live stream and the frequency of participating in relevant interactions (such as asking questions, commenting on scenic spot-related topics) can be used as interest evaluation indicators. According to the importance of different content categories to the live room and the attention degree of the audience, assign weights to each interest evaluation indicator. For example, if the travel scenic spot introduction is the core content of the live room, a higher weight can be assigned to the interest evaluation indicators for this content category.

[0190] For each audience, calculate their interest in different content categories according to their historical behavior data and interest evaluation indicators. For example, count the total duration of the audience watching travel scenic spot introduction live streams in the past period of time and its proportion in the total duration of all watched live streams, and then combine the number of times the audience participates in relevant interactions to comprehensively calculate the interest score of the audience in the travel scenic spot introduction content.

[0191] Combine the interest scores of the audience for each content category to construct an audience attribute vector. For example, if there are three content categories: tourist attraction introductions, food recommendations, and cultural and historical explanations, then the audience attribute vector can be represented as (interest score for tourist attraction introductions, interest score for food recommendations, interest score for cultural and historical explanations). This vector can intuitively reflect the audience's interest preferences for different contents.

[0192] Classify the real-time feedback data collected from the live stream room and define different types of feedback events. Common types of feedback events include like events, comment events, emoji selection events, etc. Each event type has its specific meaning and manifestation form. For example, a like event indicates the audience's recognition of the live stream content, a comment event indicates that the audience has specific opinions or ideas to express, and an emoji selection event can quickly convey the audience's emotional state. Use natural language processing techniques and pattern recognition algorithms to automatically identify and classify the real-time feedback data. For example, for comment data, the event type it belongs to can be judged through methods such as keyword matching and semantic analysis; for emoji selection data, the event type is directly determined according to the emoji icon selected by the user.

[0193] For each identified feedback event, record its detailed information, including the event type, annotation element identifier, and timestamp. If the feedback event is for a certain annotation element in the live stream room (such as the annotation box of a certain scenic spot), then record the unique identifier of this annotation element to facilitate subsequent analysis of the audience's feedback on different annotation elements. The timestamp records the specific time when the feedback event occurred, which helps to analyze the timeliness and trend of the feedback event. Integrate all the identified feedback events into a feedback event set in chronological order, and this feedback event set contains all the real-time feedback information of the audience in the live stream room.

[0194] Construct a knowledge graph related to the content of the live stream room. First, determine the entities in the knowledge graph, such as the annotation elements in the live stream room (scenic spots, products, people, etc.), content categories (tourist attraction introductions, food recommendations, etc.), and audience attributes (age, gender, interest preferences, etc.). Then, define the relationships between the entities. For example, the belonging relationship between the annotation element and the content category (a certain scenic spot belongs to the tourist attraction introduction category), and the interest association relationship between the audience attribute and the content category (the audience of a certain age group is interested in food recommendation content). Obtain relevant information from the database of the live stream platform, external data sources (such as tourist attraction databases, food recommendation databases, etc.) and fill it into the knowledge graph. At the same time, as the live stream content is updated and the audience feedback accumulates, regularly update the knowledge graph to ensure that it can accurately reflect the knowledge relationships inside and outside the live stream room.

[0195] Using knowledge graph embedding technology, entities and relationships in the knowledge graph are mapped into a low-dimensional vector space to obtain the embedding representation of each entity. These embedding representations can capture the semantic similarity and relevance between entities. For example, by training the model, entities with similar semantics in the knowledge graph are closer in the vector space. For each annotation element and the viewer attribute vector, calculate their semantic similarity in the vector space. Specifically, calculate the similarity between the entity embedding vector corresponding to the annotation element in the knowledge graph and the viewer attribute vector. Similarity calculation can use methods such as cosine similarity to judge their similarity degree by comparing the directions of the two vectors. The higher the similarity, the more the annotation element matches the viewer's interest preference.

[0196] Combining the information in the feedback event set, adjust the results obtained from the semantic similarity calculation to obtain the display weight of each annotation element. For example, if an annotation element is liked or commented on multiple times in the feedback event set, it indicates that the viewer has a high degree of attention to this annotation element, and its weight value can be appropriately increased when calculating the display weight. Sort all annotation elements according to the display weight to obtain the priority scoring result. Annotation elements with a high priority score indicate that they have a high match with the viewer's interest preference and are more concerned by the viewer, and should be preferentially displayed in the live broadcast room.

[0197] According to the display requirements of the live broadcast room and the user experience requirements, determine the display parameters of each annotation element. Common display parameters include display position, display size, display color, display duration, etc. For example, for important annotation elements (such as popular scenic spot annotations), their display position can be set in a prominent position on the screen, the display size can be appropriately enlarged, and the display color can be set to a prominent color to attract the viewer's attention.

[0198] According to the priority scoring result, assign corresponding display parameter values to each annotation element. Annotation elements with a high priority score can obtain more favorable display parameter values, such as a larger display size, a longer display duration, etc.; while annotation elements with a low priority score can appropriately reduce their display size, shorten the display duration, or even not be displayed in some cases.

[0199] Combine all the annotation elements and their corresponding display parameter values to construct a rendering strategy matrix. The rendering strategy matrix is a two-dimensional table where the rows represent different annotation elements and the columns represent different display parameters. Each element in the matrix corresponds to the specific value of an annotation element under a certain display parameter. During the live broadcast, update the rendering strategy matrix in real-time according to the audience's feedback and changes in the live content. When a new feedback event occurs or the live content is updated, recalculate the display weights and priority scores of the annotation elements and adjust the parameter values in the rendering strategy matrix accordingly. Apply the updated rendering strategy matrix to the rendering system in the live broadcast room to achieve dynamic display and optimization of the annotation elements.

[0200] In this embodiment, it is possible to intelligently adjust the display method of the annotation elements in the live broadcast room according to the audience's interest preferences and real-time feedback, improving the audience's viewing experience and live broadcast effect.

[0201] Refer to Figure 2 , an embodiment of the present invention provides a blogger live video AR glasses Jingwen annotation system 2, and the system 2 specifically includes:

[0202] The first annotation module 201 is used to collect the video stream of the travel live broadcast scene in real-time through the built-in camera and built-in sensor of the AR glasses, and obtain the live broadcast room interaction data stream in real-time through the Apache Flink stream processing engine;

[0203] The second annotation module 202 is used to obtain a set of hot topics and sentiment classification results based on the live broadcast room interaction data stream by analyzing the bullet screen word frequency, live broadcast time period, and bullet screen text;

[0204] The third annotation module 203 is used to dynamically adjust the knowledge graph retrieval strategy of the travel live broadcast scene based on the video stream of the live broadcast scene and the set of hot topics and sentiment classification results to obtain dynamic retrieval results;

[0205] The fourth annotation module 204 is used to obtain the blogger's head pose data based on the IMU unit of the AR glasses, and combine the geolocation information to construct a 3D spatial topology map of the travel live broadcast scene through the SLAM algorithm;

[0206] The fifth annotation module 205 is used to perform spatial registration on the dynamic retrieval results and the 3D spatial topology map, and combine to generate a scene enhancement annotation layer through the NeRF algorithm;

[0207] The sixth annotation module 206 is used to overlay and display the video stream and the scene enhancement annotation layer, and dynamically adjust the annotation visibility using an adaptive transparency algorithm.

[0208] It can be understood that, as Figure 1The content in the embodiments of the method for annotating scene text of the blogger live video AR glasses shown above is applicable to the embodiments of the system for annotating scene text of the blogger live video AR glasses. The specific functions implemented by the embodiments of the system for annotating scene text of the blogger live video AR glasses are the same as those in the embodiments of the method for annotating scene text of the blogger live video AR glasses shown in Figure 1 shown above, and the beneficial effects achieved are also the same as those in the embodiments of the method for annotating scene text of the blogger live video AR glasses shown in Figure 1 shown above.

[0209] It should be noted that for the content such as information interaction and execution process between the above systems, since they are based on the same concept as the method embodiments of the present invention, for their specific functions and the technical effects brought, reference can be specifically made to the method embodiment part, and details will not be repeated here.

[0210] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments, and details will not be repeated here.

[0211] Referring to Figure 3 , an embodiment of the present invention further provides a computer device 3, including: a memory 302, a processor 301, and a computer program 303 stored on the memory 302. When the computer program 303 is executed on the processor 301, the method for annotating scene text of the blogger live video AR glasses as described in any one of the above methods is implemented.

[0212] The computer device 3 can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art can understand that Figure 3 this is only an example of the computer device 3 and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine some components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0213] The so-called processor 301 may be a Central Processing Unit (CPU), and this processor 301 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0214] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as the hard disk or memory of the computer device 3. In other embodiments, the memory 302 may also be an external storage device of the computer device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the computer device 3. Further, the memory 302 may also include both the internal storage unit and the external storage device of the computer device 3. The memory 302 is used to store an operating system, application programs, a Boot Loader, data, and other programs, such as the program code of the computer program, etc. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0215] The embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it implements the blogger live video AR glasses Jingwen annotation method as described in any one of the above methods.

[0216] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the method of the above embodiment in this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium may not be an electrical carrier signal and a telecommunication signal.

[0217] In the above embodiments, the descriptions of the various embodiments have their own focuses. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0218] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0219] In the embodiments disclosed in this application, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0220] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A method for annotating the scene text of a blogger's live video AR glasses, characterized in that, The method specifically includes: Real-time collect the video stream of the tourism live broadcast scene through the built-in camera and built-in sensors of the AR glasses, and obtain the interactive data stream of the live broadcast room in real time through the Apache Flink stream processing engine; Based on the interactive data stream of the live broadcast room, obtain the hot topic set and sentiment classification result by analyzing the bullet screen word frequency, live broadcast time period and bullet screen text; Based on the video stream of the live broadcast scene, dynamically adjust the knowledge graph retrieval strategy of the tourism live broadcast scene through the hot topic set and sentiment classification result to obtain the dynamic retrieval result; Obtain the blogger's head pose data based on the IMU unit of the AR glasses, and combine the geolocation information to construct a 3D spatial topology map of the tourism live broadcast scene through the SLAM algorithm; Perform spatial registration on the dynamic retrieval result and the 3D spatial topology map, and combine to generate a scene enhancement annotation layer through the NeRF algorithm; Overlay and display the video stream and the scene enhancement annotation layer, and dynamically adjust the annotation visibility using the adaptive transparency algorithm.

2. The method according to claim 1, wherein The step of obtaining the hot topic set and sentiment classification result by analyzing the bullet screen word frequency, live broadcast time period and bullet screen text based on the interactive data stream of the live broadcast room specifically includes: Extract the bullet screen data set from the interactive data stream of the live broadcast room, perform time window segmentation on the bullet screen data set, and divide it into several bullet screen data subsets according to the live broadcast time period; Perform word frequency statistics on the bullet screen data subset within each time window, calculate the heat weight value of each term, and obtain the word frequency vector statistics result; Based on the word frequency vector statistics result, generate a hot topic set through an improved LDA topic model; Perform sentiment polarity classification on the bullet screen text of the bullet screen data subset within each time window, and calculate the sentiment score of each bullet screen using the RoBERTa-base model to obtain the sentiment classification result.

3. The method according to claim 1, wherein The step of dynamically adjusting the knowledge graph retrieval strategy of the tourism live broadcast scene through the hot topic set and sentiment classification result based on the video stream of the live broadcast scene to obtain the dynamic retrieval result specifically includes: Extract the shooting direction angle and timestamp in the video stream of the live broadcast scene, and generate a scene entity set through the object detection model. The scene entity set includes several scene entities and their bounding box coordinates and class labels; Based on the scene entity set, construct a dynamic retrieval weight vector by combining the hot topic set and sentiment classification result; Based on the dynamic retrieval weight vector, screen out the first scene entity set from the pre-set knowledge graph retrieval strategy; According to the associated historical events and their association degrees of each first scene entity in the first scene entity set, and the overlap rate between the bounding box of each first scene entity and the visual attention area of the blogger, calculate the historical relevance score of each first scene entity in the first scene entity set; Based on the historical relevance score, screen out the second scene entity set from the first scene entity set to obtain the dynamic retrieval result.

4. The method according to claim 1, wherein The step of obtaining the blogger's head pose data based on the IMU unit of the AR glasses, and combining the geolocation information to construct a 3D spatial topology map of the tourism live broadcast scene through the SLAM algorithm specifically includes: The IMU unit based on the AR glasses obtains the head pose data of the blogger, and the head pose data includes three-axis angular velocity and three-axis acceleration; Based on the head pose data, the quaternion differential equation is used to perform quaternion multiplication on the three-axis acceleration, and the gravity deviation compensation is combined with the three-axis acceleration to construct a head pose matrix; Obtain the geographic location information of the tourism live broadcast scene, input the geographic location information and the head pose matrix into the tightly coupled SLAM framework, and realize multi-source data fusion through the extended Kalman filter to obtain fused feature data; Based on the fused feature data, use the ORB-SLAM3 framework to construct a 3D scene topology map, and optimize the map structure through the topology consistency detection algorithm.

5. The method according to claim 1, characterized in that, The spatial registration of the dynamic retrieval result and the 3D spatial topology map, combined with the generation of the scene enhancement annotation layer through the NeRF algorithm, specifically includes: Calculate the coordinate correspondence between the dynamic retrieval structure and the 3D spatial topology map through the spatial registration algorithm, and construct an affine transformation matrix; Based on the affine transformation matrix, use the NeRF algorithm to calculate the color and density of each coordinate to generate a scene enhancement annotation layer; Dynamically adjust the annotation style of each annotation box in the scene enhancement annotation layer based on semantic labels.

6. The method according to claim 1, characterized in that The superposition display of the video stream and the scene enhancement annotation layer, and the dynamic adjustment of the annotation visibility using the adaptive transparency algorithm, specifically includes: Obtain multiple annotation elements of the scene enhancement annotation layer, embed the annotation elements on the frame sequence of the video stream to form target scene data, and the annotation elements include annotation content, bounding box coordinates, and semantic metadata; Obtain the gaze point coordinates of the blogger through the eye movement tracking algorithm, and calculate the gaze focus of each gaze point on the target scene data according to the gaze point coordinates; Calculate the head movement speed based on the head pose data to generate a motion attenuation factor; Determine the importance of the annotation content by calculating the semantic correlation between the set of hot topics and the semantic metadata; Based on the gaze focus, motion attenuation factor, and importance of the annotation content, determine the personalized transparency of each annotation element.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the audience portrait data and real-time feedback data of the live broadcast room; Based on the audience portrait data, determine the interest degree of each user in different contents, and construct an audience attribute vector; Based on the real-time feedback data, determine multiple feedback events to form a feedback event set, and the feedback events include event type, annotation element identifier, and timestamp; Based on the audience attribute vector and the feedback event set, calculate the display weight of each annotation element through the semantic similarity function based on knowledge graph embedding to obtain a priority scoring result; Determine the display parameters of each annotation element according to the priority scoring result to generate a rendering strategy matrix.

8. A blogger live video AR glasses scene text annotation system, characterized in that, The system specifically includes: The first annotation module is used to collect the video stream of the tourism live broadcast scene in real time through the built-in camera and built-in sensor of the AR glasses, and obtain the live broadcast room interaction data stream in real time through the Apache Flink stream processing engine; The second annotation module is used to obtain the set of hot topics and the sentiment classification result by analyzing the bullet screen word frequency, live broadcast time period, and bullet screen text based on the live broadcast room interaction data stream; The third annotation module is used to dynamically adjust the knowledge graph retrieval strategy of the tourism live broadcast scenario based on the video stream of the live broadcast scenario, and obtain dynamic retrieval results through the hot topic set and the sentiment classification results; The fourth annotation module is used to obtain the blogger's head pose data based on the IMU unit of the AR glasses, combine the geographic location information, and construct a 3D spatial topology map of the tourism live broadcast scenario through the SLAM algorithm; The fifth annotation module is used to perform spatial registration on the dynamic retrieval results and the 3D spatial topology map, and generate a scene enhancement annotation layer through the NeRF algorithm; The sixth annotation module is used to superimpose and display the video stream and the scene enhancement annotation layer, and dynamically adjust the annotation visibility by using the adaptive transparency algorithm.

9. A computer device, characterized in that, It includes: A memory, a processor, and a computer program stored on the memory. When the computer program is executed on the processor, the blogger live video AR glasses scene text annotation method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon. When the computer program is run by the processor, the blogger live video AR glasses scene text annotation method described in any one of claims 1 to 7 is implemented.

Images