A museum cultural relic intelligent guide control method and system

Abstract

The application provides a museum cultural relic intelligent guide control method and system, user face video stream and IMU data are acquired through a mobile terminal, a museum three-dimensional digital map, cultural relic semantic knowledge graph and a content database are combined, three-dimensional gaze vector is calculated to lock the cultural relic of user gaze; based on historical viewing and real-time behavior, user cognitive state is dynamically modeled and updated; crowd heat distribution is predicted by fusing real-time positions of all users, guide path is generated by multi-objective optimization taking path length, knowledge graph narrative coherence and congestion avoidance as the target, when the user reaches the target cultural relic, personalized text, audio or AR multi-modal explanation content is dynamically matched and generated from the database according to the cognitive state.

Description

Technical Field

[0001] This application belongs to the field of tour guide control, and in particular relates to an intelligent tour guide control method and system for museum artifacts. Background Technology

[0002] With the deep integration of information technology and cultural heritage protection, the construction of smart museums has become an important development direction in the global museum field. Traditional museum guided tours mainly rely on human guides or fixed audio guide devices. While human guides offer interactivity, they are limited by human resources and cannot meet the personalized needs of a large number of visitors. Traditional audio guide devices typically use button selection of exhibit numbers, resulting in a one-size-fits-all approach that lacks personalization and interactivity, leading to a passive and monotonous user experience. To improve this situation, existing technologies have introduced location-based guided tours, such as using Bluetooth beacons (iBeacon), Wi-Fi, or QR code scanning to automatically identify the user's location or nearby exhibits and trigger corresponding audio content. These technologies improve the automation level of guided tours to some extent, but their recognition accuracy is limited. When multiple exhibits are displayed densely, location information alone cannot accurately determine which artifact the user is truly interested in, resulting in a lack of targeted information delivery. Furthermore, the content presentation is still preset and cannot be dynamically adjusted based on the user's background knowledge, interests, and real-time status, resulting in insufficient personalization of the user experience.

[0003] To further enhance the intelligence and personalization of guided tours, some advanced systems have begun to build user profiles, inferring user interests through in-app clicks, favorites, and other behaviors. However, this user modeling approach is often static or delayed, making it difficult to capture the real-time, dynamic cognitive state and information needs of users during their visit. For example, it cannot perceive in real time the user's level of understanding of the current content or whether they are experiencing fatigue due to information overload. Regarding path planning, existing guided tour systems either provide fixed thematic routes or only recommend routes based on the shortest path, rarely taking into account multiple dimensions such as the inherent knowledge connections between exhibits (e.g., narrative logic), the user's personalized interests, and the real-time flow of visitors within the museum to generate intelligent viewing paths that meet personalized needs while effectively avoiding congestion. Similarly, in terms of content generation, existing technologies mostly rely on pushing pre-made complete content segments, failing to adaptively extract elements from the content library based on the user's knowledge level, interests, and cognitive load, and dynamically combine them into the most suitable multimodal explanation. Therefore, there is significant room for optimization in the depth, breadth, and form of information delivery. Summary of the Invention

[0004] To address the aforementioned problems, this application proposes a smart guide control method for museum artifacts, characterized by the following steps:

[0005] The system acquires user facial video streams and inertial measurement unit (IMU) data collected by mobile terminals and loads them into a pre-set 3D digital map of the museum, a semantic knowledge graph of cultural relics, and a structured content database.

[0006] The user's three-dimensional gaze vector is calculated by jointly using the facial video stream and IMU data. The cultural relic currently being gazed at by the user is located by calculating the intersection of the three-dimensional gaze vector with the model object in the three-dimensional digital map.

[0007] By combining users' historical viewing data and current interaction behavior, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics. The user's cognitive state includes a knowledge depth score representing knowledge reserves, an interest topic vector representing the focus, and a cognitive load index representing the ability to receive information.

[0008] Based on the updated user cognitive state, the anonymous real-time location information of all users is integrated to predict the heat distribution of the crowd. The path length, knowledge graph-based narrative coherence, and congestion avoidance are used as multiple optimization objectives to calculate and generate the next guide path.

[0009] When a user arrives at the location of the target cultural relic, based on the user's cognitive state, text, audio, or augmented reality elements that match the current knowledge depth score, interest topic vector, and cognitive load index are adaptively extracted and combined from the structured content database to generate personalized multimodal explanation content and output it.

[0010] Preferably, the step of jointly calculating the user's three-dimensional gaze vector using the facial video stream and IMU data includes:

[0011] From the facial video stream, a deep learning model is used to simultaneously estimate the user's three-dimensional head pose and two-dimensional gaze direction in the mobile terminal coordinate system;

[0012] The three-dimensional pose of the mobile terminal in the world coordinate system is calculated using the IMU data;

[0013] By combining the three-dimensional pose of the mobile terminal in the world coordinate system with the three-dimensional pose of the user's head and the two-dimensional gaze direction in its coordinate system, the three-dimensional gaze vector of the user in the world coordinate system of the museum's three-dimensional digital map is calculated through coordinate system transformation.

[0014] Preferably, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics. Specifically, the process of updating the knowledge depth score includes:

[0015] Associate a knowledge depth score with each cultural relic node in the cultural relic semantic knowledge graph, with an initial value of 0;

[0016] When a user stares at a cultural relic for more than a preset time threshold, the knowledge depth score of the cultural relic node increases by a fixed value ΔK1.

[0017] If a user actively requests a detailed explanation of a cultural relic, the knowledge depth score of the cultural relic node will be increased by an additional fixed value ΔK2.

[0018] The user's overall knowledge depth score is obtained by applying an exponential moving average to the knowledge depth scores of all the cultural relic nodes viewed.

[0019] Preferably, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics. Specifically, the process of updating the interest topic vector includes:

[0020] Each cultural relic node in the cultural relic semantic knowledge graph is associated with a predefined N-dimensional topic feature vector;

[0021] The attention weight of each cultural relic in the current interaction is calculated based on the duration of the user's gaze at each cultural relic.

[0022] The feature vectors of all cultural relics viewed by the user are weighted and summed according to their attention weights. Then, the summation is performed using an exponential moving average method and fused with the user's historical interest theme vectors to obtain the latest interest theme vector.

[0023] Preferably, the calculation to generate the next guided tour path includes:

[0024] Construct a navigation map with accessible locations within the museum as nodes and accessible paths as edges;

[0025] Define the connection between adjacent nodes and Path segment cost function , where L(i,j) is the normalized value of the path segment length, S(i,j) is the normalized value of the narrative coherence score from cultural relic i to cultural relic j, H(j) is the normalized value of the predicted congestion index of target node j, and w1, w2, w3 are preset weights.

[0026] use The search algorithm searches the navigation map for the tour path with the minimum cumulative path cost from the current location to the candidate target artifact.

[0027] Preferably, the generation and output of personalized multimodal explanation content includes:

[0028] If the knowledge depth score is lower than the first threshold, then basic core information will be provided.

[0029] If the knowledge depth score is higher than the first threshold and the cognitive load index is lower than the second threshold, then, in combination with the user's interest topic vector, additional related interesting knowledge or the activation of the augmented reality module for in-depth interaction will be provided on top of the basic explanation.

[0030] If the cognitive load index is higher than the second threshold, the explanation content will be simplified, only the most critical text information will be presented, and active path recommendation will be suspended.

[0031] Preferably, the method of locking the artifact currently being viewed by the user includes:

[0032] The starting point of the three-dimensional gaze vector is determined from the camera parameters of the mobile terminal and the position calculated by the IMU;

[0033] In a three-dimensional digital map, a ray is constructed using the starting point and the three-dimensional gaze vector;

[0034] A ray-bounding box intersection detection algorithm is used to calculate the intersection points of the ray with the bounding boxes of each cultural relic model in the 3D digital map;

[0035] Select the artifact model that intersects with the ray and is closest to the user as the artifact the user is currently viewing.

[0036] Furthermore, this application proposes an intelligent guide and control system for museum artifacts, characterized by comprising the following units:

[0037] The system acquires user facial video streams and inertial measurement unit (IMU) data collected by mobile terminals and loads them into a pre-set 3D digital map of the museum, a semantic knowledge graph of cultural relics, and a structured content database.

[0038] The user's three-dimensional gaze vector is calculated by jointly using the facial video stream and IMU data. The cultural relic currently being gazed at by the user is located by calculating the intersection of the three-dimensional gaze vector with the model object in the three-dimensional digital map.

[0039] By combining users' historical viewing data and current interaction behavior, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics. The user's cognitive state includes a knowledge depth score representing knowledge reserves, an interest topic vector representing the focus, and a cognitive load index representing the ability to receive information.

[0040] Based on the updated user cognitive state, the anonymous real-time location information of all users is integrated to predict the heat distribution of the crowd. The path length, knowledge graph-based narrative coherence, and congestion avoidance are used as multiple optimization objectives to calculate and generate the next guide path.

[0041] When a user arrives at the location of the target cultural relic, based on the user's cognitive state, text, audio, or augmented reality elements that match the current knowledge depth score, interest topic vector, and cognitive load index are adaptively extracted and combined from the structured content database to generate personalized multimodal explanation content and output it.

[0042] Preferably, the step of jointly calculating the user's three-dimensional gaze vector using the facial video stream and IMU data includes:

[0043] From the facial video stream, a deep learning model is used to simultaneously estimate the user's three-dimensional head pose and two-dimensional gaze direction in the mobile terminal coordinate system;

[0044] The three-dimensional pose of the mobile terminal in the world coordinate system is calculated using the IMU data;

[0045] By combining the three-dimensional pose of the mobile terminal in the world coordinate system with the three-dimensional pose of the user's head and the two-dimensional gaze direction in its coordinate system, the three-dimensional gaze vector of the user in the world coordinate system of the museum's three-dimensional digital map is calculated through coordinate system transformation.

[0046] Preferably, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics. Specifically, the process of updating the knowledge depth score includes:

[0047] Associate a knowledge depth score with each cultural relic node in the cultural relic semantic knowledge graph, with an initial value of 0;

[0048] When a user stares at a cultural relic for more than a preset time threshold, the knowledge depth score of the cultural relic node increases by a fixed value ΔK1.

[0049] If a user actively requests a detailed explanation of a cultural relic, the knowledge depth score of the cultural relic node will be increased by an additional fixed value ΔK2.

[0050] The user's overall knowledge depth score is obtained by applying an exponential moving average to the knowledge depth scores of all the cultural relic nodes viewed.

[0051] Preferably, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics. Specifically, the process of updating the interest topic vector includes:

[0052] Each cultural relic node in the cultural relic semantic knowledge graph is associated with a predefined N-dimensional topic feature vector;

[0053] The attention weight of each cultural relic in the current interaction is calculated based on the duration of the user's gaze at each cultural relic.

[0054] The feature vectors of all cultural relics viewed by the user are weighted and summed according to their attention weights. Then, the summation is performed using an exponential moving average method and fused with the user's historical interest theme vectors to obtain the latest interest theme vector.

[0055] Preferably, the calculation to generate the next guided tour path includes:

[0056] Construct a navigation map with accessible locations within the museum as nodes and accessible paths as edges;

[0057] Define the connection between adjacent nodes and Path segment cost function , where L(i,j) is the normalized value of the path segment length, S(i,j) is the normalized value of the narrative coherence score from cultural relic i to cultural relic j, H(j) is the normalized value of the predicted congestion index of target node j, and w1, w2, w3 are preset weights.

[0058] use The search algorithm searches the navigation map for the tour path with the minimum cumulative path cost from the current location to the candidate target artifact.

[0059] Preferably, the generation and output of personalized multimodal explanation content includes:

[0060] If the knowledge depth score is lower than the first threshold, then basic core information will be provided.

[0061] If the knowledge depth score is higher than the first threshold and the cognitive load index is lower than the second threshold, then, in combination with the user's interest topic vector, additional related interesting knowledge or the activation of the augmented reality module for in-depth interaction will be provided on top of the basic explanation.

[0062] If the cognitive load index is higher than the second threshold, the explanation content will be simplified, only the most critical text information will be presented, and active path recommendation will be suspended.

[0063] Preferably, the method of locking the artifact currently being viewed by the user includes:

[0064] The starting point of the three-dimensional gaze vector is determined from the camera parameters of the mobile terminal and the position calculated by the IMU;

[0065] In a three-dimensional digital map, a ray is constructed using the starting point and the three-dimensional gaze vector;

[0066] A ray-bounding box intersection detection algorithm is used to calculate the intersection points of the ray with the bounding boxes of each cultural relic model in the 3D digital map;

[0067] Select the artifact model that intersects with the ray and is closest to the user as the artifact the user is currently viewing.

[0068] Compared to existing technologies, this invention, by jointly processing user facial video streams and IMU data, can pinpoint the artifact a user is currently viewing. This solves the problem of traditional location-based technologies struggling to accurately determine user intent in densely populated exhibit areas, ensuring the accuracy of information delivery. Furthermore, this invention constructs a multi-dimensional user cognitive state model encompassing knowledge depth, interest themes, and cognitive load. This model assesses a user's knowledge base, interests, and information reception capabilities, laying the foundation for deeper personalized services. In path planning, it comprehensively considers the narrative coherence between exhibits, user interests, and overall visitor flow distribution, generating an optimized viewing route that meets personalized needs, maintains logical flow, and avoids congestion. Regarding the presentation of explanatory content, it selects and combines appropriate explanatory elements from a content library based on the user's specific cognitive state, achieving customized content delivery. This ensures the right depth and breadth of information transmission, avoiding information insufficiency or overload, and improving user learning efficiency and viewing satisfaction. Attached Figure Description

[0069] Figure 1 A flowchart of a specific embodiment;

[0070] Figure 2 This is a schematic diagram of the overall system architecture;

[0071] Figure 3 This is a schematic diagram of the three-dimensional gaze vector calculation;

[0072] Figure 4 A schematic diagram of a user's cognitive state model;

[0073] Figure 5 This is a schematic diagram of intelligent navigation route planning. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0075] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0076] In one embodiment, this application proposes an intelligent guide control method for museum artifacts, such as... Figure 1As shown, it includes the following steps:

[0077] S1, acquire the user's facial video stream and inertial measurement unit (IMU) data collected by the mobile terminal, and load them into the preset 3D digital map of the museum, the semantic knowledge graph of cultural relics and the structured content database;

[0078] The system continuously captures user facial video at 30 frames per second using the mobile terminal's front-facing camera, and collects three-axis angular velocity and three-axis linear acceleration data at a frequency of 100 Hz using the terminal's built-in inertial measurement unit. Upon startup, a 3D digital map is asynchronously loaded from the server. This 3D digital map is a museum FBX format model constructed using laser scanning and photogrammetry techniques, containing the precise 3D coordinates and collision body meshes of all artifacts. A semantic knowledge graph of cultural relics, constructed based on the Neo4j graph database, is also loaded. This semantic knowledge graph defines entities such as artifacts, authors, eras, and materials, as well as relationships between them such as creation, influence, and affiliation. The system is then connected to an SQL structured content database. The text, audio, AR model, and other content fragments stored in this database are all tagged with difficulty levels, related topics, and media types using metadata tags, such as... Figure 2 As shown.

[0079] S2, using the facial video stream and IMU data to jointly calculate the user's three-dimensional gaze vector, and by calculating the intersection of the three-dimensional gaze vector with the model object in the three-dimensional digital map, the cultural relic currently being gazed at by the user is located;

[0080] A lightweight convolutional neural network model is used to process real-time facial video streams, detect facial key points, and calculate the user's head pose in the device coordinate system (6DOF). A Gaze360-like model is applied to estimate the gaze direction of the eyes relative to the head. To correct for video processing latency and jitter during rapid head rotation, a Madgwick filter is used to fuse high-frequency IMU data and low-frequency video pose data to generate a smooth and accurate head pose in the world coordinate system. Combined with the user's position in the 3D map determined by the ORB-SLAM2 visual positioning algorithm, the head pose and eye gaze direction are synthesized and transformed into a 3D gaze vector in the world coordinate system. Starting from the user's virtual viewpoint, a ray casting algorithm is executed along the 3D gaze vector direction to calculate the intersection points of the rays with the collision meshes of various cultural relics in the 3D map. The first cultural relic model that is hit, such as the blue and white porcelain vase model, is locked as the target of the user's current gaze.

[0081] S3, combining the user's historical viewing data and current interaction behavior, dynamically model and update the user's cognitive state on the semantic knowledge graph of the cultural relics. The user's cognitive state includes a knowledge depth score representing knowledge reserves, an interest topic vector representing the focus, and a cognitive load index representing the ability to receive information.

[0082] The knowledge depth score is updated using a Bayesian knowledge tracking model. When a user views a certain type of cultural relic, such as bronzes, it increases the knowledge depth score of the corresponding node in the knowledge graph, such as the Shang and Zhou Dynasties and bronzes nodes. The interest topic vector is a high-dimensional floating-point vector whose dimension corresponds to the topic in the knowledge graph, such as calligraphy and painting, porcelain, jade, etc. The value of the vector is updated according to the user's dwell time and interaction frequency in front of various cultural relics, through a weighted algorithm with time decay. The cognitive load index is calculated by a pre-trained support vector machine classifier. The support vector classifier is input with the user's real-time eye movement features, such as pupil diameter changes, fixation point drift and blink frequency, as well as the frequency of subtle involuntary head movements reflected by IMU data, and outputs a continuous value between 0 and 1. The higher the value, the greater the cognitive load.

[0083] S4, based on the updated user cognitive state, integrates the anonymous real-time location information of all users to predict the heat distribution of the crowd, and calculates and generates the next guide path by taking path length, knowledge graph-based narrative coherence and congestion avoidance as multiple optimization objectives.

[0084] Anonymous location data from all online users is aggregated, and a real-time updated museum visitor heatmap is generated using a kernel density estimation algorithm, predicting congestion areas for the next ten minutes. Narrative coherence is quantified by calculating the connection strength between candidate artifacts and the current artifact on a semantic knowledge graph of artifacts; for example, a personalized PageRank-based algorithm is used to evaluate the relevance of two artifact nodes under the user's interest topics. Path planning employs an improved... The pathfinding algorithm uses a cost function G, which is a weighted sum of three parts: the physical walking distance from the starting point to the candidate point, the knowledge jump cost that is inversely proportional to the narrative coherence score, and the congestion penalty value for the path crossing the predicted congested area. By solving the multi-objective optimization to find the path that minimizes the cost function, the algorithm finds the next most worthwhile cultural relic to visit.

[0085] S5. When the user arrives at the target cultural relic location, based on the user's cognitive state, text, audio, or augmented reality elements that match the current knowledge depth score, interest topic vector, and cognitive load index are adaptively extracted and combined from the structured content database to generate personalized multimodal explanation content and output it.

[0086] Based on the current artifact's ID, all relevant content fragments are retrieved from a structured content database. Content is then filtered based on the user's knowledge depth score; for example, users with a score below 0.3 receive beginner-level content, while those above 0.8 receive expert-level content. The filtered content is ranked according to the weight of the user's interest topic vector; for example, if the user's interest in the production process has the highest weight, content introducing firing or carving techniques is prioritized. The presentation format is adjusted based on the cognitive load index; for example, when the index is high, long texts are abandoned, and a short audio explanation is selected paired with a simple AR highlighting effect. A rule-based template engine combines the selected optimal content fragments into a complete explanation; for example, the core information audio is played first, followed by the most relevant images and text, and an optional AR interactive entry is provided and pushed to the user's mobile screen.

[0087] In an optional embodiment, the user's three-dimensional gaze vector is jointly calculated using the facial video stream and IMU data, such as... Figure 3 As shown, it includes:

[0088] From the facial video stream, a deep learning model is used to simultaneously estimate the user's three-dimensional head pose and two-dimensional gaze direction in the mobile terminal coordinate system;

[0089] The three-dimensional pose of the mobile terminal in the world coordinate system is calculated using the IMU data;

[0090] By combining the three-dimensional pose of the mobile terminal in the world coordinate system with the three-dimensional pose of the user's head and the two-dimensional gaze direction in its coordinate system, the three-dimensional gaze vector of the user in the world coordinate system of the museum's three-dimensional digital map is calculated through coordinate system transformation.

[0091] The process captures a video stream of the user's face at 30 frames per second using the front-facing camera of the mobile terminal. A lightweight multi-task convolutional neural network model is deployed on the mobile terminal. This model receives 640x480 pixel video frames as input and can output two sets of key data in real time: the user's three-dimensional head pose, such as a roll angle of 2 degrees, a pitch angle of -10 degrees, and a yaw angle of 15 degrees in Euler angles; and a two-dimensional gaze direction vector of the user in the mobile terminal's screen coordinate system, such as a vector pointing to the upper right of the screen. Both sets of data are relative to the mobile terminal's coordinate system.

[0092] The mobile terminal's built-in inertial measurement unit (IMU) collects acceleration and angular velocity data 100 times per second. By processing this data using fusion algorithms such as Kalman filtering, the 3D attitude of the mobile terminal itself in the pre-constructed museum 3D digital map world coordinate system can be accurately calculated, i.e., its rotation matrix or quaternion. The mobile terminal's world attitude is then cascaded with the user's head attitude and gaze direction obtained in the terminal coordinate system in the previous step, performing a series of coordinate system transformation operations. Through a series of matrix multiplications, the user's gaze direction relative to the terminal is transformed into a global 3D gaze vector in the entire museum's 3D space. This global 3D gaze vector accurately indicates the direction the user is looking in real physical space from their eye position.

[0093] In an optional embodiment, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics, such as... Figure 4 As shown, the specific process of updating the knowledge depth score includes:

[0094] Associate a knowledge depth score with each cultural relic node in the cultural relic semantic knowledge graph, with an initial value of 0;

[0095] When a user stares at a cultural relic for more than a preset time threshold, the knowledge depth score of the cultural relic node increases by a fixed value ΔK1.

[0096] If a user actively requests a detailed explanation of a cultural relic, the knowledge depth score of the cultural relic node will be increased by an additional fixed value ΔK2.

[0097] The user's overall knowledge depth score is obtained by applying an exponential moving average to the knowledge depth scores of all the cultural relic nodes viewed.

[0098] For each cultural relic node in the knowledge graph, such as the blue-and-white porcelain vase with dragon and cloud design, a knowledge depth score is initialized, with an initial value of 0, indicating that the user has no knowledge of the relic. A gaze duration threshold of 3 seconds is set. When the gaze vector detects that the user has gazed at the blue-and-white porcelain vase with dragon and cloud design for 3.5 seconds, it is determined that the user has developed a preliminary understanding of the relic. At this time, the knowledge depth score of the cultural relic node will be increased by a fixed increment value ΔK1, for example, set to 10 points, and the knowledge depth score of the cultural relic node will be updated to 10 points.

[0099] During the user's browsing process, if they develop a stronger interest in the blue-and-white porcelain vase with dragon and cloud design and actively click the detailed explanation button on the interface, this demonstrates a deep learning intent. At this point, the knowledge depth score of the artifact node will be increased by a fixed increment ΔK2, for example, set to 20 points, making the knowledge depth score of the artifact node 10 plus 20 equal to 30 points. To measure the user's overall cognitive level, the knowledge depth scores of all artifacts the user has viewed, such as the blue-and-white porcelain vase, jade cong, and bronze ding, are calculated using an exponential moving average, resulting in a smoothly changing total score that reflects the user's current overall knowledge level.

[0100] In an optional embodiment, the user's cognitive state is dynamically modeled and updated on the cultural relic semantic knowledge graph. Specifically, the process of updating the interest topic vector includes:

[0101] Each cultural relic node in the cultural relic semantic knowledge graph is associated with a predefined N-dimensional topic feature vector;

[0102] The attention weight of each cultural relic in the current interaction is calculated based on the duration of the user's gaze at each cultural relic.

[0103] The feature vectors of all cultural relics viewed by the user are weighted and summed according to their attention weights. Then, the summation is performed using an exponential moving average method and fused with the user's historical interest theme vectors to obtain the latest interest theme vector.

[0104] A 5-dimensional theme feature vector is predefined for each artifact in the knowledge graph. These five dimensions represent dynasty, material, craftsmanship, region, and purpose, respectively. For example, the theme feature vector of a Tang Dynasty sancai (three-color glazed pottery) horse might be defined as vectors 0.9, 0.7, 0.8, 0.6, and 0.4, indicating a high relevance to the themes of Tang Dynasty, ceramics, and sancai craftsmanship. In a single viewing session, suppose a user looks at the Tang Dynasty sancai horse for 10 seconds and at a Song Dynasty porcelain bowl for 5 seconds. Attention weights are calculated based on the viewing duration; for example, if the weight equals the viewing duration, then the weight of the sancai horse is 10, and the weight of the porcelain bowl is 5.

[0105] The thematic feature vectors of all artifacts viewed by the user during this visit are weighted and summed according to their attention weights. Using the example above, a temporary session interest vector is obtained. Subsequently, to balance the user's long-term and immediate interests, an exponential moving average method is used to update the user's global interest topic vector. For example, setting the smoothing factor to 0.3, the latest interest topic vector equals 0.3 multiplied by the current session interest vector, plus 0.7 multiplied by the user's historical interest topic vectors. In this way, the user's interest model can stably reflect their inherent preferences while flexibly adapting to changes in their interests during the current viewing process.

[0106] In an optional embodiment, the calculation to generate the next segment of the guided tour path includes:

[0107] Construct a navigation map with accessible locations within the museum as nodes and accessible paths as edges;

[0108] Define the connection between adjacent nodes and Path segment cost function C , where L(i,j) is the normalized value of the path segment length, S(i,j) is the normalized value of the narrative coherence score from cultural relic i to cultural relic j, H(j) is the normalized value of the predicted congestion index of target node j, and w1, w2, w3 are preset weights.

[0109] use The search algorithm searches the navigation map for the tour path with the minimum cumulative path cost from the current location to the candidate target artifact.

[0110] The museum's floor plan is abstracted as a navigation map composed of nodes and edges. Nodes represent key locations such as in front of display cases and at intersections, while edges represent the actual passable paths connecting these locations. The passage cost between any two adjacent nodes i and j is calculated using a weighted formula. This formula comprises three parts: the first is the physical length L(i,j) of the path segment, for example, 15 meters. The second is the negative value of the narrative coherence score S(i,j). If artifact i and artifact j are closely related in the knowledge graph, such as belonging to the same period or kiln, the score is high, for example, 0.9; otherwise, it is low, for example, 0.2. The third is the normalized value H(j) of the predicted congestion index for the target node j, for example, based on historical data, the area's congestion index at 2 PM is predicted to be 0.8.

[0111] The importance of the three components is adjusted by preset weights w1, w2, and w3. For example, if the user prefers efficiency, the weight of w1 is set to 0.6; if the user is a history enthusiast, the weight of w2 is set to 0.5; and if the user wants to avoid crowds, the weight of w3 is set to 0.7. After determining the cost function, the classic method is used. The search algorithm starts from the user's current location and considers all unvisited artifacts that match the user's interests as potential endpoints. It efficiently calculates and compares all possible paths on the navigation map. It then identifies the path with the minimum cumulative cost among a series of candidate paths and recommends it to the user as the next guided tour route. Figure 5 As shown.

[0112] In an optional embodiment, generating and outputting personalized multimodal explanation content includes:

[0113] If the knowledge depth score is lower than the first threshold, then basic core information will be provided.

[0114] If the knowledge depth score is higher than the first threshold and the cognitive load index is lower than the second threshold, then, in combination with the user's interest topic vector, additional related interesting knowledge or the activation of the augmented reality module for in-depth interaction will be provided on top of the basic explanation.

[0115] If the cognitive load index is higher than the second threshold, the explanation content will be simplified, only the most critical text information will be presented, and active path recommendation will be suspended.

[0116] Two key thresholds were set: a first threshold of 30 points for knowledge depth and a second threshold of 0.7 points for cognitive load index. When a user's overall knowledge depth score is only 15 points, below 30 points, it indicates that the user may be a beginner. At this point, the user will be provided with the most basic and essential explanations about the current cultural relic, such as a voice announcement stating that this is a blue and white porcelain vase from the Yongle period of the Ming Dynasty, introducing its basic shape and use. This strategy ensures that users are not overwhelmed by too much information in the early stages of learning.

[0117] If a user's knowledge depth score reaches 45 points, exceeding the first threshold of 30 points, and their real-time cognitive load index is 0.5, below the second threshold of 0.7, this indicates that the user has the ability to absorb more information and is relaxed. At this point, in addition to the basic explanation, the explanation will be combined with the user's interest topic vector. If the vector shows a strong interest in the craft topic, further explanation will be provided on the characteristics of the cobalt blue pigment during firing and the resulting rust-like effects. Augmented reality functionality may even be triggered to virtually display the internal structure of the porcelain vase on the screen. Conversely, if the user's cognitive load index suddenly rises to 0.8, exceeding the threshold, the explanation will be immediately simplified, perhaps displaying only a single line of key text on the screen, such as "Yongle blue and white flat pot," and the audio and route recommendations will be paused to give the user an information buffer period.

[0118] In an optional embodiment, locking the artifact that the user is currently looking at includes:

[0119] The starting point of the three-dimensional gaze vector is determined from the camera parameters of the mobile terminal and the position calculated by the IMU;

[0120] In a three-dimensional digital map, a ray is constructed using the starting point and the three-dimensional gaze vector;

[0121] A ray-bounding box intersection detection algorithm is used to calculate the intersection points of the ray with the bounding boxes of each cultural relic model in the 3D digital map;

[0122] Select the artifact model that intersects with the ray and is closest to the user as the artifact the user is currently viewing.

[0123] The starting point of the gaze ray needs to be determined, i.e., the position of the user's eye in the three-dimensional space of the museum. This position is precisely calculated based on the real-time positioning results of the mobile terminal in the three-dimensional digital map, such as coordinates calculated using visual inertial odometry (VIO) technology: X-axis 10.2 meters, Y-axis 5.5 meters, Z-axis 1.6 meters, combined with a fixed offset vector from the terminal to the user's eye. The system uses the user's three-dimensional gaze vector in the world coordinate system calculated in the previous steps as the direction of the ray, thereby constructing a virtual ray in the three-dimensional digital map that originates from the user's eye and points in the direction of their gaze.

[0124] To perform efficient collision detection, each artifact model in the 3D digital map, such as vases, paintings, and sculptures, is pre-enclosed in a simple cubic bounding box. The gaze ray constructed in the previous step is then subjected to a fast ray-bounding box intersection detection operation with the bounding boxes of all artifact models in the scene. This operation returns all bounding boxes that intersect with the ray. For example, the detection result might show that the ray intersects simultaneously with the bounding box of artifact A located 2.5 meters from the user and the bounding box of artifact B located 4 meters away. The distances between these two intersection points are compared, and the one closest to the ray's origin, artifact A at 2.5 meters, is selected. Therefore, artifact A is determined and locked as the target currently being viewed by the user.

[0125] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of device 600 to perform the method provided by the present disclosure. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0126] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0127] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0128] The method and electronic device for providing product object information provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for intelligent guide and control of museum artifacts, characterized in that, Includes the following steps: The system acquires user facial video streams and inertial measurement unit (IMU) data collected by mobile terminals and loads them into a pre-set 3D digital map of the museum, a semantic knowledge graph of cultural relics, and a structured content database. The user's three-dimensional gaze vector is calculated by jointly using the facial video stream and IMU data. The cultural relic currently being gazed at by the user is located by calculating the intersection of the three-dimensional gaze vector with the model object in the three-dimensional digital map. By combining users' historical viewing data and current interaction behavior, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics. The user's cognitive state includes a knowledge depth score representing knowledge reserves, an interest topic vector representing the focus, and a cognitive load index representing the ability to receive information. Based on the updated user cognitive state, the anonymous real-time location information of all users is integrated to predict the heat distribution of the crowd. The path length, knowledge graph-based narrative coherence, and congestion avoidance are used as multiple optimization objectives to calculate and generate the next guide path. When a user arrives at the location of the target cultural relic, based on the user's cognitive state, text, audio, or augmented reality elements that match the current knowledge depth score, interest topic vector, and cognitive load index are adaptively extracted and combined from the structured content database to generate personalized multimodal explanation content and output it.

2. The method according to claim 1, characterized in that, The step of jointly calculating the user's three-dimensional gaze vector using the facial video stream and IMU data includes: From the facial video stream, a deep learning model is used to simultaneously estimate the user's three-dimensional head pose and two-dimensional gaze direction in the mobile terminal coordinate system; The three-dimensional pose of the mobile terminal in the world coordinate system is calculated using the IMU data; By combining the three-dimensional pose of the mobile terminal in the world coordinate system with the three-dimensional pose of the user's head and the two-dimensional gaze direction in its coordinate system, the three-dimensional gaze vector of the user in the world coordinate system of the museum's three-dimensional digital map is calculated through coordinate system transformation.

3. The method according to claim 1, characterized in that, The calculation process for the knowledge depth score is as follows: Associate a knowledge depth score with each cultural relic node in the cultural relic semantic knowledge graph, with an initial value of 0; When a user stares at a cultural relic for more than a preset time threshold, the knowledge depth score of the cultural relic node increases by a fixed value ΔK1. If a user actively requests a detailed explanation of a cultural relic, the knowledge depth score of the cultural relic node will be increased by an additional fixed value ΔK2. The user's overall knowledge depth score is obtained by applying an exponential moving average to the knowledge depth scores of all the cultural relic nodes viewed.

4. The method according to claim 1, characterized in that, The process for obtaining the interest topic vector is as follows: Each cultural relic node in the cultural relic semantic knowledge graph is associated with a predefined N-dimensional topic feature vector; The attention weight of each cultural relic in the current interaction is calculated based on the duration of the user's gaze at each cultural relic. The feature vectors of all cultural relics viewed by the user are weighted and summed according to their attention weights. Then, the summation is performed using an exponential moving average method and fused with the user's historical interest theme vectors to obtain the latest interest theme vector.

5. The method according to claim 1, characterized in that, The calculation to generate the next segment of the navigation path includes: Construct a navigation map with accessible locations within the museum as nodes and accessible paths as edges; Define the connection between adjacent nodes and Path segment cost function , where L(i,j) is the normalized value of the path segment length, S(i,j) is the normalized value of the narrative coherence score from cultural relic i to cultural relic j, H(j) is the normalized value of the predicted congestion index of target node j, and w1, w2, w3 are preset weights. use The search algorithm searches the navigation map for the tour path with the minimum cumulative path cost from the current location to the candidate target artifact.

6. The method according to claim 1, characterized in that, The generation and output of personalized multimodal explanation content includes: If the knowledge depth score is lower than the first threshold, then basic core information will be provided. If the knowledge depth score is higher than the first threshold and the cognitive load index is lower than the second threshold, then, in combination with the user's interest topic vector, additional related interesting knowledge or the activation of the augmented reality module for in-depth interaction will be provided on top of the basic explanation. If the cognitive load index is higher than the second threshold, the explanation content will be simplified, only the most critical text information will be presented, and active path recommendation will be suspended.

7. The method according to claim 1, characterized in that, The artifact that the user is currently looking at includes: The starting point of the three-dimensional gaze vector is determined from the camera parameters of the mobile terminal and the position calculated by the IMU; In a three-dimensional digital map, a ray is constructed using the starting point and the three-dimensional gaze vector; A ray-bounding box intersection detection algorithm is used to calculate the intersection points of the ray with the bounding boxes of each cultural relic model in the 3D digital map; Select the artifact model that intersects with the ray and is closest to the user as the artifact the user is currently viewing.

8. A smart guide and control system for museum artifacts, characterized in that, Includes the following units: The system acquires user facial video streams and inertial measurement unit (IMU) data collected by mobile terminals and loads them into a pre-set 3D digital map of the museum, a semantic knowledge graph of cultural relics, and a structured content database. The user's three-dimensional gaze vector is calculated by jointly using the facial video stream and IMU data. The cultural relic currently being gazed at by the user is located by calculating the intersection of the three-dimensional gaze vector with the model object in the three-dimensional digital map. By combining users' historical viewing data and current interaction behavior, the user's cognitive state is dynamically modeled and updated on the semantic knowledge graph of the cultural relics. The user's cognitive state includes a knowledge depth score representing knowledge reserves, an interest topic vector representing the focus, and a cognitive load index representing the ability to receive information. Based on the updated user cognitive state, the anonymous real-time location information of all users is integrated to predict the heat distribution of the crowd. The path length, knowledge graph-based narrative coherence, and congestion avoidance are used as multiple optimization objectives to calculate and generate the next guide path. When a user arrives at the location of the target cultural relic, based on the user's cognitive state, text, audio, or augmented reality elements that match the current knowledge depth score, interest topic vector, and cognitive load index are adaptively extracted and combined from the structured content database to generate personalized multimodal explanation content and output it.

9. The system according to claim 8, characterized in that, The step of jointly calculating the user's three-dimensional gaze vector using the facial video stream and IMU data includes: From the facial video stream, a deep learning model is used to simultaneously estimate the user's three-dimensional head pose and two-dimensional gaze direction in the mobile terminal coordinate system; The three-dimensional pose of the mobile terminal in the world coordinate system is calculated using the IMU data; By combining the three-dimensional pose of the mobile terminal in the world coordinate system with the three-dimensional pose of the user's head and the two-dimensional gaze direction in its coordinate system, the three-dimensional gaze vector of the user in the world coordinate system of the museum's three-dimensional digital map is calculated through coordinate system transformation.

10. The system according to claim 8, characterized in that, Dynamically modeling and updating the user's cognitive state on the cultural relic semantic knowledge graph, specifically updating the knowledge depth score, includes: Associate a knowledge depth score with each cultural relic node in the cultural relic semantic knowledge graph, with an initial value of 0; When a user stares at a cultural relic for more than a preset time threshold, the knowledge depth score of the cultural relic node increases by a fixed value ΔK1. If a user actively requests a detailed explanation of a cultural relic, the knowledge depth score of the cultural relic node will be increased by an additional fixed value ΔK2. The user's overall knowledge depth score is obtained by applying an exponential moving average to the knowledge depth scores of all the cultural relic nodes viewed.

Images