Intelligent exhibition content adaptive pushing method and system based on multi-modal perception

Abstract

The application discloses a multi-modal perception-based intelligent exhibition content adaptive pushing method and system, relates to the technical field of intelligent exhibition, and comprises the following steps: collecting multi-modal data streams and preprocessing to obtain structured data packets, extracting audience visual observation vectors and audience body surface thermal radiation change rates from the structured data packets, and performing multi-source time sequence alignment and feature fusion to generate audience multi-source feature fusion vectors; constructing and updating a personalized dynamic transfer graph of audience visited exhibit nodes and node transfer relationships according to the audience multi-source feature fusion vectors and audience real-time position coordinates; and taking the current exhibit node of the audience in the personalized dynamic transfer graph as a starting point, searching in a pre-constructed exhibition knowledge graph. The application solves the problem that the prior art is difficult to quantize audience cognitive evolution and generate a logically coherent personalized narrative path, and realizes a leap from discrete behavior response to deep cognitive guidance.

Description

Technical Field

[0001] This invention relates to the field of intelligent exhibition technology, and in particular to an adaptive push method and system for intelligent exhibition content based on multimodal perception. Background Technology

[0002] The digital exhibition field is transforming towards personalized interactive experiences. To achieve accurate content delivery, existing intelligent exhibition systems generally adopt push technology based on location and behavior perception. By deploying devices such as radio frequency identification and computer vision in the exhibition space, the system can obtain data such as visitor location and dwell time in real time, and push relevant exhibit content according to preset rules.

[0003] Existing technologies struggle to effectively capture and quantify the deep cognitive states and dynamic evolution of visitors during their tours. The shifts in visitors' interests, changes in their cognitive load, and their intrinsic need for knowledge connections are key factors affecting the effectiveness of personalized learning. Current methods primarily rely on discrete location and behavioral data, lacking the ability to extract and model deep features such as the stability of visitors' cognitive focus and the evolution of their knowledge structure. This results in a lack of logical connections between pushed content based on individual cognitive development, making it difficult to form a coherent personalized narrative path and limiting the in-depth realization of the educational function of exhibitions. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an intelligent exhibition content adaptive push method based on multimodal perception, which solves the technical problem that existing technologies are unable to quantify the evolution of audience cognition and generate logically coherent personalized narrative paths accordingly.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides an intelligent exhibition content adaptive push method based on multimodal perception, which includes: collecting multimodal data streams and preprocessing them to obtain structured data packets; extracting audience visual observation vectors and audience body surface thermal radiation change rates from the structured data packets; and performing multi-source temporal alignment and feature fusion to generate audience multi-source feature fusion vectors. Based on the multi-source feature fusion vector of the audience and the real-time location coordinates of the audience, a personalized dynamic transfer map of the exhibit nodes visited by the audience and the transfer relationship between the nodes is constructed and updated. Starting from the exhibit node where the audience is currently located in the personalized dynamic transfer graph, the association degree between the candidate exhibit node and the personalized dynamic transfer graph is calculated based on the audience's multi-source feature fusion vector in the pre-constructed exhibition knowledge graph, and a push decision package is generated. Based on the next exhibit identifier in the push decision package, retrieve the exhibition content materials, generate adaptive presentation content, and push it to the audience's terminal device; Collect audience feedback behavior data on adaptively presented content and use this feedback behavior data as a new multimodal data stream.

[0007] As a preferred embodiment of the intelligent display content adaptive push method based on multimodal perception described in this invention, the structured data packet includes: Multimodal data streams, including audience spatial observation images, audience surface thermal radiation images, and audience location identification signals, are collected and standardized to obtain structured data packets.

[0008] As a preferred embodiment of the intelligent exhibition content adaptive push method based on multimodal perception described in this invention, wherein: the audience multi-source feature fusion vector includes, Read the audience's body surface thermal radiation image sequence from the structured data packet, compare the phase gradient consistency between adjacent frames of the audience's body surface thermal radiation image sequence, and generate the audience's body surface thermal radiation change rate. The intrinsic mode decomposition of the rate of change of thermal radiation on the audience's body surface is performed to extract the low-frequency variation component of the rate of change of thermal radiation on the audience's body surface. The audience visual observation vector is read from the structured data packet, and a dynamic trajectory similarity measure is performed on the directional change trajectory of the audience visual observation vector to generate an observation stability score. The observation stability score sequence and the low-frequency change component sequence are processed through a cross-modal attention alignment mechanism to calculate the optimal time alignment offset; Based on the optimal time alignment offset, the low-frequency variation component sequence is recalibrated for temporal correlation features. The recalibrated low-frequency variation component sequence and the observation stability score sequence are then used to perform feature interaction through a multi-head attention fusion network to generate a multi-source feature fusion vector.

[0009] As a preferred embodiment of the intelligent exhibition content adaptive push method based on multimodal perception described in this invention, the personalized dynamic transfer map includes: By initializing an empty personalized dynamic transfer map, the real-time point-polygon containment detection is performed between the real-time location coordinates of the audience and the predefined boundaries of the exhibit space. The determination is made by using the ray method to determine if the coordinate point is inside the boundary polygon. When the detection result is true, the containment relationship is determined to be valid, and the audience enters the exhibit node space. Add an exhibit node to the node set based on the personalized dynamic transfer graph, and store the audience multi-source feature fusion vector as the node feature vector of the exhibit node. During the period when the real-time location coordinates of the audience satisfy the point-polygon inclusion detection, the attention persistence feature dimension in the audience's multi-source feature fusion vector is used as the main factor to iteratively update the node feature vector of the exhibit node, forming a node feature vector that reflects the attention evolution process. When the point-polygon containment detection failure indicates that the real-time position coordinates of the audience have left the spatial range of the current exhibit node, the node feature vector of the exhibit node is locked, and the duration from the entry time to the exit time is recorded as the dwell time of the exhibit node. The decay component representing the attention exit mode is extracted from the node feature vector of the previous exhibit node, and the excitation component representing the attention entry mode is extracted from the node feature vector of the current exhibit node. By analyzing the overlapping area and complementary relationship between the decay component and the excitation component on the time axis, the cognitive transfer intensity is derived as the weight of the directed transfer edge. Directed transfer edges are added to the edge set of the personalized dynamic transfer graph to construct and update the personalized dynamic transfer graph.

[0010] As a preferred embodiment of the intelligent display content adaptive push method based on multimodal perception described in this invention, the push decision package includes: The exhibit node with the latest completed attribute encapsulation in the personalized dynamic transfer graph is taken as the exhibit node where the audience is currently located. In the pre-constructed exhibition knowledge graph, exhibit nodes that are connected to the exhibit node where the audience is currently located through causal relationship edges are searched to form a set of candidate exhibit nodes. For each candidate exhibit node in the candidate exhibit node set, perform path continuity analysis in the personalized dynamic transfer graph, calculate the minimum number of hops from any node in the historical access path to the candidate node via the semantic relationship edges in the exhibition knowledge graph, and measure the structural relevance. For each candidate exhibit node in the candidate exhibit node set, the audience multi-source feature fusion vector and the pre-stored content feature vector of the candidate exhibit node are subjected to attention focus analysis. The projection inner product of the feature vector on the dimension representing attention persistence is calculated to measure the state matching degree. For each candidate exhibit node, the structural correlation degree and state matching degree are input into a lightweight feedforward neural network to obtain a comprehensive correlation fit score. From the set of candidate exhibit nodes, select the candidate exhibit node with the highest correlation score, and encapsulate its unique identifier to generate a push decision package.

[0011] As a preferred embodiment of the intelligent exhibition content adaptive push method based on multimodal perception described in this invention, the adaptive presentation of content and its push to the audience's terminal device includes: The next exhibit identifier is parsed from the push decision package. In the exhibition content material library, the next exhibit identifier is used to perform a key-value query to retrieve the exhibition content material associated with the next exhibit identifier. From the exhibition content materials, extract the display elements that match the feature dimensions of the current audience's multi-source feature fusion vector, and generate a content adaptation list; From the content adaptation list, select the presentation format of the display elements, synthesize the selected presentation format with the exhibition content materials to generate adaptive presentation content, and push the adaptive presentation content to the audience's terminal devices through the content distribution network.

[0012] As a preferred embodiment of the intelligent display content adaptive push method based on multimodal perception described in this invention, wherein: the step of using feedback behavior data as a new multimodal data stream includes, After pushing adaptive content, collect spatial observation images of the audience, thermal radiation images of the audience's body surface, and audience location identification signals; The system will associate the audience spatial observation images, audience body surface thermal radiation images, audience location identification signals collected after the adaptive presentation content is pushed with the push timestamp of the adaptive presentation content, and package them to generate feedback behavior data. Feedback behavior data is injected into the multimodal data stream as a new round of multimodal data stream.

[0013] Secondly, the present invention provides an intelligent exhibition content adaptive push system based on multimodal perception, including a fusion module, which collects multimodal data streams and preprocesses them to obtain structured data packets, extracts audience visual observation vectors and audience body surface thermal radiation change rates from the structured data packets, and performs multi-source temporal alignment and feature fusion to generate audience multi-source feature fusion vectors. The module constructs and updates a personalized dynamic transfer map of the exhibits visited by the audience and the transfer relationships between the nodes, based on the fusion vector of the audience's multi-source features and the audience's real-time location coordinates. The association module takes the exhibit node where the audience is currently located in the personalized dynamic transfer graph as the starting point, and calculates the association fit between the candidate exhibit node and the personalized dynamic transfer graph based on the audience's multi-source feature fusion vector in the pre-constructed exhibition knowledge graph, and generates a push decision package. The push module retrieves exhibition content materials based on the next exhibit identifier in the push decision package, generates adaptive presentation content, and pushes it to the audience's terminal devices. The feedback module collects audience feedback behavior data on adaptively presented content and uses this feedback behavior data as a new multimodal data stream.

[0014] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the intelligent display content adaptive push method based on multimodal perception as described in the first aspect of the present invention.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the intelligent display content adaptive push method based on multimodal perception as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By fusing visual, infrared, and position sensor data, a multi-source feature fusion vector representing the audience's cognitive state is constructed, and a personalized dynamic transfer map recording their viewing trajectory is generated. Combined with a pre-constructed exhibition knowledge graph, the correlation and fit between candidate exhibits and the audience's current cognitive state and historical trajectory are calculated. Based on this, the next exhibit to be pushed is decided, and adaptive content is generated. Audience feedback data is collected to form a closed-loop optimization, solving the problem that existing technologies struggle to quantify audience cognitive evolution and generate logically coherent personalized narrative paths. This achieves a leap from discrete behavioral responses to deep cognitive guidance. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for adaptive push of intelligent exhibition content based on multimodal perception.

[0019] Figure 2 This is a schematic diagram of an intelligent exhibition content adaptive push system based on multimodal perception. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Reference Figures 1-2 This is one embodiment of the present invention, which provides an intelligent display content adaptive push method based on multimodal perception, including the following steps: S1. Collect multimodal data streams and preprocess them to obtain structured data packets. Extract the audience visual observation vector and the audience body surface thermal radiation change rate from the structured data packets, and perform multi-source temporal alignment and feature fusion to generate audience multi-source feature fusion vectors.

[0024] S1.1 Acquire multimodal data streams of audience spatial observation images, audience body surface thermal radiation images, and audience location identification signals, and perform standardization processing to obtain structured data packets.

[0025] Furthermore, visual sensors are deployed in the exhibition space to collect images of the audience's spatial observation, infrared sensors are used to collect images of the thermal radiation of the audience's body surface, and radio frequency identification sensors are used to collect signals of the audience's location. The collected images of the audience's spatial observation, thermal radiation of the audience's body surface, and signals of location are then subjected to denoising, normalization, and timestamp alignment to form a structured data packet containing synchronized timestamps of the images of the audience's spatial observation, thermal radiation of the audience's body surface, and signals of location.

[0026] S1.2 Read the audience's body surface thermal radiation image sequence from the structured data packet, compare the phase gradient consistency between adjacent frames of the audience's body surface thermal radiation image sequence, and generate the audience's body surface thermal radiation change rate.

[0027] Furthermore, a sequence of thermal radiation images of the audience's body surface containing continuous timestamps is read from the structured data packet. For each frame of thermal radiation image of the audience's body surface in the sequence and the next frame of thermal radiation image of the audience's body surface, the local phase of the thermal radiation intensity distribution in the neighborhood of each pixel in the image is obtained. The difference in the local phase direction at the same spatial coordinates between two adjacent frames is compared. When the difference in the local phase direction exceeds a preset threshold angle, it is determined that the pixel has undergone thermal radiation change. The ratio of the number of pixels that have undergone significant changes in the whole frame to the total number of pixels is counted, and this ratio is used as the audience's body surface thermal radiation change rate at the current moment to output, generating a time series of audience body surface thermal radiation change rate synchronized with the image sequence.

[0028] S1.3 Perform intrinsic mode decomposition on the rate of change of thermal radiation on the audience's body surface to extract the low-frequency variation component of the rate of change of thermal radiation on the audience's body surface.

[0029] Furthermore, an empirical mode decomposition algorithm is applied to the time series of the audience's body surface thermal radiation change rate. This algorithm decomposes the audience's body surface thermal radiation change rate signal into a series of intrinsic mode functions arranged from high to low frequency through an iterative screening process. The first intrinsic mode function with the lowest frequency is selected as the signal component representing the slow physiological rhythm change. This signal component is the low-frequency change component of the audience's body surface thermal radiation change rate extracted.

[0030] S1.4 Read the audience visual observation vector from the structured data packet, perform dynamic trajectory similarity measurement on the directional change trajectory of the audience visual observation vector, and generate an observation stability score.

[0031] Furthermore, the audience visual observation vector sequence is read from the structured data packet. This sequence records the spatial direction vectors of the audience's head or line of sight at consecutive moments. For the direction vector at each time point in the audience visual observation vector sequence, the cosine similarity between it and the direction vector at the previous time point in the sequence is calculated, thus obtaining a similarity sequence describing the degree of continuous change in direction. Then, the standard deviation of this similarity sequence is calculated within a fixed-length time sliding window. The reciprocal of the standard deviation is defined as the observation stability score at the center of the window. By traversing the entire sequence through the sliding window, a time-synchronized observation stability score sequence is generated.

[0032] The expression for the observation stability score is: ; in, To score the observation stability, The total number of frames within the observation time window. For a moment, For the first The audience's visual observation vector at any given moment.

[0033] S1.5. The observed stability score sequence and the low-frequency change component sequence are processed through a cross-modal attention alignment mechanism to calculate the optimal time alignment offset.

[0034] Furthermore, the observation stability score sequence and the low-frequency variation component sequence are used as inputs, and a cross-modal attention alignment mechanism is applied for processing. This mechanism first maps the two sequences to the same high-dimensional feature space, calculates the attention weights between the feature vectors of each time point in the observation stability score sequence and the feature vectors of all time points in the low-frequency variation component sequence, forming an attention weight matrix. By finding an optimal monotonically non-decreasing path on this weight matrix, the sum of the attention weights of each point on the path is maximized. The relative translation of the two sequences on the time axis corresponding to this path is calculated and determined as the optimal time alignment offset.

[0035] The optimal time alignment offset expression is: ; in, This is the optimal time alignment offset. for The observation stability score at time point. Align weights for cross-modal attention under offset. The maximum time offset search radius, The length of the low-frequency variation component sequence. To observe the length of the stability score sequence, The length of the low-frequency variation component sequence. For the time index of the low-frequency variation component sequence, Timestamps for low-frequency variation component sequences. Assuming a time alignment offset, It is a sequence of low-frequency variation components.

[0036] S1.6. Based on the optimal time alignment offset, perform temporal correlation feature recalibration on the low-frequency variation component sequence. Then, perform feature interaction between the recalibrated low-frequency variation component sequence and the observation stability score sequence through a multi-head attention fusion network to generate a multi-source feature fusion vector.

[0037] Furthermore, based on the optimal time alignment offset, the low-frequency variation component sequence is shifted on the time axis to achieve initial alignment with the observation stability score sequence. The time-aligned low-frequency variation component sequence and the observation stability score sequence are then concatenated to form a multi-channel fusion sequence. This fusion sequence is input into a pre-trained multi-head attention fusion network, which contains multiple parallel self-attention layers. Each self-attention layer independently captures long-range dependencies within and between sequences from different subspaces. After concatenation and linear transformation, the outputs of each layer generate a compact audience multi-source feature fusion vector that comprehensively represents the audience's current attention state and physiological response state.

[0038] S2. Based on the multi-source feature fusion vector of the audience and the real-time location coordinates of the audience, construct and update a personalized dynamic transfer map of the exhibit nodes visited by the audience and the transfer relationship between nodes.

[0039] S2.1. By initializing an empty personalized dynamic transfer map, the real-time point-polygon inclusion detection is performed between the real-time location coordinates of the audience and the predefined boundary of the exhibit space. The ray method is used to determine whether the coordinate point is inside the boundary polygon. When the detection result is true, the inclusion relationship is determined to be valid, and the audience enters the exhibit node space.

[0040] Furthermore, an empty graph data structure is created, containing an empty set of nodes and an empty set of edges. This graph data structure is defined as the initial state of the personalized dynamic transfer graph. Next, the real-time position coordinates of the audience, typically a two-dimensional planar coordinate, are acquired from sensors. The predefined boundaries of the exhibit space are also obtained; the space of each exhibit is defined as a planar polygon. For each position update, a ray-based point-polygon inclusion detection is performed. Specifically, a ray is emitted horizontally to the right from the audience's real-time position coordinates, and the number of intersections between this ray and the boundary polygon of the exhibit's space is obtained. If the number of intersections is odd, the audience's real-time position coordinates are determined to be inside the boundary polygon, and the point-polygon inclusion detection result is true, indicating that the audience has entered the space of this exhibit node; otherwise, the detection result is false. When the detection result is true, the subsequent node creation and graph update process is triggered.

[0041] S2.2. Add an exhibit node to the node set based on the personalized dynamic transfer graph, and store the audience multi-source feature fusion vector as the node feature vector of the exhibit node.

[0042] Furthermore, upon determining that a visitor has entered a new exhibit node space, a new node is immediately added to the node set of the personalized dynamic transition graph. This node is uniquely identified as the corresponding exhibit node. Simultaneously, the visitor's multi-source feature fusion vector at the current moment is stored as an attribute value as the node feature vector of the newly added exhibit node. The visitor's multi-source feature fusion vector carries the visitor's multimodal perceptual state upon entering the exhibit space, including quantitative information on multiple dimensions such as attentional persistence and physiological response. The exhibit node's node feature vector is initialized upon creation.

[0043] S2.3 During the period when the real-time location coordinates of the audience satisfy the point-polygon inclusion detection, the attention persistence feature dimension in the audience's multi-source feature fusion vector is used as the main factor to iteratively update the node feature vector of the exhibit node, forming a node feature vector that reflects the attention evolution process.

[0044] Furthermore, while the real-time location coordinates of the audience continuously satisfy the point-polygon inclusion detection of the current exhibit's spatial boundary—that is, while the audience remains within the spatial range of the same exhibit node—an iterative update process for the node feature vector of the current exhibit node is initiated. Each iteration updates the node feature vector based on the previous moment, combined with the newly generated audience multi-source feature fusion vector. During the update, the component representing the attention persistence dimension in the audience multi-source feature fusion vector is given a higher fusion weight, or a specific update gating mechanism is used to ensure that information reflecting the intensity and stability of the audience's attention is preserved and strengthened during the iteration process. Through continuous iterative updates, the final node feature vector is no longer just the instantaneous state at the entry moment, but an integrated representation that comprehensively reflects the evolution trajectory of the audience's attention state throughout the entire stay, forming a node feature vector that reflects the attention evolution process.

[0045] S2.4 When the point-polygon containment detection fails, indicating that the real-time position coordinates of the audience have left the spatial range of the current exhibit node, the node feature vector of the exhibit node is locked, and the duration from the entry time to the exit time is recorded as the dwell time of the exhibit node.

[0046] Furthermore, when the point-polygon containment detection fails (i.e., the detection result changes from true to false), indicating that the visitor's real-time location coordinates have left the spatial range of the current exhibit node, any updates to the node feature vector of the current exhibit node are immediately stopped, and its state is locked. The locked node feature vector becomes the final feature representation of this exhibit visit. The accurate timestamp of the visitor leaving the spatial range of the current exhibit node is recorded, and combined with the previously recorded timestamp of the visitor entering the spatial range of the exhibit node, the difference between the two timestamps is obtained. This difference is precisely recorded as the dwell time at the exhibit node. The dwell time, as a key attribute of the exhibit node, together with the locked node feature vector, completes the encapsulation of the core information of this exhibit visit.

[0047] Specifically, locking is triggered at the boundary point of behavior detection (departure), ensuring that the state interval represented by the node feature vector is strictly aligned with the physical dwell time in time, avoiding mismatch between state updates and behavior intervals. The objective and verifiable behavior metric of dwell time is coupled and stored with the subjective and inferential node feature vector.

[0048] S2.5 Extract the decay component representing the attention exit mode from the node feature vector of the previous exhibit node, and extract the excitation component representing the attention entry mode from the node feature vector of the current exhibit node. By analyzing the overlapping area and complementary relationship between the decay component and the excitation component on the time axis, derive the cognitive transfer intensity as the weight of the directed transfer edge.

[0049] Furthermore, after locking the attributes of the current exhibit node, the previously locked exhibit node is retrieved from the personalized dynamic transfer map. From the locked node feature vector of the previous exhibit node, signal components representing the gradual decline in attention and the weakening trend of interest are extracted using specific feature decoupling or filtering techniques; this component is defined as the decay component. From the node feature vector of the current exhibit node (just locked), signal components representing the activation of attention and the beginning of interest accumulation are extracted; this component is defined as the activation component. The change curves of the decay and activation components are analyzed on the time axis, focusing on whether there is an overlap between the two components within the time window when the audience moves from the previous exhibit to the current exhibit, and whether the change trends of the two components show a smooth transition or a sharp switch. Based on the size of the overlap area and the complementarity of trends, a scalar value is derived through a predetermined relationship model; this scalar value is defined as the cognitive transfer intensity.

[0050] S2.6 Add directed transfer edges to the edge set of the personalized dynamic transfer graph to construct and update the personalized dynamic transfer graph.

[0051] Furthermore, following the cognitive transfer intensity calculation, a new directed edge is added to the edge set of the personalized dynamic transfer graph. The starting node of this directed edge is the previous exhibit node, and the ending node is the current exhibit node. The derived cognitive transfer intensity is assigned to this directed edge as its weight attribute. Through this operation, a transfer relationship with quantified cognitive weights connecting two consecutive visits is established in the personalized dynamic transfer graph. As the visitor's experience continues, the node set and edge set in the personalized dynamic transfer graph are dynamically constructed and updated, gradually forming a complete graph that records the visitor's unique visit trajectory, the visit status of each node, and the cognitive transfer relationships between nodes.

[0052] S3. Starting from the exhibit node where the audience is currently located in the personalized dynamic transfer graph, calculate the correlation and fit between the candidate exhibit node and the personalized dynamic transfer graph based on the audience's multi-source feature fusion vector in the pre-constructed exhibition knowledge graph, and generate a push decision package.

[0053] S3.1. Using the exhibit node with the latest completed attribute encapsulation in the personalized dynamic transfer graph as the exhibit node where the audience is currently located, search for exhibit nodes that are connected to the exhibit node where the audience is currently located through causal relationship edges in the pre-constructed exhibition knowledge graph to form a set of candidate exhibit nodes.

[0054] Furthermore, in the personalized dynamic transfer graph, based on the timestamp of the locked node, the exhibit node that most recently completed the stay time and locked the node feature vector is found. This node is determined as the exhibit node where the audience is currently located. In the pre-constructed exhibition knowledge graph, the exhibit node where the audience is currently located is used as the query starting point. A one-hop traversal is performed along the predefined directional edges in the knowledge graph that represent causal logic. All other exhibit nodes that are directly connected to the exhibit node where the audience is currently located through a causal relationship edge are collected. The collected exhibit nodes together constitute the candidate exhibit node set.

[0055] S3.2 For each candidate exhibit node in the candidate exhibit node set, perform path continuity analysis in the personalized dynamic transfer graph, calculate the minimum number of hops from any node in the historical access path to the candidate node via the semantic relationship edge in the exhibition knowledge graph, and measure the structural correlation.

[0056] Furthermore, all locked historical exhibit nodes are extracted from the personalized dynamic transfer graph, forming a historical visit path node set. For each historical node in the historical visit path node set, the shortest path length from that historical node to the currently examined candidate exhibit node is calculated in the pre-built exhibition knowledge graph, using only semantic relationship edges existing in the pre-built exhibition knowledge graph. This length, measured by the number of semantic relationship edges traversed, is called the minimum hop count. By traversing all historical nodes, a series of minimum hop counts from different historical nodes to the candidate exhibit node are obtained. The minimum value among these minimum hop counts is taken. This minimum value is measured as the structural relevance of the candidate exhibit node to the visitor's historical visit path. The smaller the value, the more direct and stronger the connection with the historical path in terms of knowledge structure.

[0057] The minimum semantic relation hop count expression is: ; in, The structural correlation of candidate exhibit nodes. As candidate exhibit nodes, For historical access path nodes, This is a set of nodes from the historical access path. This represents the minimum number of hops for a semantic relation.

[0058] S3.3 For each candidate exhibit node in the candidate exhibit node set, perform attention focus analysis on the audience multi-source feature fusion vector and the pre-stored content feature vector of the candidate exhibit node, calculate the projection inner product of the feature vector on the dimension representing attention persistence, and measure the state matching degree.

[0059] Furthermore, for each candidate exhibit node in the candidate exhibit node set, the pre-stored content feature vector describing its content attributes, which is pre-stored in the pre-built exhibition knowledge graph, is obtained, along with the audience multi-source feature fusion vector at the current moment. Attention focus analysis is performed by pre-defining or training a linear projection matrix, whose dimension is designed to project the feature vectors into a low-dimensional subspace that characterizes attention persistence. The audience multi-source feature fusion vector and the pre-stored content feature vector of the candidate exhibit node are multiplied by this linear projection matrix to obtain their projection vectors in the attention persistence subspace. The inner product between these two projection vectors is calculated; the result of the inner product is measured as the state matching degree between the candidate exhibit node and the audience's current state. A larger inner product value indicates a better match between the audience's current attention state and the exhibit content in terms of attention persistence features.

[0060] The expression for the state matching degree of the candidate exhibit node is: ; in, The state matching degree of the candidate exhibit nodes. For the audience's multi-source feature fusion vector, This is the pre-stored content feature vector of the candidate exhibit node. The projection matrix for attention focus.

[0061] S3.4 For each candidate exhibit node, input the structural correlation degree and state matching degree into a lightweight feedforward neural network to obtain a comprehensive correlation fit score.

[0062] Furthermore, for each candidate exhibit node, the structural relevance and state matching degree are combined into a two-dimensional input vector. This two-dimensional input vector is then fed into a pre-trained, structurally simple, lightweight feedforward neural network. The lightweight feedforward neural network typically contains one or more fully connected layers and a non-linear activation function, which performs non-linear fusion and weighting of the two heterogeneous metrics: structural relevance and state matching degree. The final output layer of the lightweight feedforward neural network produces a scalar value, which is the combined relevance score of the candidate exhibit node in terms of both knowledge structure relevance and audience immediate state matching.

[0063] S3.5 From the set of candidate exhibit nodes, select the candidate exhibit node with the highest correlation matching score, and encapsulate the unique identifier to generate a push decision package.

[0064] Furthermore, after calculating the relevance scores for all candidate exhibit nodes, the set of candidate exhibit nodes is traversed, and the relevance scores for each candidate exhibit node are compared. The candidate exhibit node with the highest relevance score is selected. The unique identifier of this candidate exhibit node is extracted and encapsulated into a predefined data structure or message. This data structure or message containing the decision result is generated as a push decision package to guide subsequent content retrieval and presentation.

[0065] S4. Retrieve exhibition content materials based on the next exhibit identifier in the push decision package, generate adaptive presentation content, and push it to the audience's terminal device.

[0066] S4.1 Parse the next exhibit identifier from the push decision package, and use the next exhibit identifier to perform a key-value query in the exhibition content material library to retrieve the exhibition content material associated with the next exhibit identifier.

[0067] Furthermore, the push decision package is parsed to extract the next exhibit identifier, which is typically a unique code. In the exhibition content material library storing all exhibit digital resources, a precise key-value query is performed using the extracted next exhibit identifier as the primary key. The exhibition content material library is a structured database where each row corresponds to an exhibit, with the exhibit identifier as the primary key. The remaining fields store multimedia material indexes, metadata, or binary content associated with that exhibit. The query operation returns data rows that exactly match the next exhibit identifier. From these rows, all exhibition content materials associated with that next exhibit identifier are retrieved. The retrieved exhibition content materials are the raw materials for generating the final push content.

[0068] S4.2 Extract display elements from the exhibition content materials that match the feature dimensions of the current audience's multi-source feature fusion vector, and generate a content adaptation list.

[0069] Furthermore, the multi-source feature fusion vector of the audience at the current moment is obtained, and the exhibition content materials are analyzed to extract the content feature descriptions of each type of material. The matching degree of each dimension of the audience's multi-source feature fusion vector with the content feature descriptions of different exhibition content materials is calculated. If the dimension value representing information reception efficiency or cognitive load in the audience's multi-source feature fusion vector is high, then the matching tends to be short text, key information annotation diagrams, and other elements. If the dimension value representing exploration desire or attention persistence is high, then the matching tends to be detailed explanatory videos, deep interactive 3D models, and other elements. A score matching the current audience state is obtained for each type of available exhibition content material. Based on the score sorting or threshold filtering, a content adaptation list is generated. The list lists the display elements from the next exhibit that are most suitable for the audience to obtain in the current state and their priorities.

[0070] S4.3 From the content adaptation list, select the presentation format of the display elements, combine the selected presentation format of the display elements with the exhibition content materials to generate adaptive presentation content, and push the adaptive presentation content to the audience's terminal devices through the content distribution network.

[0071] Furthermore, based on the content adaptation list and the priority of each display element in the list, combined with the currently available terminal device types and network bandwidth status, one or more technically feasible and experience-optimized display element presentation formats are selected from the content adaptation list. For example, if the list recommends prioritizing video narration, and the terminal device is a mobile phone with good network conditions, then streaming video is selected as the presentation format. The original material data corresponding to the selected display element presentation format is extracted from the exhibition content materials. Compositing processing is performed, including but not limited to format transcoding, resolution adaptation, media encapsulation, and timeline alignment and merging of different materials. After compositing processing is completed, an adaptive presentation content file or data stream is generated. The generated adaptive presentation content is distributed to the edge node closest to the viewer's terminal device through the content routing mechanism of the content delivery network, and then the edge node pushes the adaptive presentation content to the viewer's terminal device to complete the content presentation.

[0072] S5. Collect audience feedback behavior data on adaptively presented content and use the feedback behavior data as a new multimodal data stream.

[0073] S5.1 After pushing adaptive presentation content, collect audience spatial observation images, audience body surface thermal radiation images, and audience location identification signals.

[0074] Furthermore, after the adaptively presented content is successfully pushed to the viewer's terminal device and begins presentation, sensing devices deployed in the viewer's area and along their path are immediately activated. Visual sensors continuously capture images of the viewer's area, obtaining spatial observation images that include viewer posture, movements, and interaction with the terminal device. Infrared sensors simultaneously scan the same viewer area, acquiring thermal radiation images reflecting the viewer's body surface temperature distribution. Radio frequency identification (RFID) sensors receive signals from the viewer's personal identification device or terminal device, collecting viewer location identification signals. The collection of these three types of data maintains temporal continuity with the adaptively presented content push event, aiming to capture the viewer's immediate reaction after receiving the content.

[0075] S5.2. After pushing adaptive content, the audience spatial observation images, audience body surface thermal radiation images, audience location identification signals collected will be associated with the push timestamp of the adaptive content and packaged to generate feedback behavior data.

[0076] Furthermore, when pushing adaptively presented content, a precise push timestamp is recorded. Audience spatial observation images, audience surface thermal radiation images, and audience location identification signals collected after the adaptive content is pushed are each appended with their respective acquisition timestamps. These timestamped audience spatial observation images, audience surface thermal radiation images, and audience location identification signals are aligned and correlated with the aforementioned push timestamp of the adaptively presented content. This is typically achieved by matching the acquisition time and push time on the same timeline. The correlated audience spatial observation images, audience surface thermal radiation images, audience location identification signals, and associated push timestamp information are packaged and encapsulated into a structured record. This structured record is defined as feedback behavior data, comprehensively recording the audience's multimodal behavioral responses under the stimulation of specific pushed content.

[0077] S5.3 Inject feedback behavior data into the multimodal data stream as a new round of multimodal data stream.

[0078] Furthermore, the feedback behavior data is injected into the multimodal data stream undergoing data processing, following the same format specifications and data interface as the initial multimodal data stream. This injection operation typically involves appending the feedback behavior data as a new data packet to the end of the time series of the multimodal data stream, or inserting it into the logical position corresponding to its collection timestamp. Once injected, the feedback behavior data, along with subsequent new data continuously collected from sensors and unaffected by push notifications, constitutes a new round of multimodal data streams. This new round of multimodal data streams contains information about the audience's reaction to the previous push notification and will serve as input for subsequent feature extraction, state analysis, graph updates, and decision inference, thus achieving an information loop from content push to audience reaction and then to model and strategy optimization.

[0079] This embodiment also provides an intelligent exhibition content adaptive push system based on multimodal perception, including: a fusion module, which collects multimodal data streams and preprocesses them to obtain structured data packets, extracts audience visual observation vectors and audience body surface thermal radiation change rates from the structured data packets, and performs multi-source temporal alignment and feature fusion to generate audience multi-source feature fusion vectors; The module constructs and updates a personalized dynamic transfer map of the exhibits visited by the audience and the transfer relationships between the nodes, based on the fusion vector of the audience's multi-source features and the audience's real-time location coordinates. The association module takes the exhibit node where the audience is currently located in the personalized dynamic transfer graph as the starting point, and calculates the association fit between the candidate exhibit node and the personalized dynamic transfer graph based on the audience's multi-source feature fusion vector in the pre-constructed exhibition knowledge graph, and generates a push decision package. The push module retrieves exhibition content materials based on the next exhibit identifier in the push decision package, generates adaptive presentation content, and pushes it to the audience's terminal devices. The feedback module collects audience feedback behavior data on adaptively presented content and uses this feedback behavior data as a new multimodal data stream.

[0080] This embodiment also provides a computer device applicable to the adaptive push method of intelligent exhibition content based on multimodal perception, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the adaptive push method of intelligent exhibition content based on multimodal perception as proposed in the above embodiment.

[0081] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0082] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the method for adaptive push of intelligent display content based on multimodal perception as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0083] In summary, this invention constructs a multi-source feature fusion vector representing the audience's cognitive state by fusing visual, infrared, and position sensor data, and generates a personalized dynamic transition map recording their viewing trajectory. Combined with a pre-constructed exhibition knowledge graph, it calculates the correlation and fit between candidate exhibits and the audience's current cognitive state and historical trajectory. Based on this, it decides which exhibit to recommend next and generates adaptive content. Collecting audience feedback data forms a closed-loop optimization, solving the problem of existing technologies struggling to quantify audience cognitive evolution and generate logically coherent personalized narrative paths. This represents a leap from discrete behavioral responses to deep cognitive guidance.

[0084] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for intelligent exhibition content adaptive pushing based on multi-modal perception, characterized in that: include, Multimodal data streams are collected and preprocessed to obtain structured data packets. Audience visual observation vectors and audience body surface thermal radiation change rates are extracted from the structured data packets, and multi-source temporal alignment and feature fusion are performed to generate audience multi-source feature fusion vectors. Based on the multi-source feature fusion vector of the audience and the real-time location coordinates of the audience, a personalized dynamic transfer map of the exhibit nodes visited by the audience and the transfer relationship between the nodes is constructed and updated. Starting from the exhibit node where the audience is currently located in the personalized dynamic transfer graph, the association degree between the candidate exhibit node and the personalized dynamic transfer graph is calculated based on the audience's multi-source feature fusion vector in the pre-constructed exhibition knowledge graph, and a push decision package is generated. Based on the next exhibit identifier in the push decision package, retrieve the exhibition content materials, generate adaptive presentation content, and push it to the audience's terminal device; Collect audience feedback behavior data on adaptively presented content and use this feedback behavior data as a new multimodal data stream.

2. The intelligent exhibition content adaptive push method based on multimodal perception as described in claim 1, characterized in that: The structured data packet includes, Multimodal data streams, including audience spatial observation images, audience surface thermal radiation images, and audience location identification signals, are collected and standardized to obtain structured data packets.

3. The intelligent exhibition content adaptive push method based on multimodal perception as described in claim 2, characterized in that: The audience multi-source feature fusion vector includes, Read the audience's body surface thermal radiation image sequence from the structured data packet, compare the phase gradient consistency between adjacent frames of the audience's body surface thermal radiation image sequence, and generate the audience's body surface thermal radiation change rate. The intrinsic mode decomposition of the rate of change of thermal radiation on the audience's body surface is performed to extract the low-frequency variation component of the rate of change of thermal radiation on the audience's body surface. The audience visual observation vector is read from the structured data packet, and a dynamic trajectory similarity measure is performed on the directional change trajectory of the audience visual observation vector to generate an observation stability score. The observation stability score sequence and the low-frequency change component sequence are processed through a cross-modal attention alignment mechanism to calculate the optimal time alignment offset; Based on the optimal time alignment offset, the low-frequency variation component sequence is recalibrated for temporal correlation features. The recalibrated low-frequency variation component sequence and the observation stability score sequence are then used to perform feature interaction through a multi-head attention fusion network to generate a multi-source feature fusion vector.

4. The intelligent exhibition content adaptive push method based on multimodal perception as described in claim 3, characterized in that: The personalized dynamic transfer map includes, By initializing an empty personalized dynamic transfer map, the real-time point-polygon containment detection is performed between the real-time location coordinates of the audience and the predefined boundaries of the exhibit space. The determination is made by using the ray method to determine if the coordinate point is inside the boundary polygon. When the detection result is true, the containment relationship is determined to be valid, and the audience enters the exhibit node space. Add an exhibit node to the node set based on the personalized dynamic transfer graph, and store the audience multi-source feature fusion vector as the node feature vector of the exhibit node. During the period when the real-time location coordinates of the audience satisfy the point-polygon inclusion detection, the attention persistence feature dimension in the audience's multi-source feature fusion vector is used as the main factor to iteratively update the node feature vector of the exhibit node, forming a node feature vector that reflects the attention evolution process. When the point-polygon containment detection failure indicates that the real-time position coordinates of the audience have left the spatial range of the current exhibit node, the node feature vector of the exhibit node is locked, and the duration from the entry time to the exit time is recorded as the dwell time of the exhibit node. The decay component representing the attention exit mode is extracted from the node feature vector of the previous exhibit node, and the excitation component representing the attention entry mode is extracted from the node feature vector of the current exhibit node. By analyzing the overlapping area and complementary relationship between the decay component and the excitation component on the time axis, the cognitive transfer intensity is derived as the weight of the directed transfer edge. Directed transfer edges are added to the edge set of the personalized dynamic transfer graph to construct and update the personalized dynamic transfer graph.

5. The intelligent exhibition content adaptive push method based on multimodal perception as described in claim 4, characterized in that: The push decision package includes, The exhibit node with the latest completed attribute encapsulation in the personalized dynamic transfer graph is taken as the exhibit node where the audience is currently located. In the pre-constructed exhibition knowledge graph, exhibit nodes that are connected to the exhibit node where the audience is currently located through causal relationship edges are searched to form a set of candidate exhibit nodes. For each candidate exhibit node in the candidate exhibit node set, perform path continuity analysis in the personalized dynamic transfer graph, calculate the minimum number of hops from any node in the historical access path to the candidate node via the semantic relationship edges in the exhibition knowledge graph, and measure the structural relevance. For each candidate exhibit node in the candidate exhibit node set, the audience multi-source feature fusion vector and the pre-stored content feature vector of the candidate exhibit node are subjected to attention focus analysis. The projection inner product of the feature vector on the dimension representing attention persistence is calculated to measure the state matching degree. For each candidate exhibit node, the structural correlation degree and state matching degree are input into a lightweight feedforward neural network to obtain a comprehensive correlation fit score. From the set of candidate exhibit nodes, select the candidate exhibit node with the highest correlation score, and encapsulate its unique identifier to generate a push decision package.

6. The intelligent exhibition content adaptive push method based on multimodal perception as described in claim 5, characterized in that: The adaptive content presentation and push to the viewer's terminal device includes, The next exhibit identifier is parsed from the push decision package. In the exhibition content material library, the next exhibit identifier is used to perform a key-value query to retrieve the exhibition content material associated with the next exhibit identifier. From the exhibition content materials, extract the display elements that match the feature dimensions of the current audience's multi-source feature fusion vector, and generate a content adaptation list; From the content adaptation list, select the presentation format of the display elements, synthesize the selected presentation format with the exhibition content materials to generate adaptive presentation content, and push the adaptive presentation content to the audience's terminal devices through the content distribution network.

7. The intelligent exhibition content adaptive push method based on multimodal perception as described in claim 6, characterized in that: The step of treating feedback behavior data as a new multimodal data stream includes, After pushing adaptive content, collect spatial observation images of the audience, thermal radiation images of the audience's body surface, and audience location identification signals; The system will associate the audience spatial observation images, audience body surface thermal radiation images, audience location identification signals collected after the adaptive presentation content is pushed with the push timestamp of the adaptive presentation content, and package them to generate feedback behavior data. Feedback behavior data is injected into the multimodal data stream as a new round of multimodal data stream.

8. A multimodal perception-based intelligent exhibition content adaptive push system, based on the multimodal perception-based intelligent exhibition content adaptive push method according to any one of claims 1 to 7, characterized in that: include, The fusion module collects multimodal data streams and preprocesses them to obtain structured data packets. It extracts the audience's visual observation vector and the audience's body surface thermal radiation change rate from the structured data packets, and performs multi-source temporal alignment and feature fusion to generate the audience's multi-source feature fusion vector. The module constructs and updates a personalized dynamic transfer map of the exhibits visited by the audience and the transfer relationships between the nodes, based on the fusion vector of the audience's multi-source features and the audience's real-time location coordinates. The association module takes the exhibit node where the audience is currently located in the personalized dynamic transfer graph as the starting point, and calculates the association fit between the candidate exhibit node and the personalized dynamic transfer graph based on the audience's multi-source feature fusion vector in the pre-constructed exhibition knowledge graph, and generates a push decision package. The push module retrieves exhibition content materials based on the next exhibit identifier in the push decision package, generates adaptive presentation content, and pushes it to the audience's terminal devices. The feedback module collects audience feedback behavior data on adaptively presented content and uses this feedback behavior data as a new multimodal data stream.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the intelligent display content adaptive push method based on multimodal perception as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the intelligent display content adaptive push method based on multimodal perception as described in any one of claims 1 to 7.

Images