Multi-role multi-mode virtual-real composite interaction method and device, equipment and storage medium

Abstract

The invention relates to the field of data processing, and discloses a multi-role multi-mode virtual-real composite interaction method and device, equipment and a storage medium. Personal visiting information of visiting users is obtained through an image sensor, conversion rule information of user behaviors in the personal visiting information along with time hiding states is obtained through a hidden Markov model, behavior similarities of the users in different time periods are compared in combination with a dynamic time warping algorithm, and user behavior habit change information is obtained; adjusting user visiting route information according to the user behavior habit change information; and acquiring task progress information of the user in the initial user visiting route information by utilizing a finite-state machine model, acquiring logic association information hidden among different events, acquiring user state information in the visiting process of the user, and performing target route planning according to the user state information and the user visiting progress information. Instant adjustment can be carried out according to behaviors and cognitive changes of the user, and the correlation and effectiveness of interactive experience are improved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a multi-role multi-modal virtual-reality composite interaction method, device, equipment and storage medium. Background Art

[0002] Although the current solutions involving multimodal virtual-real interaction in exhibition halls have integrated technologies such as voice, vision, and touch, there are key problems. First, the interactive system relies on fixed preset rules, lacks flexibility, and cannot dynamically adapt to the unique needs and real-time feedback of individual users. Secondly, the cognitive map related to user perception has not been effectively utilized, so that the interaction lacks a deep understanding of the user's psychological state and spatial cognition, resulting in insufficient personalization of content. Thirdly, the personalized recommendation system is mainly based on static data, fails to combine real-time physical location and user navigation path, and cannot provide a truly optimized experience. Finally, the coordination between systems is imperfect, resulting in multimodal fusion often facing the problems of response delay and fragmented experience. Overall, the current solution still has a lot of room for improvement in improving the immersiveness and intelligent adaptive capabilities of interaction. Summary of the invention

[0003] The present application provides a multi-role and multi-modal virtual-reality composite interaction method, which can store asset registration in a database and audit asset change request information through a multiple audit mechanism, thereby ensuring the security of assets.

[0004] In a first aspect, an embodiment of the present application provides a multi-role multi-modal virtual-reality composite interaction method, the multi-modal virtual-reality composite interaction method comprising the following steps:

[0005] Based on the image sensor, the personal visit information of the visiting user is obtained, and the HMM hidden Markov model is used to obtain the conversion rule information of the hidden state of the user behavior in the personal visit information over time, so as to obtain the user's visit preference data;

[0006] The initial location information data of the user's visit is locked by a particle filter positioning algorithm, and a visit route is set according to the user's visit preference data and the user's visit preference data to obtain the user's visit route information;

[0007] Obtaining real-time feedback information from the user based on the user's tour route information, analyzing the real-time feedback information using a sliding window time analysis algorithm, and comparing the user's behavior similarities in different time periods in combination with a DTW dynamic time warping algorithm to obtain information on changes in user behavior habits;

[0008] Adjust the user's tour route information according to the user's behavior habit change information to obtain initial user's tour route information;

[0009] The FSM finite state machine model is used to obtain the task progress information of the user in the initial user tour route information, and the event association mining algorithm is combined to obtain the hidden logical association information between different events to obtain the user tour progress information;

[0010] The user status information of the user during the visit is obtained, and the target path is planned according to the user status information and the user visit progress information to obtain the target user visit route information.

[0011] In a possible implementation, the image sensor is used to obtain personal visit information of visiting users, and the HMM hidden Markov model is used to obtain the conversion rule information of the hidden state of user behavior over time in the personal visit information to obtain the user's visit preference data, including:

[0012] Arrange an image sensor inside the exhibition hall, and obtain personal visit information of the visiting user based on the image sensor, wherein the personal visit information includes at least gender information, age information, historical visit record information and preference information;

[0013] Using the HMM hidden Markov model to obtain the transformation regularity information of the hidden state of user behavior over time in the personal visit information;

[0014] According to the scene characteristics of the exhibition hall, the hidden state space S of the HMM hidden Markov model is defined as S = {s 1 ,s 2 ,···,s n}, where s 1 Indicates the fast browsing state, characterized by the user moving quickly and staying in front of the exhibit for less than 5 seconds; 2 Set to a medium attention state, with a stay time between 5-30 seconds;

[0015] Initial transition probability matrix A = [a ij ]n×n, where a ij represents the probability of transitioning from state i to state j, and the emission probability matrix B = [b j (O t )]n×m,b j (O t ) is the observation of O in state j t probability;

[0016] Using Gaussian mixture model fitting, assuming that each dimension of the observation vector obeys normal distribution, its probability density function is:

[0017]

[0018] Among them, x is the observation vector, μ is the mean vector, ∑ is the covariance matrix, after historical data clustering and parameter estimation initialization, the initial state probability distribution π = [π 1 ,π 2 ,···,π n ] T ,π is set according to the statistics of the initial user behavior at the entrance;

[0019] The parameters of the HMM hidden Markov model are iteratively optimized using the Baum-Weich algorithm. In each iteration, the probability of each state under a given observation sequence is calculated using the forward-backward algorithm. The transition probability is updated based on the variable update transition probability, the emission probability and the initial state probability. After iteration, the model likelihood is improved to be less than the set threshold value of 0.001.

[0020] For the observation sequence of newly collected users, the Viterbi algorithm is used to decode and obtain the hidden state sequence. The optimal path is traced back to determine the law of user behavior state conversion over time, and the user's interest-related states are counted to obtain the user's visit preference data.

[0021] In a possible implementation, the initial location information data of the user's visit is locked by a particle filter positioning algorithm, and a visit route is set according to the user's visit preference data and the user's visit preference data to obtain the user's visit route information, including:

[0022] Deploy positioning sensors in the exhibition hall, wherein the positioning sensors include at least an image sensor, an ultra-wideband positioning base station and a low-power Bluetooth beacon; obtain location information data of the user's visit according to the positioning sensors;

[0023] According to the location map and the location information data, randomly spread the particle group in the area where the user may appear, assign an initial weight to each particle, and calculate the particle likelihood according to the initial distance measurement and angle measurement data of the sensor;

[0024] When the user starts to move, the next position of each particle is predicted based on the dynamic environment information of the venue and the user's historical trajectory characteristics; the particle group is dynamically adjusted based on the standard deviation of the user's walking speed and the standard deviation of the turning angle to simulate the uncertainty of real movement, and the particle group is spread to the new position range that the user may move to, and the new particle group position is obtained;

[0025] Update particle weights based on real-time feedback data from sensors, use likelihood functions to calculate the degree of match between each particle and observation, eliminate low-weight particles, replicate and supplement near high-weight particles, and iterate to obtain initial position information data;

[0026] Based on the A* path planning algorithm, a tour route is set according to the user's tour preference data and the user's tour preference data to obtain user tour route information.

[0027] In a possible implementation, the real-time feedback information of the user in the user's tour route information is obtained, the real-time feedback information is analyzed using a sliding window time analysis algorithm, and the behavior similarity of the user in different time periods is compared in combination with a DTW dynamic time warping algorithm to obtain the user's behavior habit change information, including:

[0028] Acquire real-time feedback information from the user based on the user's tour route information, wherein the real-time feedback information at least includes exhibit interaction information, user communication content information, and crowd density information;

[0029] Determine the sliding window parameters, the window length is 3-5 minutes, and use the multivariate statistical analysis method to calculate the location-related features of the data in each sliding window. The related features include at least the length of the user's movement trajectory, the average speed, and the proportion of the stay time in each area;

[0030] The behavior feature vectors extracted by sliding windows in different time periods are used as the input of the DTW dynamic time warping algorithm, and a distance measurement matrix is ​​established. The Euclidean distance is used to measure the location feature differences of each dimension of the feature vector.

[0031] Based on the distance metric matrix, dynamic programming is used to search for the optimal regularized path. The DTW regularized path distance sequences of all window pairs are traversed, and the moving average method is used to analyze their changing trends over time to obtain information on changes in user behavior habits.

[0032] In a possible implementation, adjusting the user tour route information according to the user behavior habit change information to obtain the initial user tour route information includes:

[0033] Extract the moving speed index and the pause interval rhythm index from the user behavior habit change information, use the speed sensor to fuse the image tracking data, calculate the user's pace mean and variance in real time, and obtain the user's visiting rhythm data;

[0034] Obtain real-time status data of the visiting places, update the map information database, obtain the real-time density of people flow in each exhibition area, and expand the search layer by layer from the user's node as the root;

[0035] The heuristic function is used to calculate the distance from the node to the potential target of interest, the degree of crowd congestion and the expected stay time to weightedly evaluate the path score to screen out multiple candidate routes and obtain the initial user tour route information.

[0036] In a possible implementation, the FSM finite state machine model is used to obtain the task progress information of the user in the initial user tour route information, and the event association mining algorithm is combined to obtain the hidden logical association information between different events to obtain the user tour progress information, including:

[0037] Using the FSM finite state machine model to obtain multiple visiting tasks in the exhibition hall, the multiple visiting tasks at least include knowledge question and answer challenges, experimental operation tasks, treasure hunting and punching tasks, and immersive plot experience tasks;

[0038] Based on the multi-visit tasks, feature engineering is performed to extract period features and interval features from the time dimension; location features are used to extract exhibition area categories and exhibit popularity levels; user role features are used to extract students, parent-child families, and professionals;

[0039] The original events are converted into high-dimensional feature vectors, and the numerical features are processed using one-hot encoding. The pre-processed feature vector data is input into the April association mining algorithm. The minimum support is set to 10% and the confidence threshold is set to 70% to obtain the hidden logical association information between different events and obtain the user's visit progress information.

[0040] In a possible implementation, the acquiring of user status information during a user's visit, performing target path planning according to the user status information and user visit progress information, and obtaining target user visit route information includes:

[0041] Acquire user status information of the user during the visit, wherein the user status information at least includes collecting the user's heart rate information, body temperature information and skin conductivity information;

[0042] The SDNN value based on time domain analysis evaluates the activity of the user's autonomic nervous system and determines the user's fatigue level and excitement state. The SDNN value calculation formula is as follows:

[0043]

[0044] The mean value of the inter-column period;

[0045] According to the user's fatigue level, excitement state, body temperature information and skin conductivity information, the body temperature information is obtained at the same time to determine whether the user is in an uncomfortable state. If the user is in an uncomfortable state, the user's expression image information is obtained;

[0046] An analysis is performed based on the user's facial expression image information and discomfort state to obtain an analysis result, and a target path is planned according to the analysis result to obtain target user's visiting route information.

[0047] In a second aspect, the present application provides a multi-role multi-modal virtual-real composite interaction device, wherein the multi-modal virtual-real composite interaction device comprises the following modules:

[0048] A user preference analysis module is used to obtain personal visit information of visiting users based on image sensors, and to obtain the conversion rule information of the hidden state of user behavior over time in the personal visit information using the HMM hidden Markov model to obtain the user visit preference data;

[0049] A tour route planning module is used to lock the initial location information data of the user's visit through a particle filter positioning algorithm, set the tour route according to the user's visit preference data and the user's visit preference data, and obtain the user's tour route information;

[0050] The user habit analysis module is used to obtain the real-time feedback information of the user based on the user's tour route information, analyze the real-time feedback information using the sliding window time analysis algorithm, and compare the similarity of the user's behavior in different time periods in combination with the DTW dynamic time warping algorithm to obtain the user's behavior habit change information;

[0051] A tour route adjustment module is used to adjust the user tour route information according to the user behavior habit change information to obtain the initial user tour route information;

[0052] A visit progress evaluation module is used to obtain the task progress information of the user in the initial user visit route information by using the FSM finite state machine model, and to obtain the hidden logical association information between different events by combining the event association mining algorithm to obtain the user visit progress information;

[0053] The target route planning module is used to obtain the user status information of the user during the visit, and to plan the target path according to the user status information and the user visit progress information to obtain the target user visit route information.

[0054] In the third aspect, the present application provides a multi-role multi-modal virtual-reality composite interaction device, comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calls the instructions in the memory to enable the multi-role multi-modal virtual-reality composite interaction device to perform each step of the above-mentioned multi-role multi-modal virtual-reality composite interaction method.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute each step of the above-mentioned multi-role multi-modal virtual-reality composite interaction method.

[0056] Based on the method provided by the present application, the personal visit information of the visiting user is obtained through an image sensor, and the HMM hidden Markov model is used to obtain the conversion law information of the hidden state of the user behavior over time in the personal visit information to obtain the user's visit preference data; the initial location information data of the user's visit is locked through the particle filter positioning algorithm, and the visit route is set according to the user's visit preference data and the user's visit preference data to obtain the user's visit route information; the real-time feedback information of the user in the user's visit route information is obtained, and the real-time feedback information is analyzed by the sliding window time analysis algorithm, and the user's behavior similarity in different time periods is compared with the DTW dynamic time warping algorithm to obtain the user's behavior habit change information; the user's visit route information is adjusted according to the user's behavior habit change information to obtain the initial user's visit route information; the FSM finite state machine model is used to obtain the user's task progress information in the initial user's visit route information, and the event association mining algorithm is combined to obtain the hidden logical association information between different events to obtain the user's visit progress information; the user's state information during the user's visit is obtained, and the target path is planned according to the user's state information and the user's visit progress information to obtain the target user's visit route information. 1. Dynamically generate and update the user's cognitive map. 1. Enable the system to make instant adjustments based on user behavior and cognitive changes, improving the relevance and effectiveness of the interactive experience. 2. Convert multidimensional factors such as user space, behavior, and interests into a cognitive coordinate system. Including the comprehensive application of dimensions such as time, space, role, and theme, providing an accurate basis for the formulation of personalized interaction rules. 3. By identifying different user roles, optimize virtual character behavior and interactive content in a targeted manner. Each role has different display and interaction strategies to ensure that the needs of diverse users are met. 4. Integrate multimodal information such as voice, vision, and action to create a flexible interaction mode. Multimodal fusion technology allows users to interact with the system in a variety of ways to improve ease of use and immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of an embodiment of the first multi-role multi-modal virtual-real composite interaction method provided in the embodiments of the present application;

[0058] Figure 2 This is a flow chart of an embodiment of a second multi-role multi-modal virtual-real composite interaction method provided in an embodiment of the present application;

[0059] Figure 3 This is a flow chart of an embodiment of a third multi-role multi-modal virtual-real composite interaction method provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of the structure of a first multi-role multi-modal virtual-reality composite interaction device provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of the structure of a multi-role multi-modal virtual-reality composite interactive device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The embodiment of the present application provides a multi-role and multi-modal virtual-real composite interaction method, which can store asset registration in a database and audit asset change request information through a multiple audit mechanism, thereby solving the problem of poor security in daily asset management.

[0063] The technical scheme in the embodiment of the present application will be described below in conjunction with the accompanying drawings in the embodiment of the present application. Wherein, the terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments described here can be implemented in a sequence other than the content illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0064] It is understandable that any part of the data acquisition or collection involved in this application has been authorized by the user.

[0065] It is understandable that the execution subject of the present application can be a multi-role multi-modal virtual-reality composite interaction device, or a mobile terminal or a server, which is not specifically limited here.

[0066] The specific process of the embodiment of the present application is described below. Figure 1 , Figure 1 The flowchart of an embodiment of a multi-role multi-modal virtual-real composite interaction method provided in an embodiment of the present application includes:

[0067] 101. Obtain personal visit information of visiting users based on image sensors, use HMM hidden Markov model to obtain the conversion rule information of user behavior in personal visit information over time, and obtain user visit preference data;

[0068] It is understandable that an image sensor is arranged inside the exhibition hall, and personal visit information of the visiting user is obtained based on the image sensor, and the personal visit information at least includes gender information, age information, historical visit record information and preference information;

[0069] The HMM hidden Markov model is used to obtain the transformation regularity information of the hidden state of user behavior over time in personal visit information;

[0070] According to the scene characteristics of the exhibition hall, the hidden state space S of the HMM hidden Markov model is defined as S = {s 1 ,s 2 ,···,s n}, where s 1 Indicates the fast browsing state, characterized by the user moving quickly and staying in front of the exhibit for less than 5 seconds; 2 Set to a medium attention state, with a stay time between 5-30 seconds;

[0071] Initial transition probability matrix A = [a ij ]n×n, where a ij represents the probability of transitioning from state i to state j, and the emission probability matrix B = [b j (O t )]n×m,b j (O t ) is the observation of O in state j t probability;

[0072] Using Gaussian mixture model fitting, assuming that each dimension of the observation vector obeys normal distribution, its probability density function is:

[0073]

[0074] Among them, x is the observation vector, μ is the mean vector, ∑ is the covariance matrix, after historical data clustering and parameter estimation initialization, the initial state probability distribution π = [π 1 ,π 2 ,···,π n ] T ,π is set according to the statistics of the initial user behavior at the entrance;

[0075] The Baum-Weich algorithm is used to iteratively optimize the parameters of the HMM hidden Markov model. In each iteration, the forward-backward algorithm is used to calculate the probability of each state under a given observation sequence. The transition probability is updated based on the variable update transition probability, emission probability and initial state probability. After iteration, the model likelihood is improved to less than the set threshold of 0.001.

[0076] For the observation sequence of newly collected users, the Viterbi algorithm is used to decode and obtain the hidden state sequence. The optimal path is traced back to determine the law of user behavior state conversion over time, and the user's interest-related states are counted to obtain the user's visit preference data.

[0077] 102. Lock the initial location information data of the user's visit through the particle filter positioning algorithm, set the visit route according to the user's visit preference data and the user's visit preference data, and obtain the user's visit route information;

[0078] It is understandable that positioning sensors are deployed in the exhibition hall, and the positioning sensors include at least image sensors, ultra-wideband positioning base stations and low-power Bluetooth beacons; the location information data of the user's visit is obtained according to the positioning sensors;

[0079] According to the site map and location information data, randomly spread particle groups in areas where users may appear, assign initial weights to each particle, and calculate particle likelihood based on the initial distance and angle measurement data of the sensor;

[0080] When the user starts to move, the next position of each particle is predicted based on the dynamic environment information of the venue and the user's historical trajectory characteristics; the particle group is dynamically adjusted based on the standard deviation of the user's walking speed and the standard deviation of the turning angle to simulate the uncertainty of real movement, and the particle group is spread to the new position range that the user may move to, and the new particle group position is obtained;

[0081] Update particle weights based on real-time feedback data from sensors, use likelihood functions to calculate the degree of match between each particle and observation, eliminate low-weight particles, replicate and supplement near high-weight particles, and iterate to obtain initial position information data;

[0082] Based on the A* path planning algorithm, the tour route is set according to the user's tour preference data and the user's tour preference data to obtain the user's tour route information.

[0083] 103. Obtain real-time feedback information from users based on their tour route information, analyze the real-time feedback information using a sliding window time analysis algorithm, and compare the similarity of user behaviors at different time periods using a DTW dynamic time warping algorithm to obtain information on changes in user behavior habits;

[0084] It is understandable that the real-time feedback information of the user based on the user's tour route information is obtained, and the real-time feedback information at least includes exhibit interaction information, user communication content information, and crowd density information;

[0085] Determine the sliding window parameters, with a window length of 3-5 minutes. For the data in each sliding window, use multivariate statistical analysis methods to calculate location-related features. The relevant features include at least the length of the user's mobile trajectory, average speed, and the proportion of stay time in each area.

[0086] The behavior feature vectors extracted by sliding windows in different time periods are used as the input of the DTW dynamic time warping algorithm, and a distance measurement matrix is ​​established. The Euclidean distance is used to measure the location feature differences of each dimension of the feature vector.

[0087] Based on the distance metric matrix, dynamic programming is used to search for the optimal regularized path. The DTW regularized path distance sequences of all window pairs are traversed, and the moving average method is used to analyze their changing trends over time to obtain information on changes in user behavior habits.

[0088] 104. Adjust the user's tour route information according to the user's behavior habit change information to obtain the initial user's tour route information;

[0089] It can be understood that the moving speed index and pause interval rhythm index in the user behavior habit change information are extracted, and the speed sensor is used to fuse the image tracking data to calculate the user's pace mean and variance in real time to obtain the user's visit rhythm data;

[0090] Obtain real-time status data of the visiting places, update the map information database, obtain the real-time density of people flow in each exhibition area, and expand the search layer by layer from the user's node as the root;

[0091] The heuristic function is used to calculate the distance from the node to the potential target of interest, the degree of crowd congestion and the expected stay time to weightedly evaluate the path score to screen out multiple candidate routes and obtain the initial user tour route information.

[0092] 105. Use the FSM finite state machine model to obtain the user's task progress information in the initial user tour route information, and combine the event association mining algorithm to obtain the hidden logical association information between different events to obtain the user's tour progress information;

[0093] It can be understood that the FSM finite state machine model is used to obtain multiple visiting tasks in the exhibition hall, and the multiple visiting tasks at least include knowledge question and answer challenges, experimental operation tasks, treasure hunting and punching tasks, and immersive plot experience tasks;

[0094] Feature engineering is performed based on multiple visit tasks to extract period features and interval features from the time dimension; location features are used to extract exhibition area categories and exhibit popularity levels; user role features are used to extract students, parent-child families, and professionals;

[0095] The original events are converted into high-dimensional feature vectors, and the numerical features are processed using one-hot encoding. The pre-processed feature vector data is input into the April association mining algorithm. The minimum support is set to 10% and the confidence threshold is set to 70% to obtain the hidden logical association information between different events and obtain the user's visit progress information.

[0096] 106. Obtain user status information during the user's visit, plan a target path based on the user status information and the user's visit progress information, and obtain the target user's visit route information.

[0097] It can be understood that the user status information of the user during the visit is obtained, and the user status information at least includes collecting the user's heart rate information, body temperature information and skin conductivity information;

[0098] The SDNN value based on time domain analysis evaluates the activity of the user's autonomic nervous system and determines the user's fatigue level and excitement state. The SDNN value calculation formula is as follows:

[0099]

[0100] Where N is the length of the RR sequence between heart beats, RR i is the i-th cardiac interval, is the average value of the RR sequence interval;

[0101] According to the user's fatigue level, excitement state, body temperature information and skin conductivity information, the body temperature information is obtained at the same time to determine whether the user is in an uncomfortable state. If the user is in an uncomfortable state, the user's expression image information is obtained;

[0102] An analysis is performed based on the user's facial expression image information and discomfort state to obtain an analysis result, and a target path is planned based on the analysis result to obtain the target user's visiting route information.

[0103] Based on the method provided by the present application, the personal visit information of the visiting user is obtained through an image sensor, and the HMM hidden Markov model is used to obtain the conversion law information of the hidden state of the user behavior over time in the personal visit information, so as to obtain the user's visit preference data; the initial location information data of the user's visit is locked through the particle filter positioning algorithm, and the visit route is set according to the user's visit preference data and the user's visit preference data to obtain the user's visit route information; the real-time feedback information of the user in the user's visit route information is obtained, and the real-time feedback information is analyzed by the sliding window time analysis algorithm, and the user's behavior similarity in different time periods is compared with the DTW dynamic time warping algorithm to obtain the user's behavior habit change information; the user's visit route information is adjusted according to the user's behavior habit change information to obtain the initial user's visit route information; the FSM finite state machine model is used to obtain the user's task progress information in the initial user's visit route information, and the event association mining algorithm is combined to obtain the hidden logical association information between different events to obtain the user's visit progress information; the user's state information during the visit is obtained, and the target path is planned according to the user's state information and the user's visit progress information to obtain the target user's visit route information. 1. Dynamically generate and update the user's cognitive map. 1. Enable the system to make instant adjustments based on user behavior and cognitive changes, improving the relevance and effectiveness of the interactive experience. 2. Convert multidimensional factors such as user space, behavior, and interests into a cognitive coordinate system. Including the comprehensive application of dimensions such as time, space, role, and theme, providing an accurate basis for the formulation of personalized interaction rules. 3. By identifying different user roles, optimize virtual character behavior and interactive content in a targeted manner. Each role has different display and interaction strategies to ensure that the needs of diverse users are met. 4. Integrate multimodal information such as voice, vision, and action to create a flexible interaction mode. Multimodal fusion technology allows users to interact with the system in a variety of ways to improve ease of use and immersion.

[0104] See also Figure 2 , Figure 2 The flowchart of the third embodiment of the multi-role multi-modal virtual-real composite interaction method provided in the embodiment of the present application includes:

[0105] 201. Obtaining real-time feedback information from the user based on the user's tour route information, where the real-time feedback information at least includes exhibit interaction information, user communication content information, and crowd density information;

[0106] 202. Determine sliding window parameters, with a window length of 3-5 minutes. For the data in each sliding window, use multivariate statistical analysis methods to calculate location-related features. The related features at least include the user's moving trajectory length, average speed, and proportion of stay time in each area.

[0107] 203. The behavior feature vectors extracted by sliding windows in different time periods are used as the input of the DTW dynamic time warping algorithm, a distance measurement matrix is ​​established, and the Euclidean distance is used to measure the position feature differences of each dimension of the feature vector;

[0108] 204. Based on the distance metric matrix, dynamic programming is used to search for the optimal regularized path, the DTW regularized path distance sequence of all window pairs is traversed, and the moving average method is used to analyze its change trend over time to obtain the change information of user behavior habits.

[0109] Based on the method provided in the embodiment of the present application, by obtaining real-time feedback information from users based on the user's tour route information, the real-time feedback information is analyzed using a sliding window time analysis algorithm, and the DTW dynamic time warping algorithm is used to compare the similarities of user behaviors in different time periods to obtain information on changes in user behavior habits. This achieves a keen capture and in-depth analysis of user visit behaviors. The sliding window time analysis algorithm sorts out real-time feedback in sequence and focuses on time period features; the DTW algorithm crosses time differences and accurately compares behaviors. This can accurately locate changes in user behavior habits, such as interest transfer points and rhythm changes, and provide key basis for the venue to adjust services and optimize exhibit recommendations in a timely manner, greatly improving the user's personalized experience and making their visit more in line with their own needs.

[0110] See also Figure 3 , Figure 3 The flowchart of the third embodiment of the multi-role multi-modal virtual-real composite interaction method provided in the embodiment of the present application includes:

[0111] 301. Acquire user status information of the user during the visit, where the user status information at least includes collecting the user's heart rate information, body temperature information, and skin conductivity information;

[0112] 302. The SDNN value based on time domain analysis evaluates the activity of the user's autonomic nervous system and determines the user's fatigue level and excitement state. The SDNN value calculation formula is as follows:

[0113]

[0114] Where N is the length of the RR sequence between heart beats, RR i is the i-th cardiac interval, is the average value of the RR sequence interval;

[0115] 303. According to the user's fatigue level, excitement state, body temperature information and skin conductivity information, the body temperature information is obtained at the same time to determine whether the user is in an uncomfortable state. If the user is in an uncomfortable state, the user's expression image information is obtained;

[0116] 304. Perform analysis based on the user's facial expression image information and discomfort state to obtain analysis results, perform target path planning based on the analysis results, and obtain target user's tour route information.

[0117] Based on the method provided in the embodiment of the present application, the user status information of the user during the visit is obtained, and the target path is planned according to the user status information and the user's visit progress information to obtain the target user's visit route information. This avoids user fatigue and boredom and improves comfort; on the other hand, it matches the interests and progress, helps users to efficiently explore their favorite exhibits, makes the visit process smooth and full of surprises, and enhances user immersion and satisfaction.

[0118] The above describes the multi-role multi-modal virtual-reality composite interaction method in the embodiment of the present application. The following describes the multi-role multi-modal virtual-reality composite interaction device in the embodiment of the present application. Figure 4 , Figure 4 A schematic diagram of the structure of a multi-role multi-modal virtual-reality composite interaction device provided in an embodiment of the present application includes:

[0119] The user preference analysis module 401 is used to obtain the personal visit information of the visiting user based on the image sensor, and use the HMM hidden Markov model to obtain the conversion rule information of the hidden state of the user behavior in the personal visit information over time, so as to obtain the user's visit preference data;

[0120] The tour route planning module 402 is used to lock the initial location information data of the user's tour through the particle filter positioning algorithm, set the tour route according to the user's tour preference data and the user's tour preference data, and obtain the user's tour route information;

[0121] User habit analysis module 403 is used to obtain real-time feedback information from users based on user tour route information, analyze the real-time feedback information using a sliding window time analysis algorithm, and compare the similarity of user behaviors in different time periods in combination with a DTW dynamic time warping algorithm to obtain information on changes in user behavior habits;

[0122] A tour route adjustment module 404 is used to adjust the user tour route information according to the user behavior habit change information to obtain the initial user tour route information;

[0123] The visit progress evaluation module 405 is used to obtain the task progress information of the user in the initial user visit route information by using the FSM finite state machine model, and obtain the hidden logical association information between different events by combining the event association mining algorithm to obtain the user visit progress information;

[0124] The target route planning module 406 is used to obtain the user status information of the user during the visit, and to plan the target path according to the user status information and the user visit progress information to obtain the target user visit route information.

[0125] Based on the device provided by the present application, the image sensor is received to obtain the personal visit information of the visiting user, and the HMM hidden Markov model is used to obtain the conversion law information of the hidden state of the user behavior over time in the personal visit information, so as to obtain the user's visit preference data; the initial location information data of the user's visit is locked by the particle filter positioning algorithm, and the visit route is set according to the user's visit preference data and the user's visit preference data to obtain the user's visit route information; the real-time feedback information of the user in the user's visit route information is obtained, and the real-time feedback information is analyzed by the sliding window time analysis algorithm, and the user's behavior similarity in different time periods is compared with the DTW dynamic time warping algorithm to obtain the user's behavior habit change information; the user's visit route information is adjusted according to the user's behavior habit change information to obtain the initial user's visit route information; the FSM finite state machine model is used to obtain the user's task progress information in the initial user's visit route information, and the event association mining algorithm is combined to obtain the hidden logical association information between different events to obtain the user's visit progress information; the user's status information during the user's visit is obtained, and the target path is planned according to the user's status information and the user's visit progress information to obtain the target user's visit route information. 1. Dynamically generate and update the user's cognitive map. 1. Enable the system to make instant adjustments based on user behavior and cognitive changes, improving the relevance and effectiveness of the interactive experience. 2. Convert multidimensional factors such as user space, behavior, and interests into a cognitive coordinate system. Including the comprehensive application of dimensions such as time, space, role, and theme, providing an accurate basis for the formulation of personalized interaction rules. 3. By identifying different user roles, optimize virtual character behavior and interactive content in a targeted manner. Each role has different display and interaction strategies to ensure that the needs of diverse users are met. 4. Integrate multimodal information such as voice, vision, and action to create a flexible interaction mode. Multimodal fusion technology allows users to interact with the system in a variety of ways to improve ease of use and immersion.

[0126] See also Figure 5 , Figure 51 is a schematic diagram of the structure of a multi-role multi-modal virtual-reality composite interactive device provided by an embodiment of the present application. The multi-role multi-modal virtual-reality composite interactive device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 may be temporary storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the multi-role multi-modal virtual-reality composite interactive device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 5630 to execute a series of instruction operations in the storage medium 530 on the multi-role multi-modal virtual-reality composite interactive device 500.

[0127] The multi-role multi-modal virtual-reality composite interactive device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 5 The structure of the multi-role multi-modal virtual-reality composite interactive device shown does not constitute a limitation of the multi-role multi-modal virtual-reality composite interactive device, and may include more or less components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0128] The present application also provides a multi-role multi-modal virtual-reality composite interaction device, the computer device includes a memory and a processor, the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes each step of the multi-role multi-modal virtual-reality composite interaction method in the above-mentioned embodiments.

[0129] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes each step of the multi-role multi-modal virtual-reality composite interaction method.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0132] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0133] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-role multi-modal virtual-real composite interaction method, characterized in that: The multi-modal virtual-real composite interaction method comprises the following steps: Based on the image sensor, the personal visit information of the visiting user is obtained, and the HMM hidden Markov model is used to obtain the conversion rule information of the hidden state of the user behavior in the personal visit information over time, so as to obtain the user's visit preference data; The initial location information data of the user's visit is locked by a particle filter positioning algorithm, and a visit route is set according to the user's visit preference data and the user's visit preference data to obtain the user's visit route information; Obtaining real-time feedback information from the user based on the user's tour route information, analyzing the real-time feedback information using a sliding window time analysis algorithm, and comparing the user's behavior similarities in different time periods in combination with a DTW dynamic time warping algorithm to obtain information on changes in user behavior habits; Adjust the user's tour route information according to the user's behavior habit change information to obtain initial user's tour route information; The FSM finite state machine model is used to obtain the task progress information of the user in the initial user tour route information, and the event association mining algorithm is combined to obtain the hidden logical association information between different events to obtain the user tour progress information; The user status information of the user during the visit is obtained, and the target path is planned according to the user status information and the user visit progress information to obtain the target user visit route information.

2. A multi-role multi-modal virtual-reality composite interaction method as claimed in claim 1, characterized in that: The method of obtaining personal visit information of visiting users based on image sensors, and obtaining the transformation rule information of the hidden state of user behavior over time in the personal visit information by using the HMM hidden Markov model, and obtaining the user visit preference data includes: Arrange an image sensor inside the exhibition hall, and obtain personal visit information of the visiting user based on the image sensor, wherein the personal visit information includes at least gender information, age information, historical visit record information and preference information; Using the HMM hidden Markov model to obtain the transformation regularity information of the hidden state of user behavior over time in the personal visit information; According to the scene characteristics of the exhibition hall, the hidden state space S of the HMM hidden Markov model is defined as S = {s1, s2, ..., s n }, where s1 represents the fast browsing state, characterized by the user moving quickly and staying in front of the exhibit for less than 5 seconds; s2 is set as the moderate attention state, with a stay time between 5 and 30 seconds; Initial transition probability matrix A = [a ij ]n×n, where a ij represents the probability of transitioning from state i to state j, and the emission probability matrix B = [b j (O t )]n×m,b j (O t ) is the observation of O in state j t probability; Using Gaussian mixture model fitting, assuming that each dimension of the observation vector obeys normal distribution, its probability density function is: Among them, x is the observation vector, μ is the mean vector, ∑ is the covariance matrix, after historical data clustering and parameter estimation initialization, the initial state probability distribution π = [π1, π2, ..., π n ] T ,π is set according to the statistics of the initial user behavior at the entrance; The parameters of the HMM hidden Markov model are iteratively optimized using the Baum-Welch algorithm. In each iteration, the probability of each state under a given observation sequence is calculated using a forward-backward algorithm. The transition probability is updated based on the variable update transition probability, the emission probability and the initial state probability. After iteration, the model likelihood is improved to be less than the set threshold value of 0.

001. For the observation sequence of newly collected users, the Viterbi algorithm is used to decode and obtain the hidden state sequence. The optimal path is traced back to determine the law of user behavior state conversion over time, and the user's interest-related states are counted to obtain the user's visit preference data.

3. A multi-role multi-modal virtual-reality composite interaction method as claimed in claim 1, characterized in that: The initial location information data of the user's visit is locked by the particle filter positioning algorithm, and the visit route is set according to the user's visit preference data and the user's visit preference data to obtain the user's visit route information, including: Deploy positioning sensors in the exhibition hall, wherein the positioning sensors include at least an image sensor, an ultra-wideband positioning base station and a low-power Bluetooth beacon; obtain location information data of the user's visit according to the positioning sensors; According to the location map and the location information data, randomly spread the particle group in the area where the user may appear, assign an initial weight to each particle, and calculate the particle likelihood according to the initial distance measurement and angle measurement data of the sensor; When the user starts to move, the next position of each particle is predicted based on the dynamic environment information of the venue and the user's historical trajectory characteristics; the particle group is dynamically adjusted based on the standard deviation of the user's walking speed and the standard deviation of the turning angle to simulate the uncertainty of real movement, and the particle group is spread to the new position range that the user may move to, and the new particle group position is obtained; Update particle weights based on real-time feedback data from sensors, use likelihood functions to calculate the degree of match between each particle and observation, eliminate low-weight particles, replicate and supplement near high-weight particles, and iterate to obtain initial position information data; Based on the A* path planning algorithm, a tour route is set according to the user's tour preference data and the user's tour preference data to obtain user tour route information.

4. A multi-role multi-modal virtual-reality composite interaction method as claimed in claim 1, characterized in that: The real-time feedback information of the user is obtained according to the user's tour route information, the real-time feedback information is analyzed by using a sliding window time analysis algorithm, and the behavior similarity of the user in different time periods is compared in combination with a DTW dynamic time warping algorithm to obtain the user's behavior habit change information, including: Acquire real-time feedback information from the user based on the user's tour route information, wherein the real-time feedback information at least includes exhibit interaction information, user communication content information, and crowd density information; Determine the sliding window parameters, the window length is 3-5 minutes, and use the multivariate statistical analysis method to calculate the location-related features of the data in each sliding window. The related features include at least the length of the user's movement trajectory, the average speed, and the proportion of the stay time in each area; The behavior feature vectors extracted by sliding windows in different time periods are used as the input of the DTW dynamic time warping algorithm, and a distance measurement matrix is ​​established. The Euclidean distance is used to measure the location feature differences of each dimension of the feature vector. Based on the distance metric matrix, dynamic programming is used to search for the optimal regularized path. The DTW regularized path distance sequences of all window pairs are traversed, and the moving average method is used to analyze their changing trends over time to obtain information on changes in user behavior habits.

5. A multi-role multi-modal virtual-reality composite interaction method as claimed in claim 1, characterized in that: The adjusting the user tour route information according to the user behavior habit change information to obtain the initial user tour route information includes: Extract the moving speed index and the pause interval rhythm index from the user behavior habit change information, use the speed sensor to fuse the image tracking data, calculate the user's pace mean and variance in real time, and obtain the user's visiting rhythm data; Obtain real-time status data of the visiting places, update the map information database, obtain the real-time density of people flow in each exhibition area, and expand the search layer by layer from the user's node as the root; The heuristic function is used to calculate the distance from the node to the potential target of interest, the degree of crowd congestion and the expected stay time to weightedly evaluate the path score to screen out multiple candidate routes and obtain the initial user tour route information.

6. A multi-role multi-modal virtual-reality composite interaction method as claimed in claim 1, characterized in that: The FSM finite state machine model is used to obtain the task progress information of the user in the initial user tour route information, and the event association mining algorithm is combined to obtain the hidden logical association information between different events to obtain the user tour progress information, including: Using the FSM finite state machine model to obtain multiple visiting tasks in the exhibition hall, the multiple visiting tasks at least include knowledge question and answer challenges, experimental operation tasks, treasure hunting and punching tasks, and immersive plot experience tasks; Based on the multi-visit tasks, feature engineering is performed to extract period features and interval features from the time dimension; location features are used to extract exhibition area categories and exhibit popularity levels; user role features are used to extract students, parent-child families, and professionals; The original events are converted into high-dimensional feature vectors, and the numerical features are processed using one-hot encoding. The pre-processed feature vector data is input into the Apriori association mining algorithm. The minimum support is set to 10% and the confidence threshold is set to 70%. The hidden logical association information between different events is obtained, and the user's visit progress information is obtained.

7. A multi-role multi-modal virtual-reality composite interaction method as claimed in claim 1, characterized in that: The step of obtaining user status information during the user's visit, performing target path planning according to the user status information and the user's visit progress information, and obtaining the target user's visit route information includes: Acquire user status information of the user during the visit, wherein the user status information at least includes collecting the user's heart rate information, body temperature information and skin conductivity information; The SDNN value based on time domain analysis evaluates the activity of the user's autonomic nervous system and determines the user's fatigue level and excitement state. The SDNN value calculation formula is as follows: Where N is the length of the RR sequence between heart beats, RR i is the i-th cardiac interval, is the average value of the RR sequence interval; According to the user's fatigue level, excitement state, body temperature information and skin conductivity information, the body temperature information is obtained at the same time to determine whether the user is in an uncomfortable state. If the user is in an uncomfortable state, the user's expression image information is obtained; An analysis is performed based on the user's facial expression image information and discomfort state to obtain an analysis result, and a target path is planned according to the analysis result to obtain target user's visiting route information.

8. A multi-role multi-modal virtual-real composite interaction device, characterized in that: The multi-modal virtual-real composite interaction device comprises the following modules: A user preference analysis module is used to obtain personal visit information of visiting users based on image sensors, and to obtain the conversion rule information of the hidden state of user behavior over time in the personal visit information using the HMM hidden Markov model to obtain the user visit preference data; A tour route planning module is used to lock the initial location information data of the user's visit through a particle filter positioning algorithm, set the tour route according to the user's visit preference data and the user's visit preference data, and obtain the user's tour route information; The user habit analysis module is used to obtain the real-time feedback information of the user based on the user's tour route information, analyze the real-time feedback information using the sliding window time analysis algorithm, and compare the similarity of the user's behavior in different time periods in combination with the DTW dynamic time warping algorithm to obtain the user's behavior habit change information; A tour route adjustment module is used to adjust the user tour route information according to the user behavior habit change information to obtain the initial user tour route information; A visit progress evaluation module is used to obtain the task progress information of the user in the initial user visit route information by using the FSM finite state machine model, and to obtain the hidden logical association information between different events by combining the event association mining algorithm to obtain the user visit progress information; The target route planning module is used to obtain the user status information of the user during the visit, and to plan the target path according to the user status information and the user visit progress information to obtain the target user visit route information.

9. A multi-role multi-modal virtual-reality composite interactive device, characterized in that: The multi-role multi-modal virtual-reality composite interaction device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the multi-role multi-modal virtual-reality composite interaction device to execute each step of the multi-role multi-modal virtual-reality composite interaction method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the various steps of the multi-role multi-modal virtual-reality composite interaction method as described in any one of claims 1-7 are implemented.

Images