Immersive interaction system of multi-modal fusion virtual exhibition hall

Abstract

This invention relates to the field of virtual reality interaction technology, specifically disclosing a multimodal immersive interactive system for virtual exhibition halls. The system collects real-time multimodal interaction data from users and performs synchronous preprocessing. It inputs directional semantic terms into a type-two fuzzy logic system to generate a three-dimensional spatial semantic membership field. Based on the gesture pointing vector and hand tremor amplitude, it constructs a pointing cone region and establishes a layered two-dimensional Gaussian distribution to generate a gesture pointing probability field. Using the semantic membership field, gesture pointing probability field, and gaze point Gaussian kernel density field as independent evidence, it introduces wall boundaries and floor passages as spatial topological constraints, and uses evidence theory combination rules to fuse and generate a three-dimensional intent probability distribution field. The distribution field undergoes centroid defuzzification processing to extract the navigation target area, and a navigation path is planned using the user's current location as the starting point and the nearest floor passage entrance as the waypoint. This invention solves the problem of multimodal fuzzy intent understanding and spatial adaptation.

Description

Technical Field

[0001] This invention relates to the field of virtual reality interaction technology, and more specifically to an immersive interactive system for virtual exhibition halls that integrates multiple modalities. Background Technology

[0002] With the deep integration of virtual reality technology and the digital preservation of cultural heritage, virtual exhibition halls have become an important carrier for cultural display and distance education. In the virtual exhibition hall environment, how to achieve natural and efficient interaction between users and virtual exhibits and spaces is a technical issue of ongoing concern in this field. Existing technologies have developed various solutions for understanding user intent and navigation interaction in virtual environments. Regarding eye tracking, a head-mounted eye tracker captures the coordinates of the user's gaze point, and combined with spatial mapping of the virtual scene, it can achieve target selection or region of interest recognition based on the gaze point. Regarding gesture interaction, a depth camera captures the trajectory of hand skeleton points, and gesture commands are generated by calculating fingertip pointing or palm orientation, thereby controlling the movement of virtual characters or the operation of exhibits. Regarding voice interaction, a microphone array is used to collect voice signals, and semantic commands are extracted through automatic speech recognition technology to achieve scene switching or exhibit searching in the virtual exhibition hall.

[0003] Existing technologies lack a unified, spatially constrained ambiguity resolution mechanism for the semantic ambiguity of natural language directional words, the physical divergence of gesture directions, and the attentional fluctuations of eye-tracking fixation. As a result, when faced with multimodal collaborative commands such as "go there," which combine ambiguous semantics, rough directions, and instantaneous fixation, the system cannot accurately determine whether the user's intention is a point after the gesture ray passes through the wall, or an area that needs to be reached by detouring through a passage. This leads to command contradictions and path planning failures, causing virtual interaction to fall into a cognitive dilemma of "separation between intention understanding and spatial action." Summary of the Invention

[0004] The purpose of this invention is to provide an immersive interactive system for virtual exhibition halls that integrates multiple modalities, in order to solve the problems mentioned above.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] An immersive interactive system for virtual exhibition halls that integrates multiple modalities includes the following steps:

[0007] S1: Collect real-time multimodal interaction data of users. The multimodal interaction data includes speech signals, hand skeleton point trajectories, head orientation, and eye gaze point sequences. Perform timestamp alignment and noise filtering on the multimodal interaction data to obtain synchronized directional semantic words, gesture pointing vectors, head orientation angles, and gaze point coordinates.

[0008] S2: Input the directional semantic words into the interval type II fuzzy logic system, dynamically adjust the principal membership function and uncertainty coverage of the fuzzy set according to the head orientation angle, and generate a three-dimensional spatial semantic membership field describing the user's directional concept.

[0009] S3: Construct a pointing cone region based on the gesture pointing vector and the amplitude of hand shaking, and obtain the probability value of each coordinate point falling into the pointing cone region through spatial probability calculation to generate a gesture pointing probability field;

[0010] S4: Integrate the three-dimensional spatial semantic membership field, gesture pointing probability field, and gaze point coordinates, and introduce the wall boundaries and floor connection channels of the virtual exhibition hall as spatial topological constraints. Use evidence theory combination rules to calculate the joint confidence of each point in the space to satisfy the user's intent, and generate a three-dimensional intent probability distribution field.

[0011] S5: Perform centroid defuzzification on the three-dimensional intention probability distribution field, combine it with the reachable region mask in the spatial topological constraints, extract the probability condensation region as the navigation target region, and plan the navigation path through the floor connection passage based on the navigation target region.

[0012] As a further aspect of the present invention: S2 specifically includes:

[0013] Based on the directional semantic words, an initial interval type II fuzzy set is constructed. The initial interval type II fuzzy set consists of the principal membership function and the upper and lower membership functions that define the fuzzy range.

[0014] Based on the head orientation angle, the initial interval type II fuzzy set is shifted to the center of discourse so that the peak direction of the principal membership function is aligned with the head orientation angle, thus obtaining the orientation-corrected interval type II fuzzy set.

[0015] Based on the deviation between the gaze point coordinates and the head orientation angle, the interval width between the upper and lower membership functions is dynamically adjusted to obtain a gaze-weighted interval type II fuzzy set.

[0016] The gaze-weighted interval type II fuzzy set is mapped to the three-dimensional space of the virtual exhibition hall. By calculating the membership degree of each point in the space relative to the gaze-weighted interval type II fuzzy set, a three-dimensional spatial semantic membership field is generated.

[0017] As a further aspect of the present invention: obtaining the gaze-weighted interval type II fuzzy set specifically includes:

[0018] Calculate the spatial angle between the coordinates of the fixation point and the center line of the visual axis corresponding to the head orientation angle to obtain the fixation deviation.

[0019] A nonlinear scaling transformation is applied to the gaze deviation to generate an adjustment coefficient that reflects gaze uncertainty;

[0020] Based on the adjustment coefficient, the upper boundary value corresponding to the upper membership function is increased and the lower boundary value corresponding to the lower membership function is decreased, respectively, so as to expand the interval width between the upper membership function and the lower membership function.

[0021] The upper and lower membership functions after expanding the interval width are combined with the orientation-corrected interval type II fuzzy set to generate a gaze-weighted interval type II fuzzy set.

[0022] As a further aspect of the present invention: S3 specifically includes:

[0023] The main pointing axis is determined based on the hand pointing vector, the cone half angle is calculated based on the hand shaking amplitude, and the spatial boundary of the pointing cone region is constructed with the main pointing axis as the center line.

[0024] The directional cone region is divided into equal-distance layers along the main directional axis to obtain multiple spatial cross sections perpendicular to the main directional axis;

[0025] For each spatial section, a radial probability attenuation parameter is set according to the distance between the corresponding spatial section and the wrist reference point to establish a two-dimensional Gaussian distribution of the section.

[0026] Traverse all coordinate points in space, and calculate the probability value of each coordinate point belonging to the pointing cone region based on the two-dimensional Gaussian distribution of the spatial section to which each coordinate point belongs and the vertical distance from each coordinate point to the main pointing axis, and generate a gesture pointing probability field.

[0027] As a further aspect of the present invention: the establishment of the two-dimensional Gaussian distribution of the cross-section specifically includes:

[0028] Obtain the Euclidean distance between the current spatial section and the wrist reference point, and use the corresponding Euclidean distance as the radial distance variable;

[0029] A nonlinear mapping is performed on the radial distance variable to generate a Gaussian distribution standard deviation parameter. The larger the radial distance variable, the larger the standard deviation parameter value, so as to reflect the enhanced probability diffusion characteristics of the far-end section.

[0030] Based on the standard deviation parameter, determine the probability decay gradient of the two-dimensional Gaussian distribution of the cross section on the plane perpendicular to the principal pointing axis;

[0031] Using the point where the main pointing axis intersects the current spatial section as the distribution center, and combining the probability decay gradient, a two-dimensional Gaussian distribution corresponding to the spatial section is established.

[0032] As a further aspect of the present invention: S4 specifically includes:

[0033] The three-dimensional spatial semantic membership field, the gesture pointing probability field, and the Gaussian kernel density field generated from the gaze point coordinates are used as independent evidence to construct the basic confidence assignments for each piece of evidence.

[0034] Orthogonal sum rules are used to combine multiple basic confidence scores in pairs to gradually synthesize a joint confidence score that reflects multimodal consistency.

[0035] An inaccessible area mask is generated based on the wall boundaries of the virtual exhibition hall. The joint confidence value in the inaccessible area is set to zero to obtain the joint confidence value of obstacle correction.

[0036] Based on the spatial location of the floor connection passage, a directional gain is applied to the joint confidence of obstacle correction, which increases the confidence value of the passage entrance area and the internal area of ​​the passage, generating a three-dimensional intent probability distribution field.

[0037] As a further aspect of the present invention: the method of using orthogonal sum rules to pairwise combine multiple basic confidence level assignments to progressively synthesize a joint confidence level reflecting multimodal consistency specifically includes:

[0038] Select two from multiple basic confidence assignments as the current combination pair and calculate the conflict coefficient between the current combination pair;

[0039] When the conflict coefficient is less than the preset threshold, the current combination pair is fused using orthogonal sum rules to generate intermediate combination confidence assignments;

[0040] When the conflict coefficient is greater than or equal to a preset threshold, the current combination pair is subjected to a weighted average process, and the conflict coefficient is used as the basis for weight adjustment to generate an intermediate combination confidence assignment for conflict resolution.

[0041] The intermediate combined confidence level assignments and the remaining basic confidence level assignments are used as new combination pairs in turn. The above steps are repeated until all basic confidence level assignments are combined to obtain the joint confidence level.

[0042] As a further aspect of the present invention: S5 specifically includes:

[0043] The spatial centroid coordinates of the probability mass are obtained by integral calculation of the three-dimensional intention probability distribution field, and the corresponding spatial centroid coordinates are used as the initial target points.

[0044] Using the initial target point as the center, conduct an isosurface search outward to extract continuous spatial regions with probability values ​​higher than a preset threshold, thereby obtaining several candidate agglomeration regions.

[0045] The overlap between each candidate agglomeration region and the reachable region mask is calculated. Candidate agglomeration regions that do not intersect with the reachable region mask are eliminated, and candidate agglomeration regions that overlap with the reachable region mask are retained as navigation target regions.

[0046] Identify the floor connection entrance closest to the navigation target area, take the user's current location as the starting point, the floor connection entrance as the waypoint, and a point inside the navigation target area as the destination, and connect them sequentially to generate a navigation path.

[0047] As a further aspect of the present invention: the integral calculation of the three-dimensional intention probability distribution field to obtain the spatial centroid coordinates of the probability mass, and the use of the corresponding spatial centroid coordinates as the initial target point, specifically includes:

[0048] The three-dimensional intention probability distribution field is discretized into a grid to obtain the probability mass value and corresponding spatial coordinates of each grid cell;

[0049] The probability mass value of each grid cell is weighted and accumulated with its corresponding spatial coordinates to calculate the sum of the probability mass values ​​of all grid cells.

[0050] Divide the weighted cumulative result by the sum of the probability mass values ​​to obtain the weighted average position of the probability mass in three-dimensional space, and use the corresponding weighted average position as the spatial centroid coordinates.

[0051] Centered on the spatial centroid coordinates, the sum of the probability mass values ​​of all grid cells within a preset radius is calculated. When the ratio of the sum to the total probability mass values ​​is less than a preset ratio threshold, the spatial centroid coordinates are iteratively offset until the ratio is greater than or equal to the preset ratio threshold. The spatial centroid coordinates after the iterative offset are then used as the initial target point.

[0052] The beneficial effects of this invention are:

[0053] (1) This invention effectively solves the inherent semantic ambiguity problem of natural language instructions (such as "over there") by introducing an interval type II fuzzy logic system to process directional semantic words and combining eye-tracking gaze points to dynamically correct the uncertainty coverage of the fuzzy set. At the same time, it adopts evidence theory combination rules to perform conflict detection and adaptive fusion of gesture, gaze and semantic evidence, which improves the system's understanding accuracy of user navigation intentions in complex scenarios and reduces interaction errors caused by misjudgment of a single modality.

[0054] (2) This invention quantifies the natural jitter and far-end divergence characteristics of pointing actions by constructing a two-dimensional Gaussian probability field for pointing to a cone-shaped area, allowing users to identify the intended direction without deliberate and precise aiming. By combining wall boundaries and floor passages as spatial topological constraints to correct the intention probability field, the planned navigation path is ensured to conform to physical space rules, avoiding logical fallacies such as instructions passing through walls, and improving the naturalness of human-computer interaction and the usability of path guidance during virtual exhibitions. Attached Figure Description

[0055] The invention will now be further described with reference to the accompanying drawings.

[0056] Figure 1 This is a system block diagram of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Please see Figure 1 As shown, this invention is an immersive interactive system for virtual exhibition halls that integrates multiple modalities, comprising the following steps:

[0059] The data acquisition and preprocessing module is used to acquire real-time multimodal interaction data of users. The multimodal interaction data includes speech signals, hand skeleton point trajectories, head orientation, and eye gaze point sequences. The module performs timestamp alignment and noise filtering on the multimodal interaction data to obtain synchronized directional semantic words, gesture pointing vectors, head orientation angles, and gaze point coordinates.

[0060] The orientation semantic space mapping module inputs orientation semantic words into the interval type II fuzzy logic system, dynamically adjusts the principal membership function and uncertainty coverage of the fuzzy set according to the head orientation angle, and generates a three-dimensional spatial semantic membership field describing the orientation concept pointed to by the user.

[0061] The gesture pointing probability field generation module constructs a pointing cone region based on the gesture pointing vector and the amplitude of hand shaking, and obtains the probability value of each coordinate point falling into the pointing cone region through spatial probability calculation to generate the gesture pointing probability field.

[0062] The multimodal evidence fusion module integrates the three-dimensional spatial semantic membership field, gesture pointing probability field, and gaze point coordinates, and introduces the wall boundaries and floor connection channels of the virtual exhibition hall as spatial topological constraints. It uses evidence theory combination rules to calculate the joint confidence of each point in the space to satisfy the user's intent, and generates a three-dimensional intent probability distribution field.

[0063] The navigation target calculation and path planning module performs centroid defuzzification processing on the three-dimensional intention probability distribution field, combines the reachable region mask in the spatial topological constraints, extracts the probability condensation region as the navigation target region, and plans the navigation path through the floor connection passage based on the navigation target region.

[0064] In the data acquisition and preprocessing module, real-time multimodal interaction data of users is collected. This multimodal interaction data includes speech signals, hand skeleton point trajectories, head orientation, and eye gaze point sequences. The multimodal interaction data undergoes timestamp alignment and noise filtering to obtain synchronized directional semantic words, gesture pointing vectors, head orientation angles, and gaze point coordinates. Specifically, this includes:

[0065] Specifically, real-time multimodal interaction data of the user is collected through a wearable sensor array. Voice signals are collected via a microphone array located on the front of the head-mounted device, with a sampling frequency of 48 kHz. Hand skeletal point trajectories are acquired using a depth camera, which captures the 3D coordinates of 21 key points on the hand at a rate of 90 frames per second. Head orientation is provided by an inertial measurement unit, which outputs pitch, yaw, and roll angle data at a frequency of 100 times per second. Eye-tracking gaze sequences are acquired using an infrared eye-tracking camera, with a sampling frequency of 120 Hz, obtaining the user's 2D gaze coordinates in virtual space.

[0066] A unified time reference is added to the collected multimodal interaction data. Using the head-mounted device's master clock as a reference, an absolute timestamp is assigned to each frame of speech signal, hand skeleton data, head orientation data, and eye-tracking gaze data. Subsequently, nearest-neighbor interpolation is used for time registration. Based on the sampling time of the eye-tracking data, the speech frame, hand skeleton frame, and head orientation frame with the smallest absolute time difference from that time are found, and these data are combined into a synchronization data unit. Data units with a time difference exceeding 5 milliseconds are discarded.

[0067] First, the speech signal is preprocessed by dividing the 48 kHz speech data into 25-millisecond frames with a frame shift of 10 milliseconds. A filter with a pre-emphasis coefficient of 0.97 is used to boost high-frequency components. Then, short-time energy detection is used to filter out silent segments below 0.3 times the maximum energy threshold. Automatic speech recognition is performed on the remaining valid speech frames to extract directional indicator words as directional semantic words.

[0068] For the hand bone point trajectory, a five-point cubic smoothing filter method is used for processing. The specific process is as follows: For the hand bone point coordinates at the current moment, the coordinate values ​​of the two moments before and after it are taken, and a weighted average is calculated according to weighted coefficients. The weight of the current moment is 17, the weight of the adjacent moments before and after it is 12, and the weight of the moment one moment before and after it is -3. All weighted values ​​are summed and divided by 35 to obtain the filtered current coordinate value. Based on the filtered shoulder joint, elbow joint, and fingertip coordinates, a unit direction vector from the shoulder joint through the elbow joint to the fingertip is calculated as the gesture pointing vector.

[0069] For head orientation data, a moving average filtering method is used. Taking the current moment as the center, the pitch angle, yaw angle, and roll angle data at three moments before and after are taken, and the arithmetic mean is calculated for each. The average value is used as the filtered head orientation angle.

[0070] For eye-tracking fixation sequence, median filtering is used. The two-dimensional coordinates of the current fixation point and the two fixation points before and after it are taken, and the horizontal and vertical coordinates are sorted respectively. The median value of the sorted values ​​is taken as the filtered fixation point coordinates.

[0071] After the above timestamp alignment and noise filtering, the synchronized directional semantic words, gesture pointing vectors, head orientation angles, and gaze coordinates are output as input data for subsequent steps.

[0072] In the orientation semantic space mapping module, orientation semantic terms are input into the interval type II fuzzy logic system. Based on the head orientation angle, the principal membership function and uncertainty coverage of the fuzzy set are dynamically adjusted to generate a three-dimensional spatial semantic membership field describing the user's orientation concept. Specifically, this includes:

[0073] An initial interval type-II fuzzy set is constructed based on directional semantic words. Specifically, the typical directional direction corresponding to the directional semantic words is first determined according to their semantic categories. For example, the typical directional direction of the directional semantic word "over there" is defined as the range from 30 degrees to the left and 30 degrees to the right of the user's direct front. Using this typical directional direction as a benchmark, an initial interval type-II fuzzy set is constructed. This initial interval type-II fuzzy set consists of three membership function curves: the principal membership function located in the middle, the upper membership function located at the upper boundary, and the lower membership function located at the lower boundary. The principal membership function adopts a Gaussian function form, with its peak located at the center of the typical directional direction. The standard deviation of the Gaussian function is preset to 15 degrees. The upper membership function is located above the principal membership function, and the vertical distance between the two is defined as the upper boundary interval. The initial upper boundary interval is set to 5 degrees, meaning the value of the upper membership function is equal to the value of the principal membership function plus 5 percentage points. The lower membership function lies below the primary membership function, and the vertical distance between them is defined as the lower boundary interval. The initial lower boundary interval is set to 5 degrees, meaning the value of the lower membership function is equal to the value of the primary membership function minus 5 percentage points. The region between the upper and lower membership functions constitutes the uncertainty coverage area.

[0074] The initial interval type-II fuzzy set is shifted in its universe of discourse center based on the head orientation angle. The universe of discourse of the constructed initial interval type-II fuzzy set is defined as the azimuth space with the head orientation angle as the reference. Specifically, the current head orientation angle is obtained, which consists of pitch, yaw, and roll angles, where yaw represents the head's rotation angle in the horizontal direction. The peak direction of the principal membership function of the initial interval type-II fuzzy set is shifted from the default typical pointing direction to align with the yaw direction of the current head orientation angle. For example, if the current yaw angle is 30 degrees to the right, the peak direction of the principal membership function is set to 30 degrees to the right. Simultaneously, the upper and lower membership functions are shifted along with the principal membership function, maintaining their relative positions. The shifted principal membership function, upper membership function, and lower membership function together constitute the orientation-corrected interval type-II fuzzy set.

[0075] The fixation deviation is calculated by determining the spatial angle between the fixation point coordinates and the visual axis centerline corresponding to the head orientation angle. First, the visual axis centerline corresponding to the head orientation angle is determined; this centerline is a ray extending from the midpoint between the eyes along the head orientation direction. The current fixation point coordinates are obtained; these are two-dimensional screen coordinates, which are converted to three-dimensional spatial coordinates using the calibration parameters of the eye-tracking system. The shortest distance from the fixation point's three-dimensional coordinates to the visual axis centerline, and the length of the line connecting the fixation point and the midpoint between the eyes are calculated. The shortest distance is divided by the line length, and the resulting ratio is used to calculate the arctangent, yielding the spatial angle between the fixation point direction and the visual axis centerline direction. This spatial angle is used as the fixation deviation. The unit of the fixation deviation is degrees, ranging from 0 to 45 degrees.

[0076] A nonlinear scaling transformation is applied to the gaze deviation to generate an adjustment coefficient reflecting gaze uncertainty. The magnitude of the gaze deviation reflects the degree of deviation between the user's gaze point and the head orientation; a larger deviation indicates more scattered attention towards the pointed direction, i.e., higher gaze uncertainty. A nonlinear mapping relationship is used to convert the gaze deviation into an adjustment coefficient. Specifically, three threshold intervals for the gaze deviation are set: less than 5 degrees, between 5 and 15 degrees, and greater than 15 degrees. When the gaze deviation is less than 5 degrees, the adjustment coefficient is 0.2. When the gaze deviation is between 5 and 15 degrees, the difference between the gaze deviation and 5 degrees is divided by 10 to obtain an intermediate ratio value. This intermediate ratio value is then multiplied by 0.3 and added to obtain the adjustment coefficient. When the gaze deviation is greater than 15 degrees, the adjustment coefficient is 0.5. The adjustment coefficient, ranging from 0.2 to 0.5, is used to quantify the influence of gaze uncertainty on the boundary of the fuzzy set.

[0077] The interval width between the upper and lower membership functions is expanded based on the adjustment coefficient. First, the adjustment methods for the upper boundary value corresponding to the upper membership function and the lower boundary value corresponding to the lower membership function are determined. For orientation-corrected interval type II fuzzy sets, the initial value of the upper membership function is equal to the value of the primary membership function plus 5 percentage points, and the initial value of the lower membership function is equal to the value of the primary membership function minus 5 percentage points. Based on the adjustment coefficient generated in step 204, the upper and lower boundary values ​​are adjusted respectively. The specific adjustment method is as follows: multiply the adjustment coefficient by 20 degrees to obtain the expansion amount. Adjust the value of the upper membership function to be equal to the value of the primary membership function plus 5 percentage points plus the percentage points corresponding to the expansion amount. Simultaneously, adjust the value of the lower membership function to be equal to the value of the primary membership function minus 5 percentage points minus the percentage points corresponding to the expansion amount. For example, if the adjustment coefficient is 0.3, the expansion amount is 6 degrees. In this case, the value of the upper membership function is equal to the value of the primary membership function plus 11 percentage points, and the value of the lower membership function is equal to the value of the primary membership function minus 11 percentage points. In this way, the interval between the upper and lower membership functions is expanded from the original 10 percentage points to 10 percentage points plus the percentage points corresponding to twice the expansion amount.

[0078] The upper and lower membership functions, after expanding the interval width, are combined with the orientation-corrected interval type II fuzzy set to generate a gaze-weighted interval type II fuzzy set. The adjusted upper and lower membership functions are then combined with the principal membership functions in the generated orientation-corrected interval type II fuzzy set. The combination method is as follows: for any angular direction, the principal membership function value of the gaze-weighted interval type II fuzzy set in that direction remains unchanged, still being the orientation-corrected Gaussian function value. The upper membership function value of the gaze-weighted interval type II fuzzy set in that direction takes the adjusted upper boundary value, and the lower membership function value takes the adjusted lower boundary value. This forms a new interval type II fuzzy set composed of the principal membership function, the expanded upper membership function, and the expanded lower membership function—the gaze-weighted interval type II fuzzy set. The uncertainty coverage of this set reflects the impact of the gaze point deviation on the degree of semantic fuzziness in the orientation; the greater the gaze deviation, the wider the uncertainty coverage.

[0079] A gaze-weighted interval type II fuzzy set is mapped to the 3D space of the virtual exhibition hall to generate a 3D spatial semantic membership field. First, a 3D spatial coordinate system for the virtual exhibition hall is established, transforming the domain of the gaze-weighted interval type II fuzzy set from azimuth space to 3D space. Specifically, a polar coordinate system is established in 3D space, with the user's current location as the origin and the peak direction of the principal membership function in the gaze-weighted interval type II fuzzy set as the reference direction. For any point in the virtual exhibition hall, the orientation angle of that point relative to the user's current location is calculated, including the horizontal azimuth and vertical pitch angles. This orientation angle is substituted into the gaze-weighted interval type II fuzzy set, and the corresponding principal membership function value, upper membership function value, and lower membership function value are queried. Any one of these three membership function values ​​is taken as the membership value of the point, usually the principal membership function value. All spatial points in the virtual exhibition hall are traversed, and each point is assigned a membership value, representing the degree to which the point belongs to the user's indicated orientation concept. The membership values ​​of all spatial points constitute a three-dimensional spatial semantic membership field.

[0080] In the gesture pointing probability field generation module, a pointing cone region is constructed based on the gesture pointing vector and the amplitude of hand tremors. The probability value of each coordinate point falling into the pointing cone region is obtained through spatial probability calculation, thereby generating the gesture pointing probability field, which specifically includes:

[0081] The primary pointing axis is determined based on the hand gesture pointing vector. The hand gesture pointing vector is a unit direction vector extending from the shoulder joint through the elbow joint to the fingertip. The starting point of this vector is defined as the shoulder joint coordinate, and the ending point is defined as the fingertip coordinate. Using the direction of this vector's extension as the direction of the primary pointing axis, and the shoulder joint position as the starting point of the primary pointing axis, an infinitely extending straight line in space is formed along this direction; this straight line is the primary pointing axis.

[0082] The cone half-angle is calculated based on the amplitude of hand tremor. The calculation method for hand tremor amplitude is as follows: Obtain the three-dimensional coordinates of the fingertip in twenty consecutive frames of images, and calculate the standard deviation of these twenty coordinate points in space. Calculate the standard deviations of the x, y, and z coordinates separately, and take the square root of the sum of the squares of the three standard deviations as the hand tremor amplitude. Multiply the hand tremor amplitude by a scaling factor of 2 to obtain the cone half-angle in degrees. For example, if the hand tremor amplitude is 3 centimeters, the cone half-angle is 6 degrees. The value of the cone half-angle is limited to between 2 and 15 degrees; when the calculated result is less than 2 degrees, take 2 degrees; when it is greater than 15 degrees, take 15 degrees.

[0083] Construct the spatial boundary of the directional cone region using the main directional axis as the center line. Using the main directional axis as the center axis and the cone's half-angle as the semi-apex angle, construct an infinitely extending conical space. The vertex of this cone is located at the shoulder joint, its axis coincides with the main directional axis, and the angle between any point on the cone surface and the axis is equal to the cone's half-angle. The spatial region enclosed by the interior and boundary surfaces of this cone constitutes the spatial boundary of the directional cone region.

[0084] The directional cone-shaped region is divided into equidistant layers along the main directional axis. Starting from the shoulder joint, a plane perpendicular to the main directional axis is drawn at 5-centimeter intervals along the main directional axis. This plane intersects the directional cone-shaped region, resulting in a series of circular cross-sections. The first cross-section is located 5 centimeters from the shoulder joint, and subsequent cross-sections are drawn in the same manner until they exceed the boundaries of the virtual exhibition hall. Each cross-section corresponds to a spatial cross-section, and all cross-sections are parallel to each other and perpendicular to the main directional axis.

[0085] Obtain the Euclidean distance between the current spatial section and the wrist reference point as the radial distance variable. The wrist reference point is defined as the three-dimensional coordinates of the wrist joint. For the nth spatial section along the principal pointing axis, calculate the perpendicular distance between this section and the wrist reference point, i.e., the distance value corresponding to the index of the section where the foot of the perpendicular is drawn from the wrist reference point to the principal pointing axis. This distance value is the radial distance variable, in centimeters.

[0086] A Gaussian distribution standard deviation parameter is generated by nonlinearly mapping the radial distance variable. Three intervals are set for the radial distance variable: less than 50 cm, between 50 and 150 cm, and greater than 150 cm. When the radial distance variable is less than 50 cm, the standard deviation parameter is fixed at 15 cm. When the radial distance variable is between 50 and 150 cm, the difference between the radial distance variable and 50 cm is divided by 100 cm to obtain a median value. This median value is then multiplied by 35 cm and then added to obtain the standard deviation parameter. When the radial distance variable is greater than 150 cm, the standard deviation parameter is fixed at 50 cm. Through this mapping relationship, the larger the radial distance variable, the larger the standard deviation parameter value, indicating a more diffuse probability distribution at the distal section.

[0087] The probability decay gradient of the two-dimensional Gaussian distribution of the cross-section on the plane perpendicular to the principal pointing axis is determined based on the standard deviation parameter. For the current spatial cross-section, a Cartesian coordinate system is established with its plane as the two-dimensional coordinate system and the point where the principal pointing axis intersects the cross-section as the origin. On this plane, the probability decay gradient at any point is determined by both the distance of that point from the origin and the standard deviation parameter. Specifically, the farther a point is from the origin, the larger its probability decay gradient value and the faster its decay rate; the larger the standard deviation parameter, the smaller the probability decay gradient value at the same distance and the slower its decay rate.

[0088] Using the point where the main pointing axis intersects the current spatial cross-section as the distribution center, a two-dimensional Gaussian distribution for this spatial cross-section is established by combining the probability decay gradient. This two-dimensional Gaussian distribution is defined as follows: at any point on the cross-section, the probability that the point belongs to the pointing cone region is calculated using a Gaussian function based on the distance from the point to the distribution center and the standard deviation parameter. Specifically, the calculation method is as follows: first, calculate the Euclidean distance from the point to the distribution center; divide this distance by the standard deviation parameter to obtain the median ratio; square this median ratio and divide it by 2 to obtain the exponent value; use the natural constant e as the base and the negative of this exponent value as the exponent to perform a power operation to obtain the preliminary probability coefficient. Divide this preliminary probability coefficient by the product of 2π and the square of the standard deviation parameter to obtain the normalized probability density value. This probability density value is the two-dimensional Gaussian distribution value of that point on the current cross-section.

[0089] The algorithm iterates through all coordinate points in space, calculating the probability that each point belongs to a directional cone region based on the two-dimensional Gaussian distribution of its corresponding spatial section and the perpendicular distance from each point to the main directional axis. For any point in the virtual exhibition hall, the perpendicular distance from the point to the main directional axis is first calculated, and the projection point position of the point on the main directional axis is determined. The spatial section number to which the point belongs is determined based on the projection point position. If the projection point position falls between two adjacent sections, the two-dimensional Gaussian distribution corresponding to the closest section is taken. Then, based on the projection coordinates of the point on its corresponding section, the probability density value of the two-dimensional Gaussian distribution at that projection coordinate is retrieved. This probability density value is used as the probability value that the point belongs to the directional cone region. If the point is located outside the spatial boundary of the directional cone region, its probability value is directly set to 0.

[0090] The above calculation is performed on all spatial points, and each point obtains a probability value between 0 and 1. These probability values ​​are then organized according to spatial location to form a scalar field in three-dimensional space; this scalar field is the gesture pointing probability field. The gesture pointing probability field reflects the likelihood that each location in space is covered by the user's gesture pointing intention; the higher the probability value, the greater the likelihood that the area is pointed to by the gesture.

[0091] In the multimodal evidence fusion module, the three-dimensional spatial semantic membership field, gesture pointing probability field, and gaze point coordinates are integrated. The wall boundaries and floor connection passages of the virtual exhibition hall are introduced as spatial topological constraints. Evidence theory combination rules are used to calculate the joint confidence that each point in the space satisfies the user's intent, generating a three-dimensional intent probability distribution field, specifically including:

[0092] First, the three-dimensional spatial semantic membership field, the gesture pointing probability field, and the Gaussian kernel density field generated from the gaze point coordinates are used as independent evidence to construct the basic confidence assignments corresponding to each piece of evidence.

[0093] In the three-dimensional spatial semantic membership field, each spatial point has a membership value between 0 and 1, which represents the degree to which the point belongs to the directional concept indicated by the user. This membership value is directly used as the basic confidence assignment of the first piece of evidence; that is, for any spatial point, the first basic confidence assignment is equal to the three-dimensional spatial semantic membership value of that point.

[0094] In the gesture pointing probability field, each spatial point has a probability value between 0 and 1, representing the likelihood that the point is covered by the user's gesture pointing intention. This probability value is directly used as the basic confidence assignment of the second evidence; that is, for any spatial point, the second basic confidence assignment is equal to the gesture pointing probability value of that point.

[0095] The method for generating a Gaussian kernel density field from the gaze point coordinates is as follows: Obtain the gaze point coordinates at the current moment, which are three-dimensional spatial points. Construct a three-dimensional Gaussian kernel function centered on this point. The value of the Gaussian kernel function at any point in space is determined by the Euclidean distance from that point to the gaze point coordinates. Set the standard deviation of the Gaussian kernel to 20 cm. For any spatial point, calculate the Euclidean distance from that point to the gaze point coordinates, divide this distance by the standard deviation to obtain the median ratio, square the median ratio and divide it by 2 to obtain the exponent value, use the natural constant e as the base and the negative of the exponent value as the exponent to perform a power operation to obtain the preliminary density value. Divide this preliminary density value by the product of 2π raised to the power of 1.5 and the cube of the standard deviation to obtain the normalized density value. This density value is the Gaussian kernel density value for that point. Traverse all spatial points to obtain the Gaussian kernel density field. Use the density values ​​of each point in the Gaussian kernel density field as the basic confidence level for third evidence. Since the density value may be greater than 1, normalization is required: find the maximum and minimum density values ​​in the entire Gaussian kernel density field, and for each point, subtract the minimum density value from its density value and divide by the difference between the maximum and minimum density values ​​to obtain a normalized value between 0 and 1. This normalized value is then used as the third basic confidence level for assignment.

[0096] Thus, three basic confidence assignments have been obtained, denoted as the first basic confidence assignment, the second basic confidence assignment, and the third basic confidence assignment, respectively. Each assignment is a scalar field in three-dimensional space with a value range of 0 to 1.

[0097] Next, orthogonal sum rules are used to pairwise combine multiple basic confidence assignments, progressively synthesizing a joint confidence score that reflects multimodal consistency. The specific combination process is as follows:

[0098] Two basic confidence assignments are selected as the current pair. For example, the first and second basic confidence assignments are selected as the current pair. The conflict coefficient between the current pairs is calculated. The conflict coefficient reflects the degree of contradiction between the two pieces of evidence. It is calculated as follows: traverse all spatial points, summing the products of the two basic confidence assignments at each point; then traverse all spatial points again, summing the product of the two basic confidence assignments at each point with the difference between 1 and the product; divide the first sum by the second sum to obtain the conflict coefficient. This can be expressed mathematically as follows: ;in, Represents the first in space One point, and These represent the values ​​of the first and second basic confidence assignments at that point, respectively. This is the conflict coefficient, which ranges from 0 to 1.

[0099] A preset threshold of 0.3 is set. When the conflict coefficient is less than 0.3, the two pieces of evidence are considered to have high consistency, and the orthogonal sum rule is used to fuse the current combination pair. The specific operation of the orthogonal sum rule is as follows: For each spatial point, the product of the two basic confidence assignments is divided by the normalization constant to obtain the intermediate combination confidence assignment for that point. The normalization constant is equal to the sum of the products of the two basic confidence assignments at all spatial points. That is, for each point, the intermediate combination confidence assignment is equal to the product of the two basic confidence assignments at that point divided by the sum of the products at all points.

[0100] When the conflict coefficient is greater than or equal to 0.3, the two pieces of evidence are considered to have significant conflict, and a weighted average is used. The weights of the weighted average are determined based on the conflict coefficient: the weight of the first basic confidence assignment is set to 1 minus the conflict coefficient, and the weight of the second basic confidence assignment is set to the conflict coefficient. For each spatial point, the value of the first basic confidence assignment is multiplied by the first weight, and then the value of the second basic confidence assignment is multiplied by the second weight to obtain the intermediate combined confidence assignment for that point. That is, the intermediate combined confidence assignment equals (1 minus the conflict coefficient) multiplied by the first basic confidence assignment plus the conflict coefficient multiplied by the second basic confidence assignment.

[0101] At this point, the confidence level assignments for the intermediate combinations following the first combination have been obtained.

[0102] Then, the intermediate confidence level is assigned to the third basic confidence level as a new pair, and the above steps are repeated. The conflict coefficient between the new pairs is calculated again, using the same preset threshold of 0.3 for judgment: if the conflict coefficient is less than 0.3, orthogonal sum rule fusion is used; if the conflict coefficient is greater than or equal to 0.3, weighted average fusion is used. Finally, a joint confidence level reflecting the consistency of the three pieces of evidence is obtained. The joint confidence level is also a three-dimensional scalar field, with the value of each point between 0 and 1, representing the comprehensive confidence level of the user intent reflected by the three modalities at that point.

[0103] Next, an inaccessible area mask is generated based on the wall boundaries of the virtual exhibition hall. The wall boundaries of the virtual exhibition hall are provided by the 3D model and consist of a series of closed polygonal patches. The internal space enclosed by these patches, as well as the space occupied by the walls themselves, are marked as inaccessible areas. A binary mask corresponding to the 3D spatial mesh is generated. For each mesh point, if the point is located inside the wall or is surrounded by the wall, the mask value is set to 0; otherwise, the mask value is set to 1. This mask is the inaccessible area mask.

[0104] Set the joint confidence value to zero within the inaccessible region. For each spatial point, if its corresponding inaccessible region mask value is 0, change the joint confidence value of that point to 0; if the mask value is 1, retain the original value. This yields the joint confidence value for obstacle correction.

[0105] Finally, a directional gain is applied to the joint confidence score of obstacle correction based on the spatial location of the floor connection passages. Floor connection passages include stairs, elevators, corridors, etc., and their spatial location is represented by a set of three-dimensional coordinate points, including the passage entrance area and the passage interior area. For each spatial point, the distance to the nearest passage entrance and whether the point is located inside the passage are calculated. The gain coefficient is set to 1.5. If the point is located inside the passage, its joint confidence score of obstacle correction is multiplied by the gain coefficient; if the point is located in the passage entrance area (a sphere with a radius of 50 cm centered on the entrance), it is also multiplied by the gain coefficient; other areas remain unchanged. To ensure that the confidence score does not exceed 1, the result after multiplying by the gain coefficient is truncated; if it exceeds 1, it is taken as 1. This yields the final three-dimensional intent probability distribution field.

[0106] This three-dimensional intent probability distribution field integrates information from three modalities: semantics, gestures, and gaze. It also takes into account wall obstacles and passage guidance, and can accurately reflect the target area of ​​the user's intent navigation, providing reliable input for subsequent path planning.

[0107] In the navigation target calculation and path planning module, the 3D intention probability distribution field is subjected to centroid defuzzification processing. Combined with the reachable region mask in the spatial topological constraints, the probability agglomeration region is extracted as the navigation target region. Based on the navigation target region, a navigation path through the floor connection passage is planned, specifically including:

[0108] The 3D intent probability distribution field is discretized using a grid. The spatial region containing the 3D intent probability distribution field is divided into uniform grid cells, with each grid cell measuring 5 cm x 5 cm x 5 cm. Each grid cell corresponds to a spatial location, which is the coordinate of the center point of the grid cell. Each grid cell has a probability mass value, which is equal to the average of the 3D intent probability distribution values ​​of all points within the spatial region covered by that grid cell. This discretization process yields a dataset containing the probability mass values ​​of each grid cell and their corresponding 3D spatial coordinates.

[0109] Calculate the sum of the probability mass values ​​for all grid cells. Iterate through all grid cells, summing the probability mass values ​​for each cell to obtain the total probability mass value. This total probability mass value reflects the overall probability mass of the entire 3D intent probability distribution field.

[0110] The probability mass value of each grid cell is weighted and accumulated with its corresponding spatial coordinates. For each grid cell, its probability mass value is multiplied by its x-coordinate, y-coordinate, and z-coordinate values, respectively, to obtain three weighted values. The weighted values ​​in the x-direction of all grid cells are summed to obtain the weighted sum in the x-direction. Similarly, the weighted sums in the y-direction and z-direction are obtained.

[0111] Dividing the weighted sum by the total probability mass values ​​yields the weighted average position of the probability mass in three-dimensional space. Specifically, dividing the weighted sum in the x-direction by the total probability mass value gives the x-coordinate of the centroid. Dividing the weighted sum in the y-direction by the total probability mass value gives the y-coordinate of the centroid. Dividing the weighted sum in the z-direction by the total probability mass value gives the z-coordinate of the centroid. These three coordinate values ​​constitute the centroid coordinates.

[0112] Centered on the spatial centroid coordinates, sum the probability mass values ​​of all grid cells within a preset radius. The preset radius is set to 30 cm. Using the spatial centroid coordinates as the center and a radius of 30 cm as the sphere, traverse all grid cells, select those located within the sphere, and sum the probability mass values ​​of these cells to obtain the sum of the probability mass values ​​within the sphere.

[0113] Calculate the ratio of the sum of probabilistic masses within the sphere to the total sum of probabilistic masses. Compare this ratio to a preset ratio threshold. The preset ratio threshold is set to 0.6.

[0114] When this ratio is less than 0.6, it indicates that the concentration of probability mass around the current spatial centroid coordinates is insufficient, requiring iterative offsetting of the spatial centroid coordinates. The direction of the iterative offset is determined as the main direction of probability mass accumulation. The main direction of probability mass accumulation is obtained by calculating the weighted average offset of each grid cell within the sphere relative to the sphere's center. Specifically, for each grid cell within the sphere, calculate the difference vector between the cell's coordinates and the spatial centroid coordinates. Multiply this difference vector by the probability mass value of the cell to obtain a weighted offset vector. Accumulate the weighted offset vectors of all grid cells to obtain the total weighted offset vector. Divide the total weighted offset vector by the sum of the probability masses within the sphere to obtain the average offset vector. Add the average offset vector to the current spatial centroid coordinates and multiply by a step size factor (0.5) to obtain the new spatial centroid coordinates. The step size factor controls the amplitude of each offset to avoid oscillations.

[0115] Using the new spatial centroid coordinates as the center, repeatedly sum the probability mass values ​​of the grid cells within a 30 cm radius, and recalculate the ratio of this sum to the total probability mass. If this ratio is still less than 0.6, continue the above offset process. After each offset, recalculate the average offset vector, and perform the next offset based on the new spatial centroid coordinates. The iteration terminates when the ratio of the sum of probability masses within the sphere to the total probability mass is greater than or equal to 0.6, or when the number of iterations reaches the upper limit of 20. The upper limit of 20 iterations is set to prevent infinite loops.

[0116] When the iteration terminates, the final spatial centroid coordinates are used as the initial target point. This initial target point reflects the region with the highest concentration of probability quality in the three-dimensional intent probability distribution field, and this region covers at least 60% of the total probability quality, ensuring the representativeness and stability of the target point.

[0117] Using the initial target point as the center, an isosurface search is performed outwards. A probability threshold is set, which is 0.5 times the maximum probability value in the 3D intention probability distribution field. Starting from the initial target point, the search expands outwards to neighboring grid cells, marking all grid cells with probability values ​​greater than or equal to the threshold as the same connected region. The diffusion process uses a region growing algorithm: starting from the grid cell containing the initial target point, it checks the six adjacent grid cells. If an adjacent cell has a probability value greater than or equal to the threshold and has not yet been marked, it is added to the current region, and diffusion continues from there. This process is repeated until no new grid cells are added. This results in a continuous spatial region where all grid cells have probability values ​​higher than the threshold. This region is used as the first candidate agglomeration region.

[0118] Then, the next unlabeled grid cell with a probability value greater than or equal to the threshold is selected as the new seed point, and the above region growth process is repeated to obtain the second candidate agglomeration region. This process is repeated until all grid cells with a probability value greater than or equal to the threshold have been labeled. Finally, several candidate agglomeration regions are obtained, each of which is a connected spatial region composed of continuous grid cells.

[0119] Calculate the overlap between the candidate agglomeration region and the reachable region mask for each candidate agglomeration region. The reachable region mask is binary data; for each grid cell, the mask value is 1 if the cell is within the reachable region, and 0 otherwise. For each candidate agglomeration region, count the number of all grid cells within that region and the number of grid cells with a mask value of 1. Divide the number of grid cells with a mask value of 1 by the total number of grid cells in the region to obtain the overlap between the region and the reachable region mask. The overlap value ranges from 0 to 1.

[0120] Candidate clustered regions that do not intersect with the reachable area mask are discarded. Specifically, candidate clustered regions with an overlap of 0 are directly discarded. Candidate clustered regions with an overlap greater than 0 are retained as candidates. Among the retained candidate clustered regions, the region with the highest overlap is selected as the navigation target region. If multiple regions with the same overlap exist, the region closest to the initial target point is selected.

[0121] Identify the nearest floor connection entrance to the navigation target area. Each floor connection entrance is a predefined set of spatial point coordinates, with one coordinate point corresponding to each entrance. Calculate the coordinates of the geometric center point of the navigation target area, iterate through all floor connection entrance coordinates, calculate the Euclidean distance from each entrance to the geometric center point, and select the entrance with the smallest distance as the waypoint.

[0122] Starting from the user's current location, using the entrance to the connecting passage on that floor as a waypoint, and the geometric center of the target navigation area as the destination, the A* algorithm is used for pathfinding. The cost function of the A* algorithm is determined by the path length and whether obstacles are encountered, ensuring that the generated path lies entirely within the reachable area mask. The starting point, waypoints, and destination are connected sequentially to generate the navigation path. This path passes through the starting point, the entrance to the connecting passage on the floor, and finally reaches the center of the target navigation area.

[0123] The working principle of this invention is as follows: By acquiring speech signals, hand skeletal point trajectories, head orientation, and eye gaze point sequences, and performing timestamp alignment and noise filtering, synchronized directional semantic words, gesture pointing vectors, head orientation angles, and gaze point coordinates are obtained. The directional semantic words are input into an interval type-II fuzzy logic system. Based on the head orientation angle, the principal membership function and uncertainty coverage of the fuzzy set are dynamically adjusted. The interval width between the upper and lower membership functions is expanded by combining the deviation between the gaze point coordinates and the head orientation angle, generating a gaze-weighted interval type-II fuzzy set. This set is then mapped to a three-dimensional space to construct a semantic membership field. The principal pointing axis is determined based on the gesture pointing vector. A cone-shaped region is constructed by calculating the cone half-angle based on the hand tremor amplitude. After equidistant layering along the principal pointing axis, a two-dimensional Gaussian distribution with radial probability decay is established based on the distance between each cross-section and the wrist reference point. The probability value is calculated by traversing spatial points to generate the hand... The system uses a probability field to point to the target area. Semantic membership field, gesture pointing probability field, and Gaussian kernel density field generated from gaze point coordinates are used as independent evidence. Evidence theory is employed to combine these fields, and wall boundaries and floor connection passages are introduced as spatial topological constraints. A joint confidence level is generated through conflict coefficient judgment and a weighted fusion strategy. After zeroing the unreachable area mask and correcting the passage area gain, a three-dimensional intent probability distribution field is obtained. The three-dimensional intent probability distribution field is then discretized into a grid, and the spatial centroid coordinates of the probability quality are calculated. Iterative offsetting ensures that the probability quality percentage within a preset radius reaches a threshold, which is then used as the initial target point. Candidate agglomeration regions are extracted through isosurface search, and overlap with the reachable area mask is calculated to filter navigation target areas. Finally, a navigation path is planned using the user's current location as the starting point and the nearest floor connection passage entrance as the waypoint. This achieves accurate intent understanding and natural interactive guidance under deep fusion of multimodal information.

[0124] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A multimodal immersive interactive virtual exhibition hall system, characterized in that, Includes the following steps: The data acquisition and preprocessing module is used to acquire real-time multimodal interaction data of users. The multimodal interaction data includes speech signals, hand skeleton point trajectories, head orientation, and eye gaze point sequences. The module performs timestamp alignment and noise filtering on the multimodal interaction data to obtain synchronized directional semantic words, gesture pointing vectors, head orientation angles, and gaze point coordinates. The directional semantic space mapping module inputs directional semantic terms into an interval type II fuzzy logic system, dynamically adjusts the principal membership function and uncertainty coverage of the fuzzy set based on the head orientation angle, and generates a three-dimensional spatial semantic membership field describing the user's directional concept. The generation process of the three-dimensional spatial semantic membership field is as follows: Based on the directional semantic words, an initial interval type II fuzzy set is constructed. The initial interval type II fuzzy set consists of the principal membership function and the upper and lower membership functions that define the fuzzy range. Based on the head orientation angle, the initial interval type II fuzzy set is shifted to the center of discourse so that the peak direction of the principal membership function is aligned with the head orientation angle, thus obtaining the orientation-corrected interval type II fuzzy set. Based on the deviation between the gaze point coordinates and the head orientation angle, the interval width between the upper and lower membership functions is dynamically adjusted to obtain a gaze-weighted interval type II fuzzy set. The gaze-weighted interval type II fuzzy set is mapped to the three-dimensional space of the virtual exhibition hall. By calculating the membership degree of each point in the space relative to the gaze-weighted interval type II fuzzy set, a three-dimensional spatial semantic membership field is generated. The gesture pointing probability field generation module constructs a pointing cone region based on the gesture pointing vector and the amplitude of hand shaking, and obtains the probability value of each coordinate point falling into the pointing cone region through spatial probability calculation to generate the gesture pointing probability field. The multimodal evidence fusion module integrates the three-dimensional spatial semantic membership field, gesture pointing probability field, and gaze point coordinates, and introduces the wall boundaries and floor connection channels of the virtual exhibition hall as spatial topological constraints. It uses evidence theory combination rules to calculate the joint confidence of each point in the space to satisfy the user's intent, and generates a three-dimensional intent probability distribution field. The navigation target calculation and path planning module performs centroid defuzzification processing on the three-dimensional intent probability distribution field, and extracts the probability agglomeration region as the navigation target region by combining it with the reachable region mask in the spatial topological constraints. Based on the navigation target region, it plans a navigation path through the floor connection passage. The process of generating the navigation path is as follows: The spatial centroid coordinates of the probability mass are obtained by integral calculation of the three-dimensional intention probability distribution field, and the corresponding spatial centroid coordinates are used as the initial target points. Using the initial target point as the center, conduct an isosurface search outward to extract continuous spatial regions with probability values ​​higher than a preset threshold, thereby obtaining several candidate agglomeration regions. The overlap between each candidate agglomeration region and the reachable region mask is calculated. Candidate agglomeration regions that do not intersect with the reachable region mask are eliminated, and candidate agglomeration regions that overlap with the reachable region mask are retained as navigation target regions. Identify the floor connection entrance closest to the navigation target area, take the user's current location as the starting point, the floor connection entrance as the waypoint, and a point inside the navigation target area as the destination, and connect them sequentially to generate a navigation path.

2. The immersive interactive system for virtual exhibition halls integrating multiple modalities according to claim 1, characterized in that, The obtained gaze-weighted interval type II fuzzy set specifically includes: Calculate the spatial angle between the coordinates of the fixation point and the center line of the visual axis corresponding to the head orientation angle to obtain the fixation deviation. A nonlinear scaling transformation is applied to the gaze deviation to generate an adjustment coefficient that reflects gaze uncertainty; Based on the adjustment coefficient, the upper boundary value corresponding to the upper membership function is increased and the lower boundary value corresponding to the lower membership function is decreased, respectively, so as to expand the interval width between the upper membership function and the lower membership function. The upper and lower membership functions after expanding the interval width are combined with the orientation-corrected interval type II fuzzy set to generate a gaze-weighted interval type II fuzzy set.

3. The immersive interactive system for virtual exhibition halls integrating multiple modalities according to claim 1, characterized in that, The generated gesture pointing probability field specifically includes: The main pointing axis is determined based on the hand pointing vector, the cone half angle is calculated based on the hand shaking amplitude, and the spatial boundary of the pointing cone region is constructed with the main pointing axis as the center line. The directional cone region is divided into equal-distance layers along the main directional axis to obtain multiple spatial cross sections perpendicular to the main directional axis; For each spatial section, a radial probability attenuation parameter is set according to the distance between the corresponding spatial section and the wrist reference point to establish a two-dimensional Gaussian distribution of the section. Traverse all coordinate points in space, and calculate the probability value of each coordinate point belonging to the pointing cone region based on the two-dimensional Gaussian distribution of the spatial section to which each coordinate point belongs and the vertical distance from each coordinate point to the main pointing axis, and generate a gesture pointing probability field.

4. The immersive interactive system for virtual exhibition halls integrating multiple modalities according to claim 3, characterized in that, The establishment of the two-dimensional Gaussian distribution of the cross-section specifically includes: Obtain the Euclidean distance between the current spatial section and the wrist reference point, and use the corresponding Euclidean distance as the radial distance variable; A nonlinear mapping is performed on the radial distance variable to generate a Gaussian distribution standard deviation parameter. The larger the radial distance variable, the larger the standard deviation parameter value, so as to reflect the enhanced probability diffusion characteristics of the far-end section. Based on the standard deviation parameter, determine the probability decay gradient of the two-dimensional Gaussian distribution of the cross section on the plane perpendicular to the principal pointing axis; Using the point where the main pointing axis intersects the current spatial section as the distribution center, and combining the probability decay gradient, a two-dimensional Gaussian distribution corresponding to the spatial section is established.

5. The immersive interactive system for virtual exhibition halls integrating multiple modalities according to claim 1, characterized in that, The generation of the three-dimensional intent probability distribution field specifically includes: The three-dimensional spatial semantic membership field, the gesture pointing probability field, and the Gaussian kernel density field generated from the gaze point coordinates are used as independent evidence to construct the basic confidence assignments for each piece of evidence. Orthogonal sum rules are used to combine multiple basic confidence scores in pairs to gradually synthesize a joint confidence score that reflects multimodal consistency. An inaccessible area mask is generated based on the wall boundaries of the virtual exhibition hall. The joint confidence value in the inaccessible area is set to zero to obtain the joint confidence value of obstacle correction. Based on the spatial location of the floor connection passage, a directional gain is applied to the joint confidence of obstacle correction, which increases the confidence value of the passage entrance area and the internal area of ​​the passage, generating a three-dimensional intent probability distribution field.

6. The immersive interactive system for virtual exhibition halls integrating multiple modalities according to claim 5, characterized in that, The method of pairwise combining multiple basic confidence level assignments using orthogonal sum rules to progressively synthesize a joint confidence level reflecting multimodal consistency specifically includes: Select two from multiple basic confidence assignments as the current combination pair and calculate the conflict coefficient between the current combination pair; When the conflict coefficient is less than the preset threshold, the current combination pair is fused using orthogonal sum rules to generate intermediate combination confidence assignments; When the conflict coefficient is greater than or equal to a preset threshold, the current combination pair is subjected to a weighted average process, and the conflict coefficient is used as the basis for weight adjustment to generate an intermediate combination confidence assignment for conflict resolution. The intermediate combined confidence level assignments and the remaining basic confidence level assignments are used as new combination pairs in turn. The above steps are repeated until all basic confidence level assignments are combined to obtain the joint confidence level.

7. The immersive interactive system for virtual exhibition halls integrating multiple modalities according to claim 6, characterized in that, The step of integrating the three-dimensional intention probability distribution field to obtain the spatial centroid coordinates of the probability mass, and using the corresponding spatial centroid coordinates as the initial target point, specifically includes: The three-dimensional intention probability distribution field is discretized into a grid to obtain the probability mass value and corresponding spatial coordinates of each grid cell; The probability mass value of each grid cell is weighted and accumulated with its corresponding spatial coordinates to calculate the sum of the probability mass values ​​of all grid cells. Divide the weighted cumulative result by the sum of the probability mass values ​​to obtain the weighted average position of the probability mass in three-dimensional space, and use the corresponding weighted average position as the spatial centroid coordinates. Centered on the spatial centroid coordinates, the sum of the probability mass values ​​of all grid cells within a preset radius is calculated. When the ratio of the sum to the total probability mass values ​​is less than a preset ratio threshold, the spatial centroid coordinates are iteratively offset until the ratio is greater than or equal to the preset ratio threshold. The spatial centroid coordinates after the iterative offset are then used as the initial target point.

Images