Immersion interaction service method based on multi-agent matrix

Abstract

This invention discloses an immersive interactive service method based on a multi-agent matrix, belonging to the field of digital media interaction technology. The method includes: collecting data from multiple sensing units in physical space to construct a volumetric field matrix, recording the spatial coordinates of each voxel and its corresponding spatial attribute parameters; capturing user posture sequences and parsing the user's pointing vector to identify the user's focus direction in physical space; activating a target volumetric submatrix based on the positional relationship between the spatial attribute parameters in the volumetric field matrix and the user's pointing vector; mapping the interactive response interface in physical space based on the spatial coordinates of the target volumetric submatrix; segmenting the service content into a content voxel stream by voxelization according to the volumetric distribution features; generating an interactive response sequence by arranging the interface topology; and synchronously adjusting the presentation progress by collecting continuous changes in user posture values ​​to generate an immersive interactive service stream. This invention achieves spatially perception-driven adaptive immersive interaction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital media interaction, in particular to an immersive interaction service method based on multi-agent matrix. BACKGROUND

[0002] The immersive interaction service presents interactive digital content around the user by perceiving the physical space and the user's posture. The prior art usually constructs an interactive interface with fixed display devices or pre-calibrated projection areas, and uses cameras, inertial sensors and other devices to collect the user's posture, and maps gestures, torso orientation and other inputs into control instructions for the display content. Physical space perception is mainly used for obstacle avoidance and positioning, and no spatial data structure is formed for interactive content calling.

[0003] The fixed interactive interface and the preset projection area cannot adjust the generation position of the interface in the physical space according to the user's real-time attention direction, causing the interactive response area to be inconsistent with the space area actually pointed by the user. When the target space area is determined by the user's attention direction, the prior art lacks means to dynamically extract the corresponding sub-matrix in the volumetric field matrix and map the interactive response interface accordingly, making it difficult for the interface to cover the space area of interest to the user. The service content is usually segmented and rendered according to the time axis or screen coordinates during presentation, without reference to the volumetric distribution characteristics of the voxel units in the physical space. The service content is still segmented according to a uniform grid or time segment, without differential mapping in combination with the volume and density of the spatial grid units at different positions in the target space area, and the content presentation is disconnected from the volumetric distribution of the physical space. The arrangement of the interactive response sequence is also not in the order of the spatial grid units covered by the interactive response interface, making the distribution of digital content in space inconsistent with the spatial volumetric characteristics of the user's attention. How to make the interactive response interface locate in the volumetric field matrix according to the user's pointing vector and activate the corresponding space area, and how to voxelize the arrangement of the service content to be displayed according to the volumetric distribution characteristics of the space area and the interface topology order, are still problems to be solved in immersive interaction services. SUMMARY

[0004] In view of the problems in the prior art that the interactive response interface cannot be dynamically generated according to the user's attention direction, and the service content organization does not match the spatial volumetric distribution, an immersive interaction service method based on multi-agent matrix is provided. The method constructs a volumetric field matrix by collecting multiple perception units in the physical space, analyzes the user's pointing vector and activates the target volumetric sub-matrix, maps the interactive response interface based on the target volumetric sub-matrix, voxelizes the segmentation of the service content and arranges it in the order of the interface topology of the interactive response interface, adjusts the presentation progress of the interactive response sequence according to the continuous change value of the user's posture, dynamically matches the interactive response interface with the user's attention direction, and makes the service content presentation consistent with the spatial volumetric distribution.

[0005] To achieve the above object, the present application provides the following technical scheme: the present application provides an immersive interactive service method based on multi-agent matrix, a volume field matrix is constructed by collecting multiple sensing units in a physical space, the volume field matrix records the spatial coordinates and corresponding spatial attribute parameters of each voxel in the physical space; a user posture sequence in the physical space is captured, and a user pointing vector is analyzed from the user posture sequence, the user pointing vector is used to identify the attention direction of the user in the physical space; according to the spatial position relationship between the spatial attribute parameters in the volume field matrix and the user pointing vector, a target volume sub-matrix corresponding to the user pointing vector is activated in the volume field matrix; based on the spatial coordinates in the target volume sub-matrix, an interactive response interface is mapped in the physical space, the interactive response interface covers the physical space region corresponding to the target volume sub-matrix; the service content to be displayed is voxelized and segmented according to the volume distribution characteristics of the target volume sub-matrix, and a service content voxel stream is obtained; the service content voxel stream is arranged according to the interface topology order of the interactive response interface, and an interactive response sequence is generated; the continuous change value of the user posture sequence is collected, and the presentation progress of the interactive response sequence is synchronously adjusted according to the continuous change value, and an immersive interactive service stream is obtained. This method combines the volume information of the physical space with the user pointing behavior, organizes and presents the service content voxelized according to the user attention direction, improves the corresponding accuracy between the interactive response and the spatial position, and enhances the immersive service experience.

[0006] As one of the technical schemes of the present application, when constructing the volume field matrix, the boundary contour point cloud of the physical space is obtained, and the boundary contour point cloud is divided into a plurality of spatial grid units; the center point coordinates and the occupancy state of each spatial grid unit are extracted, and the center point coordinates and the occupancy state are taken as the spatial attribute parameters of the spatial grid unit; the spatial attribute parameters of all spatial grid units are combined according to the arrangement order of the spatial grid units in the physical space, and the volume field matrix is generated. Preferably, when the boundary contour point cloud is divided into a plurality of spatial grid units, the division granularity of the grid unit is adaptively adjusted according to the geometric complexity of the physical space, a finer division granularity is used in the region with a geometric contour curvature greater than a preset threshold, and a coarser division granularity is used in the region with a geometric contour curvature less than a preset threshold, so as to reduce data redundancy while maintaining spatial expression accuracy.

[0007] As a technical solution of the present application, when the user pointing vector is analyzed, the shoulder coordinates and wrist coordinates of the user are extracted from the user posture sequence, and the line direction between the shoulder coordinates and the wrist coordinates is calculated; the face orientation vector of the user is extracted from the user posture sequence, the line direction is weighted and synthesized with the face orientation vector to obtain a synthesized pointing direction; the synthesized pointing direction is mapped into the matrix coordinate system of the volume field matrix to generate the user pointing vector. By fusing the limb pointing and the face orientation, the actual attention direction of the user can be more accurately reflected, and the influence of pointing deviation on the activation of the interactive response region is reduced.

[0008] As a technical solution of the present application, when the target volume sub-matrix is activated, the starting point coordinates and direction vector of the user pointing vector are calculated, and ray step iteration is performed in the volume field matrix along the direction vector; the cell index of each space grid cell passed in the ray step iteration process is recorded, and the cell index is stored in the activation index list; the spatial attribute parameters corresponding to each cell index in the activation index list are extracted from the volume field matrix, and all the extracted spatial attribute parameters are combined into the target volume sub-matrix. This way can quickly determine the space region that needs to be activated along the user attention direction, reduce the calculation amount of space iteration, and improve the interactive response speed.

[0009] As a technical solution of the present application, when the interactive response interface is mapped, the space coordinates and occupancy state of each space grid cell in the target volume sub-matrix are extracted, a plurality of space grid cells with empty occupancy state are screened out to obtain an empty cell set; the space coordinates in the empty cell set are surface fitted to generate a smooth mapping surface; and the smooth mapping surface is projected into the physical space to obtain the interactive response interface. In this way, the interactive response interface can avoid the occupied area in the physical space, and improve the projectability and use safety of the interface in the real space.

[0010] As one of the technical solutions of the present application, when the service content voxel stream is generated, the original data stream of the service content to be displayed is acquired, and the original data stream is divided into a plurality of data segments in chronological order; the volume distribution characteristics of the target volume sub-matrix are extracted, the volume distribution characteristics including the cell volume and cell density of each spatial grid cell in the target volume sub-matrix; each data segment is respectively mapped into the corresponding spatial grid cell according to the corresponding relationship between the data volume of each data segment and the cell volume and cell density, so as to obtain the service content voxel stream. Preferably, when each data segment is respectively mapped into the corresponding spatial grid cell, the corresponding data compression rate is determined according to the cell density of each spatial grid cell, and the spatial grid cell with a larger cell density adopts a higher data compression rate, so as to adapt to the bearing capacity of different spatial regions and ensure the consistent presentation effect of the service content in space.

[0011] As one of the technical solutions of the present application, when the interactive response sequence is generated, the interface topology sequence of the interactive response interface is extracted, the interface topology sequence being the arrangement sequence of each spatial grid cell covered by the interactive response interface; the data segments carried by each spatial grid cell in the service content voxel stream are read in sequence according to the arrangement sequence; all the read data segments are serially spliced according to the reading sequence, so as to generate the interactive response sequence. By arranging the voxelized content according to the interface topology sequence, the service content can maintain consistent spatial logic when presented in space, and the decline of the interactive experience caused by spatial misplacement is reduced.

[0012] As one of the technical solutions of the present application, when the presentation progress of the interactive response sequence is synchronously adjusted, the torso rotation angular velocity of the user is extracted from the user posture sequence, and the torso rotation angular velocity is taken as the continuous change value; the total duration of the interactive response sequence is acquired, and the adjustment offset is determined according to the torso rotation angular velocity and the total duration; the current presentation timestamp of the interactive response sequence is moved according to the adjustment offset, so as to complete the synchronous adjustment of the presentation progress of the interactive response sequence. In this way, the user can control the advancing speed of the service content through body rotation, reduce the dependence on additional input devices, and improve the smoothness of the interactive process.

[0013] In the above technical solutions, the present application provides the following technical effects and advantages: The user pointing vector is analyzed through a user posture sequence, ray step iteration is performed in the volume field matrix according to the starting point coordinates and direction vector of the user pointing vector, the cell index of each space grid cell passed by the ray is recorded and stored in an active index list, the spatial attribute parameters corresponding to each cell index in the active index list are extracted from the volume field matrix, and are combined into a target volume sub-matrix; based on the spatial coordinates and occupancy state of each space grid cell in the target volume sub-matrix, the space grid cells with an empty occupancy state are screened to obtain an empty cell set, the spatial coordinates in the empty cell set are subjected to surface fitting, a smooth mapping surface is generated, and the smooth mapping surface is projected into the physical space to obtain an interactive response interface. The generation position of the interactive response interface is determined by the target volume sub-matrix corresponding to the user pointing vector, and is no longer fixed in the preset display area. The interactive interface covering the corresponding space region can be dynamically mapped in the physical space according to the real-time attention direction of the user, so that the interactive response interface and the attention direction of the user are kept consistent in the spatial position, and the positional deviation between the interactive interface and the actual attention region of the user is reduced.

[0014] The original data stream of the service content to be displayed is divided into a plurality of data segments in chronological order, the cell volume and cell density of each space grid cell in the target volume sub-matrix are extracted, each data segment is respectively mapped to the corresponding space grid cell according to the corresponding relationship between the data volume of each data segment and the cell volume and cell density, and a service content voxel stream is formed; the arrangement order of each space grid cell covered by the interactive response interface is extracted as an interface topology order, the data segments carried by each space grid cell in the service content voxel stream are read in sequence according to the arrangement order, all the read data segments are serially spliced according to the reading order, and an interactive response sequence is generated. The segmentation granularity of the service content voxel stream is adapted to the volume and density distribution of the space grid cells in the target space region, and the arrangement order of the interactive response sequence is consistent with the spatial topology order of the interactive response interface, so that the presentation mode of the service content in the physical space conforms to the spatial volume characteristics of the user attention region, and the spatial consistency between the digital content and the user attention space is maintained. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0016] Figure 1 is an immersive interactive service method flowchart based on a multi-agent matrix; Figure 2 is a user pointing vector generation method flowchart; Figure 3 is a service content voxel flow generation and interactive response method flow chart; Figure 4 is a change curve of a line direction, a face orientation vector and a synthetic pointing direction in a user posture sequence; Figure 5 is a space grid unit volume, density and data compression rate distribution curve in a target volume sub-matrix. DETAILED DESCRIPTION

[0017] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0018] Referring to Figure 1 The present application provides an immersive interactive service method based on a multi-agent matrix, which comprises the following steps: collecting a plurality of sensing units in a physical space to construct a volume field matrix, wherein the volume field matrix records the spatial coordinates and corresponding spatial attribute parameters of each voxel in the physical space; capturing a user posture sequence in the physical space, and analyzing a user pointing vector from the user posture sequence, wherein the user pointing vector is used to identify the attention direction of the user in the physical space; activating a target volume sub-matrix corresponding to the user pointing vector in the volume field matrix according to the spatial position relationship between the spatial attribute parameters in the volume field matrix and the user pointing vector; mapping an interactive response interface in the physical space based on the spatial coordinates in the target volume sub-matrix, wherein the interactive response interface covers the physical space region corresponding to the target volume sub-matrix; voxelizing and segmenting service content to be displayed according to the volume distribution characteristics of the target volume sub-matrix to obtain a service content voxel flow; arranging the service content voxel flow according to the interface topology order of the interactive response interface to generate an interactive response sequence; collecting the continuous change value of the user posture sequence, and synchronously adjusting the presentation progress of the interactive response sequence according to the continuous change value to obtain an immersive interactive service flow.

[0019] In a specific implementation, the boundary contour point cloud of the physical space is obtained by: collecting a depth image and a color image of the physical space through a plurality of sensing units, generating a three-dimensional point cloud of the physical space from the depth image, and extracting points located on the boundary of the physical space from the three-dimensional point cloud to form the boundary contour point cloud.

[0020] When dividing the boundary contour point cloud into several spatial grid cells, the geometric contour curvature of each boundary contour point in the boundary contour point cloud is calculated. The geometric contour curvature is calculated using the following formula:

[0021] in, This represents the geometric contour curvature between two adjacent boundary contour points in the boundary contour point cloud. The index number in the boundary contour point cloud is The normal vector of the boundary contour points; The index number in the boundary contour point cloud is The normal vector of the boundary contour points; The index number in the boundary contour point cloud is The spatial coordinates of the boundary contour points; The index number in the boundary contour point cloud is The spatial coordinates of the boundary contour points; It is expressed as the average point spacing of the boundary contour point cloud, with the dimension of length (L); and This represents the index number of two adjacent boundary contour points in the boundary contour point cloud, with the index number starting from 1 and incrementing. The normal vector is obtained by performing a planar fit on the local neighborhood of each boundary contour point, with the local neighborhood radius being 5 times the average point spacing of the boundary contour point cloud.

[0022] After calculating the geometric contour curvature, it is compared with a preset threshold. The preset threshold is determined based on the geometric contour curvature distribution of all boundary contour points in the boundary contour point cloud, and the median of the geometric contour curvature distribution is taken as the preset threshold. The preset threshold and the geometric contour curvature use the same dimensional processing method to maintain consistency in the comparison benchmark. For the region where the geometric contour curvature is greater than the preset threshold, a finer division granularity is used, setting the side length of the spatial grid cell to twice the average point spacing of the boundary contour point cloud; for the region where the geometric contour curvature is less than the preset threshold, a coarser division granularity is used, setting the side length of the spatial grid cell to six times the average point spacing of the boundary contour point cloud. The boundary contour point cloud is divided using the above method, resulting in several spatial grid cells.

[0023] When extracting the center point coordinates and occupancy status of each spatial grid cell, the geometric center position of the spatial grid cell in the physical space coordinate system is calculated for each spatial grid cell, and the three-dimensional coordinates of the geometric center position are used as the center point coordinates. It is then determined whether the spatial grid cell contains at least one boundary contour point from the boundary contour point cloud. If the spatial grid cell contains at least one boundary contour point, the occupancy status is marked as occupied; if the spatial grid cell does not contain any boundary contour points, the occupancy status is marked as empty.

[0024] The center point coordinates and occupancy status of all spatial grid cells are used as spatial attribute parameters for each spatial grid cell. These parameters are then combined into a volumetric field matrix according to the arrangement order of the spatial grid cells in the physical space. The arrangement order follows the ascending index of the spatial grid cells along the X, Y, and Z axes in the physical space coordinate system. Each element in the volumetric field matrix corresponds to a spatial grid cell, and the matrix element stores the center point coordinates and occupancy status of that spatial grid cell.

[0025] In some embodiments, the average point spacing of the boundary contour point cloud is obtained by calculating the average distance between each boundary contour point in the boundary contour point cloud and its nearest neighbor boundary contour point.

[0026] It is understandable that for regions in physical space where the curvature of the boundary contour changes drastically, using a finer granularity of division can preserve the detailed features of the boundary contour; for regions where the curvature of the boundary contour is gentle, using a coarser granularity of division can reduce the number of spatial grid cells and reduce the data size of the volume field matrix.

[0027] In specific implementation, please refer to Figure 2 The capture of user pose sequences in physical space is accomplished by acquiring the 3D coordinate sequence of user skeletal key points using a depth sensor. Each frame of the user pose sequence contains the 3D coordinates of multiple user skeletal key points, including the left shoulder key point, right shoulder key point, left wrist key point, right wrist key point, nose tip key point, left inner corner of eye key point, and right inner corner of eye key point. The 3D coordinates of the user skeletal key points are established in the coordinate system of the user pose sequence acquisition device.

[0028] When extracting shoulder coordinates from a user's pose sequence, the 3D coordinates of the left and right shoulder keypoints are read from each frame of the sequence. The midpoint between these coordinates is calculated and used as the shoulder coordinate. Similarly, when extracting wrist coordinates from a user's pose sequence, the 3D coordinates of the left and right wrist keypoints are read from each frame. The midpoint between these coordinates is calculated and used as the wrist coordinate.

[0029] When calculating the direction of the line connecting shoulder and wrist coordinates, the shoulder coordinates are subtracted from the wrist coordinates to obtain the line vector. This line vector is then normalized to obtain the line direction. The line direction is a unit vector, representing the direction from the shoulder coordinates to the wrist coordinates. Each frame in the user pose sequence corresponds to one line direction.

[0030] When extracting the user's facial orientation vector from the user's pose sequence, the 3D coordinates of the nose tip keypoint, the left inner corner keypoint, and the right inner corner keypoint are read from each frame of the user's pose sequence. The midpoint between the 3D coordinates of the left and right inner corner keypoints is calculated to obtain the eye center coordinates. The eye center coordinates are then subtracted from the nose tip keypoint coordinates to obtain the facial orientation vector. The facial orientation vector is then normalized to obtain a unitized facial orientation vector.

[0031] The direction of the connecting line is weighted and synthesized with the facial orientation vector to obtain the synthesized pointing direction. The synthesized pointing direction is calculated using the following formula:

[0032] in, Indicates the direction of composition; The weighting coefficients represent the direction of the connection. Set to 0.4; Indicates the direction of the connection; This represents the weighting coefficients of the facial orientation vector. Set to 0.6; This represents the vector indicating the direction of the face. and The sum is 1. The reason for setting it to 0.4 is that the direction of the arm connection reflects the hand's pointing intention, but there is inertial shaking and limb swinging during arm movement, and the stability of the arm connection direction in the user's posture sequence is lower than that of the face orientation vector. The rationale for setting it to 0.6 is that the facial orientation vector is generated from stable facial key points on the head, and the facial orientation vector directly reflects the user's visual focus direction in physical space. The stability of the facial orientation vector in the user's posture sequence is higher than that of the arm connection direction.

[0033] The process of mapping the synthesized pointing direction to the matrix coordinate system of the volume field matrix to generate the user pointing vector includes: obtaining the rotation matrix and translation vector between the user posture sequence acquisition device coordinate system and the matrix coordinate system of the volume field matrix; rotating the synthesized pointing direction using the rotation matrix to obtain the synthesized pointing direction in the matrix coordinate system; rotating and translating the shoulder coordinates using the rotation matrix and translation vector to obtain the shoulder coordinates in the matrix coordinate system; and generating the user pointing vector in the matrix coordinate system, starting from the shoulder coordinates and using the synthesized pointing direction as the direction. The user pointing vector is used to identify the user's direction of interest in physical space.

[0034] See Figure 4 In the graph, the horizontal axis represents the frame number of the captured data, ranging from 0 to 600 frames, and the vertical axis represents the angle value in degrees, ranging from approximately -35 degrees to 35 degrees. The graph shows the angular trends of the three curves—the connection direction, the face orientation vector, and the composite pointing direction—as the frame number changes.

[0035] The curve of the connecting line direction exhibits significant fluctuations, ranging from approximately -33 degrees to 33 degrees, and is relatively coarse, indicating substantial jitter and noise in the connecting line direction within the user's posture sequence. The curve of the facial orientation vector changes more smoothly, ranging from approximately -20 degrees to 20 degrees, with significantly less fluctuation than the connecting line direction, indicating higher stability in facial orientation. The synthesized pointing direction curve lies between the connecting line direction and the facial orientation vector, showing a smoother overall trend, with its maximum and minimum values ​​falling between the two, consistent with the weighted coefficients. , The settings.

[0036] From frame 0 to frame 100, all three curves show an upward trend, with the angle gradually increasing from 0 degrees to the range of approximately 20 to 30 degrees. From frame 100 to frame 250, the curves decrease overall, with the maximum amplitude of the connecting line direction decreasing to approximately -30 degrees. The fluctuations in the facial orientation vector and the direction of synthesis are relatively small, with the lowest being approximately -20 to -25 degrees. From frame 250 to frame 400, the angle increases again, reaching a peak similar to the initial stage. From frame 400 to frame 600, the angle generally tends to decrease.

[0037] In practice, each spatial grid cell in the volumetric field matrix has a unique cell index, which increments from 1 according to the order of the spatial grid cells in the volumetric field matrix. When activating the target volumetric submatrix based on the spatial attribute parameters in the volumetric field matrix and the spatial positional relationship with the user-pointing vector, the starting point coordinates and direction vector are extracted from the user-pointing vector. The starting point coordinates are the shoulder coordinates in the matrix coordinate system, and the direction vector is the composite pointing direction in the matrix coordinate system; the direction vector is a unit vector.

[0038] The step size for ray traversal is set to 0.5 times the side length of the smallest spatial grid cell in the volume field matrix. The side length of the smallest spatial grid cell is obtained from the partitioning granularity used when constructing the volume field matrix. The partitioning granularity is adaptively adjusted according to the physical spatial geometric complexity. The side length of the spatial grid cell in fine-grained regions is twice the average point spacing of the boundary contour point cloud, and the side length of the spatial grid cell in coarse-grained regions is six times the average point spacing of the boundary contour point cloud. When performing ray traversal along the direction vector in the volume field matrix, the starting coordinates are used as the ray's starting point, and the direction vector is used as the ray's forward direction, advancing sequentially with the set step size. After each step, the spatial grid cell where the advanced position is located is obtained, and it is determined which spatial grid cell the advanced position falls into. The cell index of the spatial grid cell where the advanced position is located is obtained. The cell index is compared with the cell index already recorded in the active index list. If the active index list does not contain the same cell index, the cell index is stored in the active index list; if the active index list already contains the same cell index, the process continues without recording it again. The ray stepping traversal continues along the direction vector until the position after advancement exceeds the boundary of the volume field matrix.

[0039] After obtaining the list of active indices, the spatial attribute parameters corresponding to each cell index in the list are extracted from the volumetric field matrix. These spatial attribute parameters include the coordinates of the center point of the spatial grid cell and its occupancy status. All extracted spatial attribute parameters are then combined into the target volumetric submatrix according to the storage order of the cell indices in the list.

[0040] When creating an interactive response interface based on spatial coordinate mapping in the target volume submatrix, extract the spatial coordinates and occupancy status of each spatial grid cell in the target volume submatrix. Each element in the target volume submatrix contains the center point coordinates and occupancy status of a spatial grid cell. Traverse all elements in the target volume submatrix, filter out spatial grid cells with empty occupancy status, and add the center point coordinates of the empty spatial grid cells to the set of empty cells.

[0041] Surface fitting is performed on the spatial coordinates of the empty cell set. Before fitting, the spatial coordinates of all spatial grid cells in the empty cell set are normalized to make them dimensionless, thus eliminating the influence of dimensions on the fitting calculation. The formula for surface fitting is:

[0042] in, Represents the implicit function value of a smoothed mapped surface, when Time, spatial coordinates Located on a smooth mapped surface; Represents the spatial coordinates of spatial grid cells in the set of empty cells; Indicates the number of spatial grid cells in the set of empty cells; Indicates the 1st vacancy in the set of empty cells The radial basis function weight coefficients corresponding to each spatial grid cell; Represents the radial basis functions of a thin plate spline. ,in Radial distance; Indicates the 1st vacancy in the set of empty cells Spatial coordinates of each spatial grid cell; Represents the coefficients of the constant term in surface fitting; Indicating surface fitting The coefficient of the first term of the coordinate; Indicating surface fitting The coefficient of the first term of the coordinate; Indicating surface fitting The coefficient of the first-order term of the coordinate system.

[0043] During the surface fitting process, the spatial coordinates of each spatial grid cell in the set of empty cells are substituted into the formula, and then... At the same time, regularization constraints are added. and A system of linear equations is established using the spatial coordinates of all spatial grid cells in the set of empty cells. Solving the system of linear equations yields... The specific value.

[0044] Solving for the implicit function yields the solution. Then, within the spatial range where the set of empty cells is located, sample spatial coordinate points at a set resolution, and substitute the sampled spatial coordinate points into the implicit function. Extract to satisfy Generate a smooth mapped surface from all sampled spatial coordinate points.

[0045] The smoothed mapped surface is projected into physical space to obtain the interactive response interface. The projection process is as follows: each surface point on the smoothed mapped surface is projected into physical space along the normal direction of the physical space reference plane, forming an interactive response interface that covers the physical space region corresponding to the target volume submatrix.

[0046] In some embodiments, the spatial grid cells in the set of vacant cells are spatial grid cells whose center point coordinates and occupancy status are both empty. An empty occupancy status means that the spatial grid cell does not contain any boundary contour points.

[0047] It can be understood that the interactive response interface covers the physical space region corresponding to the target volume submatrix, and the interactive response interface is located in the empty region pointed to by the user's pointing vector in the physical space.

[0048] In specific implementation, please refer toFigure 3 The process involves acquiring the raw data stream of the service content to be displayed. This raw data stream can include video, audio, image, or 3D model data streams. When dividing the raw data stream into segments in chronological order, the total duration of the raw data stream is used as the dividing criterion. A fixed slice length is set, and the data is divided along the timeline of the raw data stream, starting from the beginning and continuing for each fixed slice length until the end of the raw data stream. Each data segment corresponds to a time interval, and the amount of data in each segment is determined based on the fixed slice length and the bitrate of the raw data stream.

[0049] When extracting the capacity distribution features of the target ductile submatrix, the capacity distribution features include the cell volume and cell density of each spatial grid cell in the target ductile submatrix. For each spatial grid cell in the target ductile submatrix, the cell volume is obtained by extracting the three side lengths of the spatial grid cell in the physical space coordinate system and multiplying the three side lengths. The three side lengths are the lengths of the spatial grid cell along the X-axis, Y-axis, and Z-axis of the physical space coordinate system, respectively. The cell density is obtained by counting the number of boundary contour points contained within the spatial grid cell and dividing the number of boundary contour points by the cell volume. The boundary contour points are obtained from the boundary contour point cloud of physical space collected when constructing the ductile field matrix, and the number of boundary contour points contained within each spatial grid cell is determined when the spatial grid cell is divided.

[0050] Based on the correspondence between the data volume of each data segment and the cell volume and density, when mapping each data segment to its corresponding spatial grid cell, the data volume of the data segment is directly proportional to the cell volume of the spatial grid cell; the larger the cell volume of the spatial grid cell, the larger the data segment data volume is allocated to it. When allocating the data segment data volume, first calculate the cell volume ratio of each spatial grid cell based on the cell volumes of all spatial grid cells in the target volume submatrix; then multiply the total data volume of each data segment by the cell volume ratio of the corresponding spatial grid cell to obtain the data segment data volume allocated to each spatial grid cell.

[0051] When mapping each data segment to its corresponding spatial grid cell, the data compression ratio is determined based on the cell density of each spatial grid cell. The data compression ratio is calculated using the following formula:

[0052] in, Describes the nth element in the target volume submatrix Data compression rate corresponding to each spatial grid cell The value range is from 0 to 1; Describes the nth element in the target volume submatrix Cell density of a spatial grid cell; This represents the maximum cell density of all spatial grid cells in the target volume submatrix. When the... When the cell density of a spatial grid cell equals the maximum value, Equals 1; when the first The smaller the cell density of each spatial grid cell, The closer it is to 0. When the cell density of a spatial grid cell is high, that spatial grid cell uses a higher data compression ratio; When the cell density of a spatial grid cell is low, the spatial grid cell uses a lower data compression rate.

[0053] When compressing the data segment allocated to each spatial grid cell according to the data compression ratio, the data segment is input into the compression encoder. The compression encoder compresses the data according to the data compression ratio corresponding to that spatial grid cell and outputs the compressed data segment. The compressed data segment is stored in the corresponding spatial grid cell, and each spatial grid cell and the compressed data segment form a service content voxel. The service content voxels corresponding to all spatial grid cells in the target volume submatrix are combined according to the arrangement order of the spatial grid cells in the target volume submatrix to obtain the service content voxel stream.

[0054] When orchestrating the service content voxel stream according to the interface topology order of the interactive response interface, the interface topology order of the interactive response interface is extracted. The interface topology order is the arrangement order of the various spatial grid cells covered by the interactive response interface. To extract the interface topology order, the cell indices of all spatial grid cells covered by the interactive response interface are obtained. The spatial grid cells covered by the interactive response interface are arranged from smallest to largest along the first principal direction coordinate of the interactive response interface. Among the spatial grid cells with the same first principal direction coordinate, they are arranged from smallest to largest according to the second principal direction coordinate, thus obtaining the interface topology order. The first principal direction is the tangent direction with the largest curvature change on the interactive response interface, and the second principal direction is the tangent direction on the interactive response interface perpendicular to the first principal direction.

[0055] When reading data fragments carried by each spatial grid cell in the service content voxel stream in the order of arrangement, the order of arrangement of spatial grid cells in the interface topology is used as the reading order. The service content voxel corresponding to each spatial grid cell in the service content voxel stream is accessed sequentially, and compressed data fragments are read from the service content voxel. All the read compressed data fragments are then serially concatenated according to the reading order to generate an interactive response sequence. During the serial concatenation process, adjacent compressed data fragments are linked end-to-end, maintaining the reading order to form a continuous data stream.

[0056] In some embodiments, service content voxels in the service content voxel stream are stored in spatial grid cells corresponding to the target volume submatrix, and service content voxels and spatial grid cells correspond one-to-one through cell indices. The service content voxel corresponding to the spatial grid cell covered by the interactive response interface in the service content voxel stream is found using the cell index, and the compressed data fragment carried by the service content voxel is read.

[0057] It is understandable that cell density reflects the density of boundary contour points within a spatial grid cell. The higher the cell density, the richer the geometric details at that spatial grid cell, and the higher the tolerance for data compression. Using a higher data compression rate can reduce the data size of the service content voxel stream without affecting the presentation of content in the corresponding spatial grid cell of the interactive response interface.

[0058] See Figure 5 In the figure, the horizontal axis represents the spatial grid cell number, ranging from 0 to 500, indicating the arrangement order of each spatial grid cell in the target volume submatrix. The vertical axis represents the numerical value, including cell volume, cell density, and data compression ratio, with the vertical axis scale covering a numerical range from 0 to 110.

[0059] The solid blue line in the figure represents the cell volume, which remains relatively stable between 0 and 2, exhibiting minor fluctuations. This indicates that the volume distribution of the spatial grid cells in the target volume submatrix is ​​relatively uniform, with minimal volume changes. The dashed orange line represents the cell density, which varies considerably, primarily between approximately 30 and 100, with occasional extremely low values. This reflects significant differences in the density of boundary contour points within each spatial grid cell; some cell boundaries are richly detailed and dense, while others are relatively sparse. The dotted green line represents the data compression ratio, which is generally below 0.05 and shows a relatively gradual trend with cell density.

[0060] Based on the technical content of the above embodiment, the cell volume corresponds to the physical size of the spatial grid cell, the cell density reflects the density of boundary contour points within the spatial grid cell, and the data compression rate is calculated by the ratio of the cell density to the maximum density. The large fluctuations in cell density in the figure cause corresponding changes in the data compression rate, indicating that the data compression rate is dynamically adjusted according to the cell density; higher density results in a higher compression rate, and lower density results in a lower compression rate, achieving differentiated compression of voxel data of service content. Simultaneously, the relative stability of the cell volume ensures consistent spatial scale of data allocation, avoiding uneven data distribution due to excessive volume differences.

[0061] In practice, when acquiring continuously changing values ​​of the user's posture sequence, the user's torso rotational angular velocity is extracted from the user posture sequence. The user posture sequence is acquired by a depth sensor and contains multiple posture frames. Each posture frame has a corresponding acquisition timestamp and includes the 3D coordinates of the left and right shoulder keypoints. For each posture frame, the 3D coordinates of the left and right shoulder keypoints are subtracted from the 3D coordinates of the right shoulder keypoint to obtain the torso vector. The torso vector is then normalized to obtain a unit torso vector.

[0062] To calculate the angle between unit torso vectors in two adjacent attitude frames, the unit torso vectors of the previous and subsequent attitude frames are multiplied by a dot product. The dot product result is used as input to an inverse cosine function, which calculates the angle between the unit torso vectors of the two adjacent attitude frames. Dividing the angle between the unit torso vectors of the two adjacent attitude frames by the time interval between the acquisition timestamps of the previous and subsequent attitude frames yields the torso rotational angular velocity. The magnitude of the torso rotational angular velocity indicates the speed of the user's rotation around the torso's longitudinal axis in physical space, while the sign of the torso rotational angular velocity indicates the direction of rotation around the torso's longitudinal axis in physical space.

[0063] To obtain the total duration of the interactive response sequence, read the start and end timestamps of the interactive response sequence, subtract the start timestamp from the end timestamp, and get the total duration of the interactive response sequence.

[0064] The adjustment offset is calculated based on the product of the trunk rotational angular velocity and the total duration of the interaction response sequence, using the following formula:

[0065] in, This indicates the adjustment offset, which is used to move the current presentation timestamp of the interactive response sequence; The torso rotation speed adjustment coefficient is dimensionless and its value is equal to the angle between the unit torso vectors of two adjacent posture frames (in radians) divided by the time interval between the acquisition timestamps of two adjacent posture frames (in seconds). This represents the total duration of the interactive response sequence, measured in seconds. The total duration is obtained by subtracting the start timestamp from the end timestamp of the interactive response sequence. When the value is positive, the adjustment offset is positive, indicating that the current presentation timestamp of the interactive response sequence moves forward; when... When the value is negative, the adjustment offset is negative, indicating that the current presentation timestamp of the interactive response sequence is moved forward.

[0066] When moving the current presentation timestamp of the interactive response sequence according to the adjustment offset, the current presentation timestamp of the interactive response sequence is obtained. The current presentation timestamp is added to the adjustment offset to obtain the synchronized current presentation timestamp. Using the synchronized current presentation timestamp, the corresponding data segments are read from the interactive response sequence for presentation, completing the synchronized adjustment of the presentation progress of the interactive response sequence and obtaining an immersive interactive service flow.

[0067] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An immersive interactive service method based on a multi-agent matrix, characterized in that, include: Multiple sensing units in the physical space are collected to construct a volume field matrix, which records the spatial coordinates and corresponding spatial attribute parameters of each voxel in the physical space. Capture the user gesture sequence in the physical space, and parse the user pointing vector from the user gesture sequence. The user pointing vector is used to identify the user's direction of attention in the physical space. Based on the spatial attribute parameters in the volume field matrix and the spatial positional relationship between the user pointing vector, activate the target volume submatrix corresponding to the user pointing vector in the volume field matrix; Based on the spatial coordinates in the target volume submatrix, an interactive response interface is mapped in the physical space, and the interactive response interface covers the physical space region corresponding to the target volume submatrix. The service content to be displayed is segmented into voxels according to the volume distribution characteristics of the target volume submatrix to obtain the service content voxel stream. The service content voxel stream is arranged according to the interface topology order of the interactive response interface to generate an interactive response sequence; The continuous change values ​​of the user posture sequence are collected, and the presentation progress of the interaction response sequence is synchronously adjusted according to the continuous change values ​​to obtain an immersive interactive service flow.

2. The immersive interactive service method based on a multi-agent matrix according to claim 1, characterized in that, The construction of the volumetric field matrix by acquiring multiple sensing units in the physical space includes: Obtain the boundary contour point cloud of the physical space and divide the boundary contour point cloud into several spatial grid units; Extract the center point coordinates and occupancy status of each spatial grid cell, and use the center point coordinates and occupancy status as the spatial attribute parameters of the spatial grid cell. The spatial attribute parameters of all spatial grid cells are combined according to the arrangement order of the spatial grid cells in the physical space to generate the volume field matrix.

3. The immersive interactive service method based on a multi-agent matrix according to claim 1, characterized in that, The step of capturing the user gesture sequence in the physical space and parsing the user pointing vector from the user gesture sequence includes: Extract the user's shoulder and wrist coordinates from the user's pose sequence, and calculate the direction of the line connecting the shoulder and wrist coordinates; Extract the user's facial orientation vector from the user's posture sequence, and then weight and synthesize the connecting direction with the facial orientation vector to obtain the synthesized pointing direction; The synthesized pointing direction is mapped onto the matrix coordinate system of the volume field matrix to generate the user pointing vector.

4. The immersive interactive service method based on a multi-agent matrix according to claim 1, characterized in that, The step of activating the target volume submatrix corresponding to the user-pointing vector in the volume field matrix based on the spatial attribute parameters in the volume field matrix and the spatial positional relationship between the user-pointing vector includes: Calculate the starting coordinates and direction vector of the user-pointing vector, and perform a ray step traversal along the direction vector in the volume field matrix; Record the cell index of each spatial grid cell traversed during the ray step traversal, and store the cell index in the active index list; Extract the spatial attribute parameters corresponding to each cell index in the activation index list from the volume field matrix, and combine all the extracted spatial attribute parameters into the target volume submatrix.

5. The immersive interactive service method based on a multi-agent matrix according to claim 1, characterized in that, The step of mapping the interactive response interface in the physical space based on the spatial coordinates in the target volume submatrix includes: Extract the spatial coordinates and occupancy status of each spatial grid cell in the target volume submatrix, and filter out multiple spatial grid cells with empty occupancy status to obtain a set of empty cells; Surface fitting is performed on the spatial coordinates of the set of vacant units to generate a smooth mapped surface; The smoothed mapped surface is projected onto the physical space to obtain the interactive response interface.

6. The immersive interactive service method based on a multi-agent matrix according to claim 1, characterized in that, The step of voxelizing and segmenting the service content to be displayed according to the volume distribution features of the target volume submatrix to obtain the service content voxel stream includes: Obtain the raw data stream of the service content to be displayed, and divide the raw data stream into several data segments in chronological order; Extract the capacity distribution features of the target volume submatrix, wherein the capacity distribution features include the cell volume and cell density of each spatial grid cell in the target volume submatrix; Based on the correspondence between the data volume of each data segment and the cell volume and cell density, each data segment is mapped to the corresponding spatial grid cell to obtain the service content voxel stream.

7. The immersive interactive service method based on a multi-agent matrix according to claim 1, characterized in that, The step of arranging the service content voxel stream according to the interface topology order of the interactive response interface to generate an interactive response sequence includes: Extract the interface topology order of the interactive response interface, where the interface topology order is the arrangement order of the various spatial grid units covered by the interactive response interface. Read the data segments carried by each spatial grid unit in the service content voxel stream in the order described above; All the data fragments read are sequentially concatenated in the order they were read to generate the interactive response sequence.

8. The immersive interactive service method based on a multi-agent matrix according to claim 1, characterized in that, The step of collecting continuous change values ​​of the user gesture sequence and synchronously adjusting the presentation progress of the interaction response sequence based on the continuous change values ​​includes: Extract the user's torso rotation angular velocity from the user's posture sequence, and use the torso rotation angular velocity as the continuously changing value; Obtain the total duration of the interaction response sequence, and calculate the adjustment offset based on the product of the torso rotation angular velocity and the total duration; The current presentation timestamp of the interactive response sequence is moved according to the adjustment offset to complete the synchronous adjustment of the presentation progress of the interactive response sequence.

9. The immersive interactive service method based on a multi-agent matrix according to claim 2, characterized in that, When dividing the boundary contour point cloud into several spatial grid units, the division granularity of the grid units is adaptively adjusted according to the geometric complexity of the physical space. A finer division granularity is used in areas where the geometric contour curvature is greater than a preset threshold, and a coarser division granularity is used in areas where the geometric contour curvature is less than a preset threshold.

10. The immersive interactive service method based on a multi-agent matrix according to claim 6, characterized in that, When mapping each data segment to the corresponding spatial grid cell, the corresponding data compression ratio is determined according to the cell density of each spatial grid cell. The higher the cell density of the spatial grid cell, the higher the data compression ratio is used.

Images