A multimodal interactive method and system for virtual images in a smart exhibition hall

Abstract

The present application relates to the field of multimodal interaction technology, and specifically to a multimodal interaction method and system for a virtual image of a smart exhibition hall, the method comprising: obtaining point cloud data of a smart exhibition hall, and dividing the point cloud data of the smart exhibition hall into voxel grids to obtain each voxel grid; analyzing the local structural disorder of each point cloud data in each voxel grid to cluster the point cloud data in each voxel grid, and then constructing the local structural abnormality of each point cloud data in the voxel grid, and combining the abnormal detection results of each point cloud data in the voxel grid to determine the point cloud abnormal deviation of each point cloud data in the voxel grid, based on which the point cloud data is eliminated, and all the eliminated point cloud data are filtered, and a virtual scene of a smart exhibition hall is established through three-dimensional reconstruction and multimodal information to perform multimodal interaction. The present application can improve the optimization accuracy of point cloud data in smart exhibition halls, increase data purity, and improve visual interaction effects.

Description

Technical Field

[0001] The present application relates to the field of multimodal interaction technology, and in particular to a multimodal interaction method and system for virtual images of a smart exhibition hall. Background Art

[0002] Virtual Reality (VR) and Augmented Reality (AR) are increasingly being used in smart exhibition halls. Virtual Reality uses computers to simulate real scenes in smart exhibition halls, and uses simulated virtual environments to give users an immersive experience. To enhance the user's immersive experience and satisfaction, multiple modal fusion interactions are used to integrate multiple perception channels, including vision, hearing, and touch, to reproduce the real scenes of smart exhibition halls. AR devices are used to achieve virtual interaction to stimulate users' multi-sensory experience, thereby creating an immersive viewing and learning experience for users.

[0003] At present, in the process of 3D visual reproduction of the smart exhibition hall, it is necessary to reconstruct the 3D model of the smart exhibition hall through the three-dimensional point cloud data of the smart exhibition hall. However, the collection process of the point cloud data of the smart exhibition hall is easily affected by external noise interference, which makes the point cloud data of the smart exhibition hall highly redundant and irregular. It is often necessary to optimize the point cloud data of the smart exhibition hall through voxel filtering algorithm in order to improve the quality and accuracy of the point cloud data of the smart exhibition hall.

[0004] The traditional voxel filtering algorithm takes the centroid of the voxel grid as the representative point to approximately replace all the point cloud data in the voxel grid. However, due to the interference of external noise, there will be a certain offset error in the centroid of the voxel grid, which will affect the quality and accuracy of the point cloud data of the smart exhibition hall, and ultimately lead to poor 3D visual reproduction of the smart exhibition hall, reducing the user's immersive experience in the smart exhibition hall. Summary of the invention

[0005] In order to solve the above technical problems, the purpose of this application is to provide a multimodal interaction method and system for virtual images in a smart exhibition hall. The technical solutions adopted are as follows:

[0006] The present application provides a multimodal interaction method for a virtual image of a smart exhibition hall, comprising the following steps:

[0007] Obtaining point cloud data of the smart exhibition hall, and dividing the point cloud data of the smart exhibition hall into voxel grids to obtain each voxel grid;

[0008] The local structural disorder of each point cloud data in each voxel is obtained through the local density distribution of the neighboring point cloud data of each point cloud data in each voxel, and the difference degree of local density between each point cloud data and its neighboring point cloud data, so as to cluster the point cloud data in each voxel;

[0009] According to the local structural disorder of each point cloud data in the voxel grid and the deviation of the local structural disorder between each point cloud data and the point cloud data in its cluster, the local structural abnormality of each point cloud data in the voxel grid is constructed, and the abnormal detection results of each point cloud data in the voxel grid are combined to determine the point cloud abnormal deviation of each point cloud data in the voxel grid;

[0010] The point cloud data is eliminated through the abnormal deviation of each point cloud data in the voxel grid, and all the eliminated point cloud data are filtered to obtain the optimized point cloud data of the smart exhibition hall;

[0011] The optimized point cloud data of the smart exhibition hall is reconstructed in three dimensions, and combined with multimodal information to establish a virtual scene of the smart exhibition hall for multimodal interaction.

[0012] Preferably, the process of acquiring the local density further comprises: inputting all the point cloud data points in each voxel grid into a density peak clustering algorithm, and outputting the local density of each point cloud data in each voxel grid.

[0013] Preferably, the setting condition of the cutoff distance of the density peak clustering algorithm is: to ensure that the number of point cloud data whose distance to each point cloud data is less than the cutoff distance accounts for a preset proportion of the total number of point cloud data in the voxel grid.

[0014] Preferably, the process of determining the neighboring point cloud data of each point cloud data is: taking each point cloud data as the center and the cutoff distance as the radius, the point cloud data within the cutoff neighborhood is recorded as the neighboring point cloud data of each point cloud data.

[0015] Preferably, the calculation process of the local structural disorder of each point cloud data in each voxel is:

[0016] ; In the formula, is the local structural disorder of the jth point cloud data in the i-th cell, is the permutation entropy of the local density of all neighboring point cloud data of the jth point cloud data in the i-th cell, and are the local densities of the jth point cloud data and its kth neighboring point cloud data in the i-th volume grid, respectively.

[0017] Preferably, the metric distance in the process of clustering the point cloud data in each voxel is the absolute value of the difference between the local structural disorder corresponding to different point cloud data.

[0018] Preferably, the calculation process of the local structural abnormality of each point cloud data in the voxel grid is:

[0019] ; In the formula, is the local structural abnormality of the jth point cloud data in the i-th cell, is the number of point cloud data in the cluster where the jth point cloud data in the i-th cell is located, is the exponential normalization function, is the local structural disorder of the sth point cloud data in the cluster where the jth point cloud data in the i-th cell is located, is the local structural disorder of the jth point cloud data in the i-th cell.

[0020] Preferably, the calculation process of the point cloud abnormal deviation of each point cloud data in the voxel grid is:

[0021] ; In the formula, is the point cloud abnormal deviation of the jth point cloud data in the i-th cell, is the LOF outlier value of the j-th point cloud data in the i-th cell, is the local structural abnormality of the jth point cloud data in the i-th cell.

[0022] Preferably, the process of determining whether to remove the point cloud data is as follows:

[0023] For each voxel grid, the point cloud abnormal deviation of all point cloud data in the voxel grid is threshold segmented, and the point cloud data with point cloud abnormal deviation higher than the segmentation threshold is removed from the voxel grid.

[0024] An embodiment of the present application also provides a multimodal interactive system for virtual images of a smart exhibition hall, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of any one of the above methods when executing the computer program.

[0025] From the above, it can be seen that the multimodal interaction method and system of virtual images of a smart exhibition hall provided by the present application have at least the following beneficial effects:

[0026] The present application analyzes the local structure of the point cloud data in each voxel grid, and uses the normalized result of the local structure disorder to perform weighted summation on the deviation difference of the local structure disorder, so that the measurement result of the local structure abnormality can accurately reflect the abnormal structural characteristics of the point cloud data in different clusters, and uses the normalized result of the LOF abnormal value to represent the abnormal characteristic coefficient of the point cloud data offset, and enhances the characteristics of the point cloud abnormal offset, so that the measured point cloud abnormal offset can better reflect the significant characteristics of the point cloud abnormal offset, which is used to more accurately eliminate the noise point cloud with abnormal point cloud offset in the voxel grid in the future;

[0027] The present application eliminates the noisy point clouds with abnormal point cloud offsets within the voxel grid, and selects the representative points of each voxel grid through the remaining standard point cloud data in each voxel grid, so that the representative points selected by the voxel filtering algorithm will not be affected by external noise interference, avoiding a certain offset error in the center of mass within the voxel grid, which affects the quality and accuracy of the point cloud data of the smart exhibition hall after optimization processing, and ultimately improves the effect of 3D visual reproduction of the smart exhibition hall and enhances the user's immersive experience in the smart exhibition hall. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 A flowchart of the steps of a multimodal interaction method of virtual images in a smart exhibition hall provided in this application;

[0030] Figure 2 The aircraft model point cloud data after traditional voxel filtering optimization processing provided in this application;

[0031] Figure 3 The aircraft model point cloud data is processed by voxel filtering optimization in this application. DETAILED DESCRIPTION

[0032] In order to further explain the technical means and effects adopted by this application to achieve the predetermined invention purpose, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the specific implementation method, structure, features and effects of a multimodal interactive method and system for virtual images of a smart exhibition hall proposed in this application. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0033] Unless otherwise specified and limited, terms such as "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such articles or devices. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the article or device including the element. In addition, the term "and\or" used herein includes any and all combinations of one or more related listed items. All technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application.

[0034] The following is a detailed description of a multimodal interaction method and system for virtual images of a smart exhibition hall provided by the present application in conjunction with the accompanying drawings.

[0035] See also Figure 1 , which shows a flowchart of a multimodal interaction method of a virtual image of a smart exhibition hall provided by an embodiment of the present application, including the following steps:

[0036] The purpose of this embodiment is to eliminate the noise point cloud with abnormal point cloud offset within the voxel grid, so that the representative points selected in the voxel filtering algorithm can accurately show the structural characteristics of the smart exhibition hall, avoiding the impact on the quality and accuracy of the point cloud data of the smart exhibition hall; then, the 3D model of the smart exhibition hall is reconstructed by three-dimensional reconstruction technology, and a virtual scene is established based on the reconstructed 3D model of the smart exhibition hall and the multimodal information fusion calculation in the smart exhibition hall, and finally the visual interaction between the virtual scene and the user is realized through an external display device, thereby enhancing the user's immersive experience in the smart exhibition hall.

[0037] Step 1: obtain the point cloud data of the smart exhibition hall, and divide the point cloud data of the smart exhibition hall into voxel grids to obtain each voxel grid.

[0038] First, the point cloud data of the smart exhibition hall is collected by a laser radar scanner. Specifically, in this embodiment, taking the aircraft exhibition hall as an example, a handheld SLAM100 laser radar scanner is used to form a 270°360° spherical field of view by rotating scanning, and point cloud data at various positions in the aircraft exhibition hall are collected to ensure the integrity of the point cloud data of the aircraft exhibition hall and generate all the point cloud data of the smart exhibition hall.

[0039] Furthermore, the voxel filtering algorithm is used to optimize the point cloud data of the smart exhibition hall. First, all the point cloud data of the smart exhibition hall needs to be divided into voxel grids of the same size. It should be noted that the implementer can set the division method for the specific division of the voxel grid of the point cloud data of the smart exhibition hall in the actual application scenario, and there is no special restriction on this in this embodiment. Preferably, the specific division process in this embodiment is as follows:

[0040] The first step is to traverse the three-dimensional values ​​of all point cloud data in the smart exhibition hall through statistical methods, and obtain the maximum boundary values ​​of all point cloud data in the three dimensions of X, Y, and Z respectively. and minimum boundary value .

[0041] The second step is to determine the dimension lengths of the three dimensions in the space where the smart exhibition hall is located based on the maximum and minimum boundary values ​​of all point cloud data of the smart exhibition hall. , where the length of the edge of the X dimension in the space where the smart exhibition hall is located is The maximum boundary value in the X dimension With minimum boundary value The difference between the Y dimension and the side length of the space where the smart exhibition hall is located. The maximum boundary value in the Y dimension With minimum boundary value The difference between the two dimensions; the length of the dimension side in the Z dimension of the space where the smart exhibition hall is located The maximum boundary value in the Z dimension With minimum boundary value The difference.

[0042] The third step is to calculate the length of the three dimensions of the space where the smart exhibition hall is located. , set the side lengths of the three dimensions of each voxel grid to be , in this embodiment, , the implementer can adaptively select the dimension length of each voxel grid according to the actual size of the smart exhibition hall, divide all the point cloud data of the smart exhibition hall into voxel grids, and obtain Individual elements, among which is the rounding function, the dimension side length The unit of length is m.

[0043] Step 2: The local structural disorder of each point cloud data in each voxel is obtained through the local density distribution of the neighboring point cloud data of each point cloud data in each voxel, as well as the difference in local density between each point cloud data and its neighboring point cloud data, so as to cluster the point cloud data in each voxel.

[0044] Since point cloud data collection is affected by external noise, the point cloud data in the voxel grid is highly redundant and irregular, which can easily lead to a certain offset of the center of mass in the voxel grid. At this time, the center of mass of the voxel grid cannot accurately express the structural characteristics of the smart exhibition hall. The traditional voxel filtering algorithm uses the center of mass in the voxel grid as a representative point, which approximately replaces all point cloud data points in the voxel grid, which will reduce the accuracy of the point cloud data after optimization. Therefore, in order to improve the accuracy of the point cloud data after optimization, it is necessary to analyze the point cloud data features in each voxel grid.

[0045] The dense structural features of the point cloud data in each voxel grid are analyzed, and all the point cloud data points in each voxel grid are input into the DPC density peak clustering algorithm (Density Peak Clustering, DPC). In this embodiment, the selection method of the truncation distance of the DPC density peak clustering algorithm is: for each point cloud data, it is necessary to ensure that the number of point cloud data whose distance to each point cloud data is less than the truncation distance accounts for a preset ratio of the total number of point cloud data in the voxel grid, where the preset ratio in this embodiment is 2%. In this embodiment, the truncation distance is determined by the 2% ratio. In actual application scenarios, the implementer can set it by himself. The DPC density peak clustering algorithm outputs the local density of each point cloud data in each voxel grid. The DPC density peak clustering algorithm is a well-known technology, and the specific process will not be repeated.

[0046] Furthermore, for the point cloud data in each voxel grid, each point cloud data is taken as the center and the truncation distance is taken as the radius, and the point cloud data in the truncation neighborhood of each point cloud data are recorded as the neighboring point cloud data of each point cloud data.

[0047] Under the influence of external noise interference, noisy point cloud data will appear in the voxel grid. There will be a large difference between the local density of the noise point cloud data and the local density of the normal point cloud data in the local range, making the point cloud structure in the local range have obvious irregular characteristics, while the local density difference between the normal point cloud data in the local range will be relatively small, and the point cloud structure in the local range has a strong regularity feature.

[0048] Through the above analysis, the local structural disorder of each point cloud data in each voxel grid is calculated based on the local density distribution of the neighboring point cloud data of each point cloud data in each voxel grid, and the difference in local density between each point cloud data and its neighboring point cloud data:

[0049] ; In the formula, is the local structural disorder of the jth point cloud data in the i-th cell, is the permutation entropy of the local density of all neighboring point cloud data of the jth point cloud data in the i-th cell, and are the local densities of the jth point cloud data and its kth neighboring point cloud data in the i-th cell. The calculation of permutation entropy is a well-known technique, and the specific process will not be described in detail.

[0050] The local structure disorder reflects the irregular characteristics of the point cloud structure within the local range of the point cloud data in the voxel grid. Among them, the permutation entropy represents the disorder degree of the local density of the point cloud data points in the local range, and the local density difference represents the change of the point cloud structure in the local range. The irregular characteristics of the point cloud structure are measured by combining the permutation entropy with the local density difference, which more accurately reflects the irregular characteristics of the point cloud structure.

[0051] Furthermore, in order to more accurately analyze the abnormal features of the local structure of the point cloud data in the voxel grid, all the point cloud data in each voxel grid are input into the k-means clustering algorithm. The metric distance in the algorithm is the absolute value of the difference between the local structure disorder corresponding to the point cloud data. The number of clusters is 2, and the k-means clustering algorithm outputs two clusters. The k-means clustering algorithm is a well-known technology, and the specific process will not be repeated here.

[0052] Step 3: construct the local structural abnormality of each point cloud data in the voxel grid according to the local structural disorder of each point cloud data in the voxel grid and the deviation of the local structural disorder between each point cloud data and the point cloud data in its cluster.

[0053] The point cloud data in each voxel are divided into two clusters by the difference in local structural disorder between point cloud data. One cluster represents a set of point cloud data corresponding to higher local structural disorder, and the other cluster represents a set of point cloud data corresponding to lower local disorder. If the local structural disorder of a point cloud data deviates from other point cloud data in the cluster, the more it can reflect the irregular changes in the local structure of the point cloud data. At this time, the abnormal features of the neglected local structure of the point cloud are clearer, that is, the more it can reflect the phenomenon of point cloud deviation when disturbed by external noise.

[0054] Through the above analysis, the local structural abnormality of each point cloud data in each voxel grid is calculated according to the local structural disorder of each point cloud data in the voxel grid and the deviation of the local structural disorder between each point cloud data and the point cloud data in its cluster:

[0055] ; In the formula, is the local structural abnormality of the jth point cloud data in the i-th cell, is the number of point cloud data in the cluster where the jth point cloud data in the i-th cell is located, is the exponential normalization function, is the local structural disorder of the sth point cloud data in the cluster where the jth point cloud data in the i-th cell is located, is the local structural disorder of the jth point cloud data in the i-th cell.

[0056] The normalized result of the local structure disorder is used to perform weighted summation on the deviation difference of the local structure disorder, so that the measurement result of the local structure abnormality can accurately reflect the abnormal structure characteristics of the point cloud data in different clusters, thereby improving the accuracy of measuring the local structure abnormal characteristics of the point cloud data.

[0057] The larger the abnormal structural features of the point cloud data in the voxel grid, the more it can reflect the point cloud offset characteristics when the point cloud data is affected by external noise interference, resulting in a certain offset of the center of mass in the voxel grid. Therefore, in order to reduce the impact of point cloud offset in the voxel grid, it is necessary to eliminate the noise point cloud with abnormal point cloud offset in the voxel grid.

[0058] In order to enhance the significant features of different point cloud data offsets, all point cloud data in each voxel grid are input into the LOF anomaly detection algorithm (Local Outlier Factor). The nearest neighbor parameter in the algorithm is 10. The LOF anomaly detection algorithm outputs the LOF anomaly value of each point cloud data in each voxel grid. The LOF anomaly detection algorithm is a well-known technology and the specific process will not be repeated here.

[0059] The LOF outlier value of each point cloud data in each voxel grid can measure the offset abnormal characteristics of each point cloud data in the voxel grid. In order to make the selected representative points more accurately reflect the structural characteristics of the smart exhibition hall and avoid affecting the quality and accuracy of the point cloud data of the smart exhibition hall, the normalized result of the LOF outlier value represents the abnormal characteristic coefficient of the point cloud data offset, and the offset characteristics of the point cloud data in the voxel grid are measured in combination with the local structural abnormality of the point cloud data.

[0060] According to the above analysis, the point cloud abnormality offset of each point cloud data in each voxel grid is calculated according to the local structural abnormality of each point cloud data in the voxel grid and the LOF abnormality value of each point cloud data in the voxel grid:

[0061] ; In the formula, is the point cloud abnormal deviation of the jth point cloud data in the i-th cell, is the LOF outlier value of the j-th point cloud data in the i-th cell.

[0062] The local structural abnormality of point cloud data reflects the structural abnormality characteristics under the influence of noise. The more significant the structural abnormality characteristics are, the more likely it is that the point cloud will be offset abnormally. At the same time, the normalized result of the LOF abnormality value represents the abnormal characteristic coefficient of the point cloud data offset, and the characteristics of the point cloud abnormal offset are enhanced, so that the measured point cloud abnormal offset can better reflect the significant characteristics of the point cloud abnormal offset, which is used to accurately eliminate the noise point cloud with abnormal point cloud offset in the voxel grid in the future.

[0063] Step 4: The point cloud data is eliminated by judging the abnormal deviation of each point cloud data in the voxel grid, and all the eliminated point cloud data are filtered to obtain the optimized point cloud data of the smart exhibition hall.

[0064] Generally speaking, the point cloud data corresponding to a higher point cloud anomaly deviation is more likely to be a noise point cloud generated by external noise interference, and the deviation anomaly characteristics of the noise point cloud are more significant; while the point cloud data corresponding to a lower point cloud anomaly deviation is more likely to be a real point cloud, and the centroid of the real point cloud can accurately reflect the structural characteristics of the smart exhibition hall, and can effectively improve the accuracy of the point cloud data after optimization processing.

[0065] Specifically, the point cloud abnormal deviation of all point cloud data in each voxel grid is input into the maximum inter-class variance algorithm, and the maximum inter-class variance algorithm outputs a segmentation threshold. The maximum inter-class variance algorithm is a well-known technology, and the specific process is not repeated here.

[0066] Furthermore, point cloud data with abnormal point cloud deviation higher than the segmentation threshold are removed from the voxel grid. In this embodiment, all remaining point cloud data in each voxel grid are recorded as all standard point cloud data in each voxel grid, and all standard point cloud data in all voxel grids are input into the voxel filtering algorithm. The output of the voxel filtering algorithm is the optimized point cloud data of the smart exhibition hall, wherein the voxel filtering algorithm is a well-known technology and the specific process will not be repeated here.

[0067] Specifically, in this embodiment, taking the point cloud data of the aircraft model in the smart exhibition hall as an example, the point cloud data of the aircraft model after the traditional voxel filtering optimization processing is as follows: Figure 2 As shown in FIG. 1 , the point cloud data at the edge of the aircraft model has a relatively obvious point cloud offset phenomenon, and the selected representative points cannot accurately show the structural characteristics of the aircraft model; the point cloud data of the aircraft model after the voxel filtering optimization process in this embodiment is as follows: Figure 3 As shown in the figure, point cloud data with high abnormal deviation are removed from the voxel grid, and then all standard point cloud data in each voxel grid are optimized by voxel filtering, so that the representative points selected by voxel filtering can more accurately show the structural characteristics of the aircraft model. Figure 3The quality and accuracy of the aircraft model point cloud data have been improved. Figure 2 and Figure 3 The X-axis, Y-axis, and Z-axis are respectively the X, Y, and Z coordinate axes in the three-dimensional coordinate system.

[0068] Step 5: Reconstruct the optimized point cloud data of the smart exhibition hall in three dimensions, and combine it with multimodal information to establish a virtual scene of the smart exhibition hall for multimodal interaction.

[0069] Furthermore, in the 3D reconstruction process, the optimized point cloud data of the smart exhibition hall is reconstructed in 3D by using 3D reconstruction technology to achieve 3D visual reproduction of the smart exhibition hall and obtain the 3D model of the smart exhibition hall after 3D reconstruction. At the same time, in the process of multimodal information acquisition, multimodal information of the smart exhibition hall in the real scene is acquired through multiple sensing devices, including temperature information in the environment, object information in the environment, and sound information in the environment.

[0070] In the process of multimodal information fusion, the 3D model of the smart exhibition hall after three-dimensional reconstruction and the multimodal information in the smart exhibition hall are fused and calculated to establish the virtual scene of the smart exhibition hall. During the fusion calculation, the virtual scene is constructed based on the 3D model of the smart exhibition hall after three-dimensional reconstruction, the temperature information in the environment, the item information in the environment, and the sound information in the environment, and the temperature information in the environment, the item information in the environment, and the sound information in the environment are mapped and simulated, and finally the virtual scene of the smart exhibition hall is obtained. Among them, the position tracking can use infrared, ultrasonic or camera technology to achieve real-time tracking of the entrant's position to ensure that the virtual scene is synchronized with the entrant's position. The viewing angle adjustment can be achieved through programming, and the viewing angle of the virtual scene is automatically adjusted according to the entrant's position and action to keep it consistent with the entrant's viewing angle. In the access interaction process, by connecting the virtual scene to an external display device, the display device is a virtual enhancement device, and the virtual scene of the smart exhibition hall is provided to the user through the virtual enhancement device, so as to achieve visual interaction and meet the user's immersive experience. Among them, three-dimensional reconstruction and fusion calculation to establish virtual scenes are both well-known technologies, and the specific process will not be repeated.

[0071] Based on the same inventive concept as the above method, an embodiment of the present application also provides a multimodal interaction system for virtual images of a smart exhibition hall, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the steps of any one of the above-mentioned multimodal interaction methods for virtual images of a smart exhibition hall are implemented.

[0072] It is to be understood that the sequence of the embodiments of the present application described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of the present specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0073] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. The above content is only an implementation method of this application and is not used to limit the scope of this application. Any equivalent structure or equivalent process transformation made using the content of this application specification and drawings, or directly or indirectly used in other related technical fields, is also included in the protection scope of this application.

Claims

1. A multimodal interaction method for virtual images in a smart exhibition hall, characterized in that: The following steps are involved: Obtaining point cloud data of the smart exhibition hall, and dividing the point cloud data of the smart exhibition hall into voxel grids to obtain each voxel grid; The local structural disorder of each point cloud data in each voxel is obtained through the local density distribution of the neighboring point cloud data of each point cloud data in each voxel, and the difference degree of local density between each point cloud data and its neighboring point cloud data, so as to cluster the point cloud data in each voxel; According to the local structural disorder of each point cloud data in the voxel grid and the deviation of the local structural disorder between each point cloud data and the point cloud data in its cluster, the local structural abnormality of each point cloud data in the voxel grid is constructed, and the abnormal detection results of each point cloud data in the voxel grid are combined to determine the point cloud abnormal deviation of each point cloud data in the voxel grid; The point cloud data is eliminated through the abnormal deviation of each point cloud data in the voxel grid, and all the eliminated point cloud data are filtered to obtain the optimized point cloud data of the smart exhibition hall; The optimized point cloud data of the smart exhibition hall is reconstructed in three dimensions, and combined with multimodal information, a virtual scene of the smart exhibition hall is established for multimodal interaction; The expression of the local structural disorder is: ; In the formula, is the local structural disorder of the jth point cloud data in the i-th cell, is the permutation entropy of the local density of all neighboring point cloud data of the jth point cloud data in the i-th cell, and are the local densities of the jth point cloud data and its kth neighboring point cloud data in the i-th cell; The expression of the local structure abnormality is: ; In the formula, is the local structural abnormality of the jth point cloud data in the i-th cell, is the number of point cloud data in the cluster where the jth point cloud data in the i-th cell is located, is the exponential normalization function, is the local structural disorder of the sth point cloud data in the cluster where the jth point cloud data in the i-th cell is located, is the local structural disorder of the jth point cloud data in the i-th cell.

2. The multimodal interaction method of a virtual image of a smart exhibition hall according to claim 1, characterized in that: The process of acquiring the local density further includes: inputting all the point cloud data points in each voxel grid into a density peak clustering algorithm, and outputting the local density of each point cloud data in each voxel grid.

3. The multimodal interaction method of a virtual image of a smart exhibition hall as claimed in claim 2, characterized in that: The setting condition of the cutoff distance of the density peak clustering algorithm is: to ensure that the number of point cloud data whose distance to each point cloud data is less than the cutoff distance accounts for a preset proportion of the total number of point cloud data in the voxel grid.

4. The multimodal interaction method of a virtual image of a smart exhibition hall as claimed in claim 3, characterized in that: The process of determining the neighboring point cloud data of each point cloud data is as follows: taking each point cloud data as the center and the cutoff distance as the radius, the point cloud data within the cutoff neighborhood is recorded as the neighboring point cloud data of each point cloud data.

5. The multimodal interaction method of a virtual image of a smart exhibition hall according to claim 1, characterized in that: The metric distance in the process of clustering the point cloud data in each voxel is the absolute value of the difference between the local structural clutter corresponding to different point cloud data.

6. The multimodal interaction method of a virtual image of a smart exhibition hall according to claim 1, characterized in that: The calculation process of the point cloud abnormal deviation of each point cloud data in the voxel grid is as follows: ; In the formula, is the point cloud abnormal deviation of the jth point cloud data in the i-th cell, is the LOF outlier value of the j-th point cloud data in the i-th cell, is the local structural abnormality of the jth point cloud data in the i-th cell.

7. The multimodal interaction method of a virtual image of a smart exhibition hall according to claim 1, characterized in that: The process of determining the removal of point cloud data is as follows: For each voxel grid, the point cloud abnormal deviation of all point cloud data in the voxel grid is threshold segmented, and the point cloud data with point cloud abnormal deviation higher than the segmentation threshold is removed from the voxel grid.

8. A multimodal interactive system of virtual images of a smart exhibition hall, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images