Digital vision multi-dimensional virtual interaction system for smart museum

Abstract

The invention relates to the technical field of augmented reality, in particular to an intelligent museum digital vision multi-dimensional virtual interaction system. The operation influence of each skeleton point of a tourist in each operation is obtained according to the movement condition of the same skeleton point in any two times of training in the training period of each operation, and the influence index of each skeleton point of the tourist is obtained by combining the edge bending degree in a hand RGB image; according to the difference of the times of executing different operations by each cultural relic, adjusting the times of using each operation by the tourist and the times of executing each operation by the cultural relic, and obtaining an identification index of each operation by each cultural relic; and the total amount of each resource of the AR interaction system is adjusted in combination with the influence index and the identification index, and the distribution amount of each cultural relic to each skeleton point of the tourist in each operation under each resource is obtained. According to the method, the resource allocation amount is dynamically adjusted by combining the operation habits of tourists, cultural relic characteristics and the influence degree of skeleton points on the operation of the tourists, and the interaction effect of the tourists is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of augmented reality, and in particular to a digital visual multi-dimensional virtual interactive system for a smart museum. Background Art

[0002] As an important carrier of historical and cultural heritage, museums are gradually moving away from traditional display methods and towards a digital and intelligent development path. Smart museums not only provide realistic exhibits, but also allow visitors to interact with historical relics at a deeper level, creating an intelligent space that integrates knowledge dissemination and cultural experience, greatly enriching the public's cultural life and learning methods.

[0003] In the AR interaction scene of the smart museum, efficient and accurate recognition of visitors' actions and operation intentions in complex environments is a key issue. During the AR interaction process, environmental factors may cause the camera to be blocked when collecting visitors' hand movements, causing errors in key point recognition, resulting in response delays and interaction errors, and thus inappropriate resource allocation of the AR interaction system, reducing the interaction effect of visitors. Summary of the invention

[0004] In order to solve the technical problem that errors in key point recognition during the interaction process cause inappropriate system resource allocation and reduce the interaction effect, the purpose of the present invention is to provide a smart museum digital visual multi-dimensional virtual interaction system, and the technical solutions adopted are as follows: The present invention proposes a smart museum digital visual multi-dimensional virtual interactive system, the system comprising: The data acquisition module is used to obtain the skeleton points of the RGB images of the hand of the visitor at each moment of each training period for each operation during the interactive learning period, the number of times the visitor uses each operation during the tour period, and the number of times each cultural relic in the museum performs each operation during the historical exhibition period; The skeleton point influence analysis module is used to obtain the operational influence of each skeleton point of the tourist in each operation according to the similarity of the movement of the same skeleton point in any two trainings within the training period of each operation and the movement of each skeleton point in each training; obtain the influence index of each skeleton point of the tourist according to the operational influence of each skeleton point of the tourist in all operations and the edge curvature of the hand RGB image at each moment of the training period of each operation; The cultural relic operation identification module is used to adjust the number of times tourists use each operation during the tour period and the number of times each cultural relic performs each operation during the historical exhibition period according to the difference in the number of times each cultural relic performs different operations during the historical exhibition period, and obtain the identification index of each cultural relic for each operation; The resource allocation module is used to adjust the total amount of each resource of the AR interactive system according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the tourist, so as to obtain the allocation amount of each cultural relic for each skeleton point of the tourist in each operation under each resource.

[0005] Furthermore, obtaining the operational influence of each skeleton point of the tourist on each operation includes: Obtain the running speed and acceleration of each skeleton point of the tourist at each moment of each training in the training period of each operation; The last training in the training period of each operation is recorded as the standard training of each operation; the importance of each skeleton point of the tourist in each operation is obtained according to the movement of each skeleton point of the tourist in the standard training of each operation and the degree of acceleration fluctuation at all times; Arrange the movement speed of each skeleton point of the tourist at all moments of each training in time sequence to obtain the speed sequence of each skeleton point of the tourist in each training; form a training pair by any two trainings in the training period of each operation, obtain the average of the overall speed difference values ​​of the speed sequence of the two trainings of each skeleton point of the tourist in all training pairs of each operation, negatively correlate and normalize the average, and obtain the stability of each skeleton point of the tourist in each operation; According to the importance and the stability, the operational influence of each skeleton point of the visitor in each operation is obtained, and both the importance and the stability are positively correlated with the operational influence.

[0006] Furthermore, obtaining the importance of each skeleton point of the tourist in each operation includes: Obtain the hand depth images of tourists at each moment of the standard training for each operation and the duration of the standard training; The pixel points at the same position of the skeleton points of the hand RGB image at each moment in the hand depth image are recorded as the skeleton points of the hand depth image at each moment, and the skeleton points of the hand depth image are converted into point cloud data; curve fitting is performed on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training of each operation to obtain the running trajectory of each skeleton point of the tourist in the standard training of each operation; the ratio of the length of the running trajectory to the duration is used as the standard average speed of each skeleton point of the tourist in each operation; Obtain the variance of the acceleration of each skeleton point of the tourist at all moments of the standard training of each operation as the discrete value of the acceleration of each skeleton point of the tourist in each operation; The importance of each skeleton point of the visitor in each operation is obtained according to the standard average speed and the acceleration discrete value, and both the standard average speed and the acceleration discrete value are positively correlated with the importance.

[0007] Furthermore, the obtaining of the influence index of each skeleton point of the tourist includes: The ratio of the duration of the training period to the interactive learning period of each operation is taken as the learning occupancy of each operation; According to the prominence degree of the same skeleton point in the local range of the hand RGB image at each moment of the standard training for each operation, the overall recognition degree of each skeleton point of the tourist in each operation is obtained; According to the operational influence of each skeleton point of the tourist in each operation, the overall recognition degree and the learning occupancy degree, the comprehensive influence value of each skeleton point of the tourist in each operation is obtained; the cumulative sum of the comprehensive influence values ​​of each skeleton point of the tourist in all operations is normalized to obtain the influence index of each skeleton point of the tourist.

[0008] Furthermore, the step of obtaining the overall recognition degree of each skeleton point of the tourist in each operation includes: For the hand RGB image at each moment, a circular area formed by taking each skeleton point of the hand RGB image as the center and half of the minimum distance between each skeleton point and the other skeleton points as the radius is used as the analysis area of ​​each skeleton point of the hand image; Performing corner point detection on the analysis area, counting the total number of corner points in the analysis area as the corner point amount of each skeleton point; performing edge detection on the analysis area, calculating the mean value of the curvature of all pixel points on the edge of the analysis area as the structural prominence value of each skeleton point; The product of the corner point quantity and the structural prominence value of each skeleton point is calculated as the local recognition degree of the corresponding skeleton point; the mean of the local recognition degree of the same skeleton point in the hand RGB images at all moments of the standard training of each operation is normalized to obtain the overall recognition degree of each skeleton point of the tourist in each operation.

[0009] Furthermore, obtaining the identification index of each cultural relic for each operation includes: The ratio of the number of times tourists use each operation during their visit to the total number of times they use all operations is used as the tourists' own influence indicator for each operation. The ratio of the number of times each cultural relic performs each operation to the total number of operations of all types during the historical exhibition period is used as the cultural relic impact indicator for each operation; The standard deviation of the number of times each cultural relic performs all kinds of operations during the historical exhibition period is normalized to obtain the operational influence of each cultural relic; The operation influence is used as the weight of the cultural relic influence index, the difference between the constant 1 and the operation influence is used as the weight of the self-influence index, and the self-influence index and the cultural relic influence index are weighted and summed to obtain the identification index of each cultural relic for each operation.

[0010] Furthermore, the total amount of each resource of the AR interactive system is adjusted according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the tourist, so as to obtain the distribution amount of each cultural relic for each skeleton point of the tourist in each operation under each resource, including: According to the recognition index of each cultural relic for each operation, the importance of each skeleton point of the tourist in each operation, and the influence index of each skeleton point of the tourist, the attention degree of each cultural relic to each skeleton point of the tourist in each operation is obtained; the recognition index, the importance and the influence index are all positively correlated with the attention degree; The total amount of each resource of the AR interactive system is obtained; the total amount of each resource of the AR interactive system is adjusted using the attention degree to obtain the distribution amount of each skeleton point of each cultural relic to the tourist in each operation under each resource.

[0011] Furthermore, the total amount of each resource of the AR interactive system is adjusted by using the attention degree to obtain the distribution amount of each operation of each skeleton point of each cultural relic to the tourist under each resource, including: Based on the attention of each cultural relic to all skeleton points of tourists in each operation, the attention of each cultural relic to each skeleton point of tourists in each operation is normalized to obtain the resource allocation weight of each cultural relic to each skeleton point of tourists in each operation; The resource allocation weight is used to weight the total amount of each resource of the AR interactive system, and the allocation amount of each cultural relic to each skeleton point of the tourist in each operation under each resource is obtained.

[0012] Furthermore, the method for obtaining the overall speed difference value of the speed sequence of two trainings in the training pair is a DTW algorithm.

[0013] Furthermore, the method for performing curve fitting on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training for each operation is the least squares method.

[0014] The present invention has the following beneficial effects: In the embodiment of the present invention, there are differences in the operating habits of tourists, which may cause the matching degree of some skeleton points in the training period to be low, so that the AR device has a poor recognition effect on each operation. The similarity of the movement of the same skeleton point in any two trainings of each operation and the movement of the skeleton point are used to analyze the influence of the skeleton point in each operation to obtain the operation influence, and combined with the image features in the local range of the skeleton point, the influence index of the skeleton point is analyzed to measure the influence of the skeleton point on various operations of tourists; the number of times tourists use each operation during the tour period presents the operating habits of tourists, and the number of times cultural relics perform each operation during the historical exhibition period presents the characteristics of cultural relics. Combined with the operating habits of tourists themselves and the characteristics of cultural relics, the recognition index of cultural relics for each operation is analyzed to measure the accuracy of each cultural relic performing each operation; the recognition index and the influence index are combined to dynamically allocate and adjust each resource of the AR interactive system, so that the resource allocation of the AR interactive system is more appropriate, the response delay and interaction error caused by the recognition errors of other skeleton points are reduced, and the interaction effect of tourists is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A system structure diagram of a digital visual multi-dimensional virtual interactive system for a smart museum provided by an embodiment of the present invention; Figure 2 A structural diagram of a skeleton point influence analysis module provided by one embodiment of the present invention; Figure 3 A schematic diagram of a computer device of a smart museum digital visual multi-dimensional virtual interactive device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation, structure, features and effects of a digital visual multi-dimensional virtual interactive system for a smart museum proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0019] The specific scheme of a smart museum digital visual multi-dimensional virtual interactive system provided by the present invention is described in detail below with reference to the accompanying drawings.

[0020] Embodiment 1: See also Figure 1 , which shows a system block diagram of a smart museum digital visual multi-dimensional virtual interaction system provided by an embodiment of the present invention. The system includes: a data acquisition module 110, a skeleton point influence analysis module 120, a cultural relic operation identification module 130, and a resource allocation module 140.

[0021] The data acquisition module 110 is used to obtain the skeleton points of the RGB image of the hand of the visitor at each moment of each training in the training period of each operation in the interactive learning period, the number of times the visitor uses each operation during the tour period, and the number of times each cultural relic in the museum performs each operation during the historical exhibition period.

[0022] In order to allow visitors to get a closer look at the museum's cultural relics and enhance their visiting experience, when visitors enter the relevant visiting areas set by the museum, staff will provide them with augmented reality (AR) devices, which can render and display three-dimensional models based on the cultural relics exhibited in the visiting area. Visitors can perform corresponding operations and interactions through relevant gestures to observe the rendered cultural relic models.

[0023] Visitors need to wear the AR device on their heads. The AR device is equipped with an RGB camera and a depth camera to capture the hand movements and spatial depth of the visitors. The images captured by the RGB camera and the depth camera are recorded as hand RGB images and hand depth images, respectively.

[0024] Before using AR devices to observe cultural relics, tourists need to learn simple interactive operations according to the operation prompts of the AR devices. The time period for tourists to learn interactive operations is recorded as the interactive learning period, and the time period for tourists to learn each interactive operation is recorded as the training period for each operation. The learning process of interactive operations is: First, the AR device guides tourists to perform simple gestures, such as clicking, grabbing, and rotating, through simple animation examples. Then, tourists imitate the gestures in the animation, and use the RGB camera and depth camera to obtain the RGB image and depth image of the tourists' hands at each moment during the interactive learning period. The two images are combined and the open posture estimation algorithm is used to obtain the skeleton points of the hand RGB image at each moment. Finally, the hand RGB image of each training for each operation is matched with the skeleton points of the animation. When the match is successful, the next interactive operation is guided until all interactive operations provided by the AR device are matched successfully.

[0025] Each training requires a certain amount of time, that is, each training has multiple RGB images of the hand at different moments, and the animation of each operation contains multiple frames of template images. The matching method is as follows: the RGB images of the hand at all moments in the time period of each training of each operation and the multiple frames of template images of the animation are recorded as images to be matched and standard images in turn, and the skeleton points of each image to be matched are matched with those of each standard image using a scale-invariant feature transformation algorithm to obtain matching pairs of skeleton points of each image to be matched with those of each standard image; the mean of the Euclidean distance between two skeleton points in all matching pairs of skeleton points of each image to be matched with those of each standard image is normalized to obtain matching indicators between each image to be matched and each standard image; if the matching indicators between all images to be matched and any standard image are greater than the preset matching threshold, the matching of the RGB images of the hand and the skeleton points of the animation within the time period of each training of each operation is successful; otherwise, the matching fails.

[0026] It should be noted that in the embodiment of the present invention, an open posture estimation algorithm is used to obtain skeleton points in the template image of the animation; the preset matching threshold is an empirical value of 0.9, which can be set by the implementer according to the specific situation; the Norm function is used for normalization processing; the image acquisition frequency of the RGB camera and the depth camera is once every 0.1 seconds, which can be set by the implementer according to the specific situation. Among them, the open posture estimation algorithm and the scale-invariant feature transformation algorithm are both well-known technologies to those skilled in the art and will not be described in detail here.

[0027] When all the operations of the visitor are matched successfully, the visitor operates and interacts with the cultural relics exhibited in the museum through all the operations learned above and observes the rendered cultural relic models. During this process, the skeleton points of the visitor's hands are continuously tracked, and each skeleton point of the visitor's hands can be recognized at all times in the process.

[0028] The time period from the last moment of the interactive learning period to the current moment is recorded as the tourist's tour period, and the number of times the tourist uses each operation during the tour period is counted; the week before the current day is regarded as the historical exhibition period, and tourists perform various operations on the three-dimensional model of the cultural relics through the AR devices they wear. The cultural relics perform each operation to facilitate tourists to understand the cultural relic information, and the number of times each cultural relic performs each operation during the historical exhibition period is counted.

[0029] The AR interaction in the smart museum is mainly through the interaction and matching of the skeleton points of the visitors themselves with the skeleton points in their virtual rendering model. During the interaction, the acquisition device on the AR device captures the operation of the visitors' hands and converts it into model rendering. The poor recognition accuracy of the AR device for the visitors' operations and the response delay of the rendering model operated on the device will greatly affect the visitors' interactive experience. Therefore, this solution dynamically adjusts the allocation of resources of the AR interaction system to the skeleton points to optimize the interaction effect.

[0030] The skeleton point influence analysis module 120 is used to obtain the operational influence of each skeleton point of the tourist in each operation based on the similarity of the movement of the same skeleton point in any two trainings within the training period of each operation, and the movement of each skeleton point in each training; and obtain the influence index of each skeleton point of the tourist based on the operational influence of each skeleton point of the tourist in all operations and the edge curvature in the hand RGB image at each moment in the training period of each operation.

[0031] See also Figure 2 , which shows a structural diagram of a skeleton point influence analysis module provided by an embodiment of the present invention, and the skeleton point influence analysis module includes: an importance analysis unit 121, an influence analysis unit 122, and an influence index analysis unit 123.

[0032] The importance analysis unit 121 is used to obtain the running speed and acceleration of each skeleton point of the tourist at each moment of each training in the training period of each operation; record the last training in the training period of each operation as the standard training of each operation; and obtain the importance of each skeleton point of the tourist in each operation based on the movement conditions of each skeleton point of the tourist in the standard training of each operation and the degree of acceleration fluctuation at all moments.

[0033] An inertial sensor is installed at each joint of the tourist's hand to measure the movement speed and acceleration of each skeleton point of the tourist at each moment during the interactive learning period. It should be noted that the data collection frequency of the inertial sensor and the RGB camera is the same, and the movement speed refers to the angular velocity.

[0034] Preferably, in some possible implementation modes of the embodiments of the present invention, the method for obtaining the importance includes: obtaining the hand depth image of the tourist at each moment of the standard training of each operation and the duration of the standard training; recording the pixel points at the same position of the skeleton points of the hand RGB image at each moment in the hand depth image as the skeleton points of the hand depth image at each moment, and converting the skeleton points of the hand depth image into point cloud data; performing curve fitting on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training of each operation to obtain the running trajectory of each skeleton point of the tourist in the standard training of each operation; taking the ratio of the length of the running trajectory to the duration as the standard average speed of each skeleton point of the tourist in each operation; obtaining the variance of the acceleration of each skeleton point of the tourist at all moments of the standard training of each operation as the acceleration discrete value of each skeleton point of the tourist in each operation; and obtaining the importance of each skeleton point of the tourist in each operation based on the standard average speed and the acceleration discrete value.

[0035] There are differences in the degree of influence of different skeleton points of tourists on different operations. For example, the grasping operation is mainly identified by identifying the relative movement of fingers and palms. The palm is basically motionless, and the fingers have a larger range of movement. Compared with the palm, the skeleton points at the finger position move faster and the acceleration fluctuation is more obvious. Therefore, the skeleton points at the finger position are more important in the grasping operation.

[0036] When using each operation, visitors need to determine the running track of each skeleton point. The faster the speed of the skeleton point in the running track and the more obvious the acceleration fluctuation of the skeleton point at all times of each operation, the more significant the angle change of the joint at the corresponding position of the skeleton point, and the more important the skeleton point is in each operation. The standard average speed is the average speed of the running track of the skeleton point in each operation. Therefore, the standard average speed is positively correlated with the acceleration discrete value and importance.

[0037] In the embodiment of the present invention, the product of the standard average speed of each skeleton point of the tourist in each operation and the acceleration discrete value is normalized to obtain the importance of each skeleton point of the tourist in each operation.

[0038] It should be noted that in the embodiment of the present invention, the Norm function is used for normalization processing, and other normalization methods may also be selected, such as function conversion, maximum and minimum normalization, etc., which are not limited here; the least squares method is used to perform curve fitting on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training of each operation. Among them, the least squares method and the method of converting the pixel points in the depth image into point cloud data are both well-known technologies to those skilled in the art, and will not be repeated here.

[0039] The influence analysis unit 122 is used to obtain the operational influence of each skeleton point of the visitor in each operation.

[0040] Preferably, in some possible implementation modes of the embodiments of the present invention, the method for obtaining the operational influence includes: arranging the movement speed of each skeleton point of the tourist at all moments of each training in time series to obtain the speed sequence of each skeleton point of the tourist in each training; forming a training pair by any two trainings in the training period of each operation, obtaining the mean of the overall difference in speed of the speed sequence of the two trainings of each skeleton point of the tourist in all training pairs of each operation, negatively correlating and normalizing the mean to obtain the stability of each skeleton point of the tourist in each operation; obtaining the operational influence of each skeleton point of the tourist in each operation according to the importance and stability, and the importance and stability are both positively correlated with the operational influence.

[0041] Due to the differences in tourists' operating habits, the matching degree of some skeleton points during the training period may be low, resulting in poor recognition of each operation by the AR device. To solve this problem, it is necessary to conduct an in-depth analysis of the matching of tourists' skeleton points during multiple trainings during the training period. If the movement of the skeleton points during multiple trainings during each operation is more similar, and the actual movement of the skeleton points during the tour period is more important, then the skeleton points will have more influence in each operation and the greater the influence of the operation.

[0042] Stability reflects the similarity of the motion of skeleton points in different trainings during the training period. When the stability is greater, the motion of skeleton points in different trainings during the training period is more similar, the motion of skeleton points during the tour period is more stable, and the possibility of skeleton points being successfully matched during the tour period is greater, making the skeleton points more influential in each operation. Importance analyzes the importance of skeleton points in each operation based on the motion of the last training in the training period of each operation. Standard training presents the actual motion of tourists' skeleton points during the tour period; when the importance is greater, the actual motion of skeleton points during the tour period is more important, and the skeleton points are more influential in each operation. Therefore, importance and stability are positively correlated with the influence of operation.

[0043] It should be noted that this embodiment uses a dynamic time warping (DTW) algorithm to obtain DTW values ​​of two speed sequences, wherein the DTW algorithm is a well-known technology for those skilled in the art and will not be described in detail herein.

[0044] In a specific implementation of the embodiment of the present invention, the operation influence is expressed by a formula: In the formula, is the influence of the a-th skeleton point of the tourist on the b-th operation; is the DTW value of the velocity sequence of the tourist's a-th skeleton point in the m-th training pair of the b-th operation for two trainings; For all training pairs of tourists operating in type b; is the stability of the a-th skeleton point of the tourist in the b-th operation; is the importance of the a-th skeleton point of the tourist in the b-th operation; exp is an exponential function with a natural constant as the base. It should be noted that this embodiment uses the exp function to Perform negative correlation and normalization.

[0045] The influence index analysis unit 123 is used to obtain the influence index of each skeleton point of the tourist.

[0046] In an embodiment of the present invention, a method for obtaining an influence index is as follows: taking the ratio of the duration of the training period to the interactive learning period of each operation as the learning occupancy of each operation; obtaining the overall recognition of each skeleton point of the tourist in each operation based on the prominence of the same skeleton point in the local range of the hand RGB image at each moment of the standard training of each operation; obtaining the comprehensive influence value of each skeleton point of the tourist in each operation based on the operational influence, overall recognition and learning occupancy of each skeleton point of the tourist in each operation; and normalizing the cumulative sum of the comprehensive influence values ​​of each skeleton point of the tourist in all operations to obtain the influence index of each skeleton point of the tourist.

[0047] Environmental factors may cause the camera to be blocked when capturing tourists' hand movements. During the monitoring of skeleton points during the tour period, some skeleton points may be lost or mismatched, which interferes with the matching of each operation. This embodiment analyzes the overall recognition of skeleton points by analyzing the image features of the local range of skeleton points. The specific method for obtaining the overall recognition is as follows: For the hand RGB image at each moment, a circular area formed by taking each skeleton point of the hand RGB image as the center and half of the minimum distance between each skeleton point and the other skeleton points as the radius is used as the analysis area for each skeleton point of the hand image; corner point detection is performed on the analysis area, and the total number of corner points in the analysis area is counted as the corner point quantity of each skeleton point; edge detection is performed on the analysis area, and the mean curvature of all pixel points on the edge of the analysis area is calculated as the structural prominence value of each skeleton point; the product of the corner point quantity and the structural prominence value of each skeleton point is calculated as the local recognition degree of the corresponding skeleton point; the mean of the local recognition degree of the same skeleton point in the hand RGB images of all moments of the standard training for each operation is normalized to obtain the overall recognition degree of each skeleton point of the tourist in each operation.

[0048] This embodiment analyzes the difficulty of identifying skeleton points through the number of corner points and edge curvature characteristics in the analysis area of ​​the skeleton points. If the number of corner points and edge curvature in the analysis area of ​​the skeleton points is larger, it means that the features of the skeleton points are more prominent in the image compared to other skeleton points, and the probability of matching failure of the skeleton points is smaller, then the skeleton points are easier to identify. The structural prominence value reflects the edge curvature in the analysis area of ​​the skeleton points. The larger the structural prominence value, the greater the edge curvature in the analysis area of ​​the skeleton points. Therefore, the product of the number of corner points of the skeleton points and the structural prominence value is used as the local recognition degree to analyze the difficulty of the skeleton points in the hand RGB image.

[0049] It should be noted that this embodiment uses the Harris corner extraction algorithm and the Canny operator to perform corner detection and edge detection on the hand RGB image in sequence. The Harris corner extraction algorithm and the Canny operator are both well-known technologies to those skilled in the art and will not be described in detail here.

[0050] If the operational influence is greater, the accuracy of judging each operation of tourists by identifying skeleton points during the tour period is higher, which means that the skeleton points are more influential in each operation, and the influence of the skeleton points on each operation is greater. If the overall recognition is greater, it means that each skeleton point is easier to identify in each operation, the position of the skeleton point during the tour period is more accurate, the accuracy of the skeleton point being correctly identified in each operation is higher, and the influence of the skeleton point on each operation is greater. If the learning occupancy is greater, it means that the training time for each operation is longer, and the accuracy of tourists using each operation during the browsing period is higher. Therefore, the operational influence, overall recognition and learning occupancy are all positively correlated with the influence index.

[0051] In a specific implementation of the embodiment of the present invention, the influence index of each skeleton point of the tourist is expressed by the formula: In the formula, is the influence index of the a-th skeleton point of the tourist; B is the number of types of operations; is the influence of the a-th skeleton point of the tourist on the b-th operation; is the overall recognition degree of the a-th skeleton point of the tourist in the b-th operation; is the learning occupancy of the b-th operation; is the total number of moments in the standard training period for the b-th operation; is the structural prominence value of the a-th skeleton point of the tourist in the RGB image of the hand at the n-th moment of the standard training for the b-th operation; is the local recognition degree of the a-th skeleton point of the tourist in the hand RGB image at the n-th moment of the standard training for the b-th operation; is the comprehensive influence value of the a-th skeleton point of the tourist in the b-th operation; Norm is the normalization function.

[0052] The cultural relic operation identification module 130 is used to adjust the number of times tourists use each operation during the tour period and the number of times each cultural relic performs each operation during the historical exhibition period according to the difference in the number of times each cultural relic performs different operations during the historical exhibition period, and obtain the identification index of each cultural relic for each operation.

[0053] During the tour period, the order and method of tourists' operations on different cultural relics are affected and intervened by tourists' operating habits and cultural relic characteristics. Tourists will have operating habits during the interaction with AR devices. For example, if a tourist is more accustomed to making detailed observations, the AR interactive system needs to pay more attention to the "view" or "rotate" operations of cultural relics and pay less attention to other operations. Cultural relic characteristics also affect tourists' operations. For example, tourists may prioritize zooming in on exquisite cultural relics to view the details of the cultural relics, so exquisite cultural relics often perform grabbing operations. Tourists may prioritize observing artworks or historical exhibits from different angles, so artworks often perform rotation operations.

[0054] The difference in the number of times that cultural relics perform different operations during the historical exhibition period reflects the greater the degree of influence of cultural relics on tourists' own operations. The number of times tourists use each operation during the tour period reflects their operating habits, and the number of times cultural relics perform each operation during the historical exhibition period reflects the characteristics of cultural relics. Combining the above factors, the identification index of each cultural relic for each operation is obtained.

[0055] Preferably, in some possible implementation modes of the embodiments of the present invention, the method for obtaining the identification index includes: taking the ratio of the number of times the tourist uses each operation during the tour period to the total number of times all types of operations are used as the tourist's own influence index for each operation; taking the ratio of the number of times each cultural relic performs each operation during the historical exhibition period to the total number of times all types of operations are performed as the cultural relic influence index for each operation; normalizing the standard deviation of the number of times each cultural relic performs all types of operations during the historical exhibition period to obtain the operation influence of each cultural relic; taking the operation influence as the weight of the cultural relic influence index, and the difference between the constant 1 and the operation influence as the weight of the own influence index, performing weighted summation on the own influence index and the cultural relic influence index to obtain the identification index of each cultural relic for each operation.

[0056] If the operation influence is greater, the difference in the number of different operations performed by each cultural relic during the historical exhibition period is greater, and there are large differences between different tourists in the characteristics of cultural relics and the operations performed by cultural relics, indicating that each cultural relic has a greater impact on the tourists' own operations. The self-influence index shows the characteristics of tourists' own habits, and the cultural relic influence index shows the characteristics of cultural relics; if the operation influence is greater, the influence of each cultural relic on the tourists' own operations is greater, indicating that the influence of cultural relic characteristics on tourists' operations during the interaction process is more significant, and paying more attention to cultural relic influence indicators is helpful to improve the recognition accuracy of each operation of tourists during the interaction process; conversely, the greater the influence of tourists' operation habits on cultural relics, paying more attention to self-influence indicators is helpful to improve the recognition accuracy of each operation of tourists during the interaction process.

[0057] The identification index of each artifact for each operation is expressed as: In the formula, is the identification index of each cultural relic for the bth operation; WY is the influence of the operation on each cultural relic; is the cultural relic impact index of each cultural relic on the bth operation; It is the tourists’ own impact indicator in the bth operation.

[0058] The resource allocation module 140 is used to adjust the total amount of each resource of the AR interactive system according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the tourist, and obtain the allocation amount of each cultural relic for each skeleton point of the tourist in each operation under each resource.

[0059] In some possible implementations of the present invention, the method for obtaining the distribution amount includes: obtaining the attention of each cultural relic to each skeleton point of the tourist in each operation according to the identification index of each cultural relic for each operation, the importance of each skeleton point of the tourist in each operation, and the influence index of each skeleton point of the tourist; obtaining the total amount of each resource of the AR interaction system; using the attention degree to adjust the total amount of each resource of the AR interaction system, and obtaining the distribution amount of each cultural relic to each skeleton point of the tourist in each operation under each resource.

[0060] The influence index reflects the degree of influence of the skeleton point on the various operations of tourists; if the influence index is larger, it means that each skeleton point of tourists plays a more important role when tourists perform various operations. In order to ensure the smoothness and accuracy of the operation, each skeleton point needs to be identified more finely, and the attention paid to each skeleton point is higher. The recognition index reflects the accuracy of each operation performed by each cultural relic; if the recognition index is larger, the recognition accuracy of each cultural relic for each operation of tourists during the interaction process is more accurate. In order to improve the recognition reliability and interaction quality during the interaction process, each cultural relic needs to perform each operation more finely, and each cultural relic pays more attention to each operation. The importance reflects the importance of the skeleton point in each operation. In order to ensure the accuracy and completeness of skeleton recognition, more attention needs to be paid to the skeleton point with greater importance. Therefore, the recognition index, importance and influence index are all positively correlated with the attention.

[0061] During the interaction process, the resources of the AR interaction system need to be dynamically adjusted based on the attention of the skeleton points. The skeleton points with greater attention should be allocated more resources, thereby improving the recognition accuracy of the skeleton points, ensuring that the AR interaction system can recognize each operation more accurately and enhancing the interactive experience of tourists.

[0062] In this embodiment, the specific method for obtaining the allocation amount is: based on the attention of each cultural relic to all skeleton points of tourists in each operation, the attention of each cultural relic to each skeleton point of tourists in each operation is normalized to obtain the resource allocation weight of each cultural relic to each skeleton point of tourists in each operation; the total amount of each resource of the AR interactive system is weighted by the resource allocation weight to obtain the allocation amount of each cultural relic to each skeleton point of tourists in each operation under each resource.

[0063] The distribution of each skeleton point of each tourist for each resource in each operation is expressed by the formula: In the formula, The distribution amount of the a-th skeleton point of each cultural relic to the tourist under each resource in the b-th operation; is the attention degree of each cultural relic to the a-th skeleton point of the tourist in the b-th operation; A is the total number of skeleton points of the tourist; Assign a weight to the resources of the a-th skeleton point of each cultural relic for the tourist in the b-th operation; is the total amount of each resource of the AR interactive system; is the identification index for each cultural relic for the bth operation; is the importance of the a-th skeleton point of the tourist in the b-th operation; is the influence index of the a-th skeleton point of the tourist.

[0064] It should be noted that, in the embodiment of the present invention, the resource types of the AR interactive system include: GPU resources, CPU resources, RAM resources and network bandwidth, etc. During the tour period, when each skeleton point of the tourist performs each operation on each cultural relic, the allocation amount of each resource of the AR interactive system is obtained using the above content.

[0065] In different interaction scenarios, combined with tourists' own operating habits and cultural relics characteristics, the influence of skeleton points on tourists' various operations, and the importance of skeleton points in each operation, the allocation of each resource of the AR interaction system to each operation of skeleton points for each cultural relic is dynamically adjusted to improve the accuracy of skeleton point recognition by the AR interaction system and reduce the interference caused by other skeleton point recognition errors, thereby enhancing the tourists' interactive experience.

[0066] So far, the present invention is completed.

[0067] Embodiment 2: Figure 3 A computer device schematic diagram of a smart museum digital visual multi-dimensional virtual interactive device provided by an embodiment of the present invention. For example, Figure 3 As shown, the computer device includes: a memory 201, a processor 202, and a computer program 203 stored in the memory 201 and running on the processor 202, wherein when the processor 202 executes the computer program 203, the computer device can execute any one of the smart museum digital visual multi-dimensional virtual interaction systems introduced above.

[0068] In addition, an embodiment of the present application also protects a device, which may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to execute a smart museum digital visual multi-dimensional virtual interaction system provided by an embodiment of the present application.

[0069] In this embodiment, the functional modules of the device can be divided according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0070] It should be understood that the device provided in this embodiment is used to execute the above-mentioned smart museum digital visual multi-dimensional virtual interaction system, and thus can achieve the same effect as the above-mentioned implementation method.

[0071] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a device, the processing module may be used to control and manage the actions of the device. The storage module may be used to support the device to execute mutual program codes, etc.

[0072] The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits disclosed in the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0073] Embodiment 3: This embodiment also provides a computer-readable storage medium, which stores computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement a smart museum digital visual multi-dimensional virtual interaction system provided by the above embodiment.

[0074] Among them, the device and computer-readable storage medium provided in this embodiment are used to execute the corresponding system provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding system provided above, and will not be repeated here.

[0075] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0076] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A smart museum digital visual multi-dimensional virtual interactive system, characterized in that: The system includes: The data acquisition module is used to obtain the skeleton points of the RGB images of the hand of the visitor at each moment of each training period for each operation during the interactive learning period, the number of times the visitor uses each operation during the tour period, and the number of times each cultural relic in the museum performs each operation during the historical exhibition period; The skeleton point influence analysis module is used to obtain the operational influence of each skeleton point of the tourist in each operation according to the similarity of the movement of the same skeleton point in any two trainings within the training period of each operation and the movement of each skeleton point in each training; obtain the influence index of each skeleton point of the tourist according to the operational influence of each skeleton point of the tourist in all operations and the edge curvature of the hand RGB image at each moment of the training period of each operation; The cultural relic operation identification module is used to adjust the number of times tourists use each operation during the tour period and the number of times each cultural relic performs each operation during the historical exhibition period according to the difference in the number of times each cultural relic performs different operations during the historical exhibition period, and obtain the identification index of each cultural relic for each operation; The resource allocation module is used to adjust the total amount of each resource of the AR interactive system according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the tourist, so as to obtain the allocation amount of each cultural relic for each skeleton point of the tourist in each operation under each resource.

2. According to claim 1, a smart museum digital visual multi-dimensional virtual interactive system is characterized in that: The obtaining of the operational influence of each skeleton point of the visitor on each operation includes: Obtain the running speed and acceleration of each skeleton point of the tourist at each moment of each training in the training period of each operation; The last training in the training period of each operation is recorded as the standard training of each operation; the importance of each skeleton point of the tourist in each operation is obtained according to the movement of each skeleton point of the tourist in the standard training of each operation and the degree of acceleration fluctuation at all times; Arrange the movement speed of each skeleton point of the tourist at all moments of each training in time sequence to obtain the speed sequence of each skeleton point of the tourist in each training; form a training pair by any two trainings in the training period of each operation, obtain the average of the overall speed difference values ​​of the speed sequence of the two trainings of each skeleton point of the tourist in all training pairs of each operation, negatively correlate and normalize the average, and obtain the stability of each skeleton point of the tourist in each operation; According to the importance and the stability, the operational influence of each skeleton point of the visitor in each operation is obtained, and both the importance and the stability are positively correlated with the operational influence.

3. A smart museum digital visual multi-dimensional virtual interactive system according to claim 2, characterized in that: The obtaining of the importance of each skeleton point of the visitor in each operation includes: Obtain the hand depth images of tourists at each moment of the standard training for each operation and the duration of the standard training; The pixel points at the same position of the skeleton points of the hand RGB image at each moment in the hand depth image are recorded as the skeleton points of the hand depth image at each moment, and the skeleton points of the hand depth image are converted into point cloud data; curve fitting is performed on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training of each operation to obtain the running trajectory of each skeleton point of the tourist in the standard training of each operation; the ratio of the length of the running trajectory to the duration is used as the standard average speed of each skeleton point of the tourist in each operation; Obtain the variance of the acceleration of each skeleton point of the tourist at all moments of the standard training of each operation as the discrete value of the acceleration of each skeleton point of the tourist in each operation; The importance of each skeleton point of the visitor in each operation is obtained according to the standard average speed and the acceleration discrete value, and both the standard average speed and the acceleration discrete value are positively correlated with the importance.

4. The smart museum digital visual multi-dimensional virtual interactive system according to claim 2 is characterized in that: The obtaining of the influence index of each skeleton point of the tourist includes: The ratio of the duration of the training period to the interactive learning period of each operation is taken as the learning occupancy of each operation; According to the prominence degree of the same skeleton point in the local range of the hand RGB image at each moment of the standard training for each operation, the overall recognition degree of each skeleton point of the tourist in each operation is obtained; According to the operational influence of each skeleton point of the tourist in each operation, the overall recognition degree and the learning occupancy degree, the comprehensive influence value of each skeleton point of the tourist in each operation is obtained; the cumulative sum of the comprehensive influence values ​​of each skeleton point of the tourist in all operations is normalized to obtain the influence index of each skeleton point of the tourist.

5. The smart museum digital visual multi-dimensional virtual interactive system according to claim 4 is characterized in that: The obtaining of the overall recognition degree of each skeleton point of the visitor in each operation includes: For the hand RGB image at each moment, a circular area formed by taking each skeleton point of the hand RGB image as the center and half of the minimum distance between each skeleton point and the other skeleton points as the radius is used as the analysis area of ​​each skeleton point of the hand image; Performing corner point detection on the analysis area, counting the total number of corner points in the analysis area as the corner point amount of each skeleton point; performing edge detection on the analysis area, calculating the mean value of the curvature of all pixel points on the edge of the analysis area as the structural prominence value of each skeleton point; The product of the corner point quantity and the structural prominence value of each skeleton point is calculated as the local recognition degree of the corresponding skeleton point; the mean of the local recognition degree of the same skeleton point in the hand RGB images at all moments of the standard training of each operation is normalized to obtain the overall recognition degree of each skeleton point of the tourist in each operation.

6. The smart museum digital visual multi-dimensional virtual interactive system according to claim 1 is characterized in that: The obtaining of identification indicators of each cultural relic for each operation includes: The ratio of the number of times tourists use each operation during their visit to the total number of times they use all operations is used as the tourists' own influence indicator for each operation. The ratio of the number of times each cultural relic performs each operation to the total number of operations of all types during the historical exhibition period is used as the cultural relic impact indicator for each operation; The standard deviation of the number of times each cultural relic performs all kinds of operations during the historical exhibition period is normalized to obtain the operational influence of each cultural relic; The operation influence is used as the weight of the cultural relic influence index, the difference between the constant 1 and the operation influence is used as the weight of the self-influence index, and the self-influence index and the cultural relic influence index are weighted and summed to obtain the identification index of each cultural relic for each operation.

7. The smart museum digital visual multi-dimensional virtual interactive system according to claim 3 is characterized in that: The method of adjusting the total amount of each resource of the AR interactive system according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the tourist to obtain the distribution amount of each cultural relic for each skeleton point of the tourist in each operation under each resource includes: According to the recognition index of each cultural relic for each operation, the importance of each skeleton point of the tourist in each operation, and the influence index of each skeleton point of the tourist, the attention degree of each cultural relic to each skeleton point of the tourist in each operation is obtained; the recognition index, the importance and the influence index are all positively correlated with the attention degree; The total amount of each resource of the AR interactive system is obtained; the total amount of each resource of the AR interactive system is adjusted using the attention degree to obtain the distribution amount of each skeleton point of each cultural relic to the tourist in each operation under each resource.

8. The smart museum digital visual multi-dimensional virtual interactive system according to claim 7 is characterized in that: The method of using the attention degree to adjust the total amount of each resource of the AR interactive system to obtain the distribution amount of each skeleton point of each cultural relic to each tourist under each resource in each operation includes: Based on the attention of each cultural relic to all skeleton points of tourists in each operation, the attention of each cultural relic to each skeleton point of tourists in each operation is normalized to obtain the resource allocation weight of each cultural relic to each skeleton point of tourists in each operation; The resource allocation weight is used to weight the total amount of each resource of the AR interactive system, and the allocation amount of each cultural relic to each skeleton point of the tourist in each operation under each resource is obtained.

9. The smart museum digital visual multi-dimensional virtual interactive system according to claim 2 is characterized in that: The method for obtaining the overall speed difference value of the speed sequence of two trainings in the training pair is the DTW algorithm.

10. The smart museum digital visual multi-dimensional virtual interactive system according to claim 3 is characterized in that: The method for performing curve fitting on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training for each operation is the least squares method.

Images