Smart Museum Digital Visual Multi-dimensional Virtual Interaction System

Abstract

The present invention relates to the field of augmented reality technology, and in particular to a digital visual multi-dimensional virtual interactive system for a smart museum. The present invention obtains the operational influence of each skeleton point of a visitor in each operation according to the movement of the same skeleton point in any two trainings within the training period of each operation, and obtains the influence index of each skeleton point of the visitor in combination with the edge curvature in the hand RGB image; according to the difference in the number of times each cultural relic performs different operations, the number of times the visitor uses each operation and the number of times the cultural relic performs each operation are adjusted to obtain the recognition index of each cultural relic for each operation; the total amount of each resource of the AR interactive system is adjusted in combination with the influence index and the recognition index, and the allocation amount of each cultural relic to each skeleton point of the visitor in each operation under each resource is obtained. The present invention combines the tourists' own operating habits and cultural relic characteristics, the degree of influence of the skeleton point on the tourists' operations, dynamically adjusts the resource allocation amount, and improves the interactive effect of the tourists.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to a smart museum digital visual multi-dimensional virtual interaction system. Background Art

[0002] As crucial vehicles for the preservation of history and culture, museums are gradually moving away from traditional display methods and toward a digital and intelligent path. Smart museums not only offer realistic displays but also enable deeper interaction between visitors and historical artifacts, creating intelligent spaces that blend knowledge dissemination with cultural experience, greatly enriching the public's cultural life and learning methods.

[0003] In AR interaction scenarios within smart museums, efficient and accurate recognition of visitor movements and operational intentions in complex environments is a key issue. During AR interaction, environmental factors can cause occlusions in the camera's acquisition of visitor hand movements, leading to errors in key point recognition, response delays, and interaction errors. This can lead to inappropriate resource allocation within the AR interaction system, reducing visitor interaction effectiveness. Summary of the Invention

[0004] In order to solve the technical problem that errors in key point recognition during the interaction process lead to inappropriate allocation of system resources and reduced interaction effects, the purpose of the present invention is to provide a smart museum digital visual multi-dimensional virtual interaction system. The technical solutions adopted are as follows:

[0005] The present invention proposes a smart museum digital visual multi-dimensional virtual interaction system, which includes:

[0006] The data acquisition module is used to obtain the skeleton points of the RGB image of the hand of the visitor at each moment of each training session for each operation during the interactive learning period, the number of times the visitor used each operation during the tour period, and the number of times each operation was performed for each artifact in the museum during the historical exhibition period;

[0007] The skeleton point influence analysis module is used to obtain the operational influence of each skeleton point of the visitor on each operation based on the similarity of the movement of the same skeleton point in any two training sessions within the training period of each operation, as well as the movement of each skeleton point in each training session. The influence index of each skeleton point of the visitor is obtained based on the operational influence of each skeleton point of the visitor on all operations and the degree of edge curvature in the RGB image of the hand at each moment in the training period of each operation.

[0008] The cultural relic operation recognition module is used to adjust the number of times tourists use each operation during the tour period and the number of times each cultural relic performs each operation during the historical exhibition period according to the difference in the number of times each cultural relic performs different operations during the historical exhibition period, and obtain the recognition index of each cultural relic for each operation;

[0009] The resource allocation module is used to adjust the total amount of each resource of the AR interactive system according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the visitor, so as to obtain the allocation amount of each cultural relic for each skeleton point of the visitor in each operation under each resource.

[0010] Furthermore, obtaining the operational influence of each skeleton point of the visitor in each operation includes:

[0011] Obtain the running speed and acceleration of each skeleton point of the tourist at each moment of each training in the training period of each operation;

[0012] The last training in the training period of each operation is recorded as the standard training of each operation; the importance of each skeleton point of the tourist in each operation is obtained based on the movement of each skeleton point of the tourist in the standard training of each operation and the degree of acceleration fluctuation at all times;

[0013] Arrange the movement speeds of each skeleton point of the visitor at all moments of each training in time sequence to obtain the speed sequence of each skeleton point of the visitor in each training; form a training pair from any two training sessions in the training period of each operation, obtain the average of the overall speed difference values ​​of the speed sequence of the two training sessions of each skeleton point of the visitor in all training pairs of each operation, perform negative correlation and normalization on the average, and obtain the stability of each skeleton point of the visitor in each operation;

[0014] The operational influence of each skeleton point of the visitor in each operation is obtained according to the importance and the stability, and both the importance and the stability are positively correlated with the operational influence.

[0015] Furthermore, obtaining the importance of each skeleton point of the visitor in each operation includes:

[0016] Obtain the hand depth image of the tourist at each moment of the standard training for each operation and the duration of the standard training;

[0017] The pixel points at the same position in the hand depth image of the skeleton point of the hand RGB image at each moment are recorded as the skeleton points of the hand depth image at each moment, and the skeleton points of the hand depth image are converted into point cloud data; curve fitting is performed on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training for each operation to obtain the running trajectory of each skeleton point of the visitor in the standard training for each operation; the ratio of the length of the running trajectory to the duration is used as the standard average speed of each skeleton point of the visitor in each operation;

[0018] Obtain the variance of the acceleration of each skeleton point of the tourist at all moments of standard training for each operation as the discrete value of the acceleration of each skeleton point of the tourist in each operation;

[0019] The importance of each skeleton point of the visitor in each operation is obtained according to the standard average speed and the acceleration discrete value, and the standard average speed and the acceleration discrete value are both positively correlated with the importance.

[0020] Furthermore, the obtaining of the influence index of each skeleton point of the tourist includes:

[0021] The ratio of the training period to the interactive learning period of each operation is used as the learning occupancy of each operation;

[0022] Obtain the overall recognition degree of each skeleton point of the visitor in each operation based on the prominence degree of the same skeleton point in the local range of the hand RGB image at each moment of the standard training for each operation;

[0023] According to the operational influence of each skeleton point of the tourist in each operation, the overall recognition degree and the learning occupancy, the comprehensive influence value of each skeleton point of the tourist in each operation is obtained; the cumulative sum of the comprehensive influence values ​​of each skeleton point of the tourist in all operations is normalized to obtain the influence index of each skeleton point of the tourist.

[0024] Furthermore, obtaining the overall recognition degree of each skeleton point of the visitor in each operation includes:

[0025] For each moment of the hand RGB image, the analysis area for each skeleton point in the hand RGB image is a circular area with each skeleton point in the hand RGB image as the center and half of the minimum distance between each skeleton point and the other skeleton points as the radius;

[0026] Performing corner detection on the analysis area, counting the total number of corner points in the analysis area as the corner point quantity of each skeleton point; performing edge detection on the analysis area, calculating the mean curvature of all pixel points on the edge of the analysis area as the structural prominence value of each skeleton point;

[0027] The product of the corner point quantity and the structural prominence value of each skeleton point is calculated as the local recognition degree of the corresponding skeleton point; the mean of the local recognition degree of the same skeleton point in the hand RGB images at all moments of the standard training for each operation is normalized to obtain the overall recognition degree of each skeleton point of the visitor in each operation.

[0028] Furthermore, obtaining the identification index of each cultural relic for each operation includes:

[0029] The ratio of the number of times tourists use each operation during their visit to the total number of all operations is used as the tourists' own influence indicator for each operation;

[0030] The ratio of the number of times each artifact performs each operation to the total number of times it performs all operations during the historical exhibition period is used as the artifact impact indicator for each operation.

[0031] Normalize the standard deviation of the number of times each artifact performs all kinds of operations during the historical exhibition period to obtain the operational impact of each artifact;

[0032] The operation influence is used as the weight of the cultural relic influence index, the difference between a constant 1 and the operation influence is used as the weight of the self-influence index, and the self-influence index and the cultural relic influence index are weighted and summed to obtain the identification index of each cultural relic for each operation.

[0033] Furthermore, the total amount of each resource of the AR interactive system is adjusted according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the visitor, to obtain the allocation amount of each cultural relic for each skeleton point of the visitor in each operation under each resource, including:

[0034] Obtain the attention level of each cultural relic to each visitor's skeleton point in each operation based on the recognition index of each cultural relic for each operation, the importance of each visitor's skeleton point in each operation, and the influence index of each visitor's skeleton point; the recognition index, the importance, and the influence index are all positively correlated with the attention level;

[0035] The total amount of each resource of the AR interactive system is obtained; the total amount of each resource of the AR interactive system is adjusted using the attention level to obtain the distribution amount of each skeleton point of each cultural relic to each visitor under each resource in each operation.

[0036] Furthermore, the use of the attention level to adjust the total amount of each resource of the AR interactive system to obtain the distribution amount of each skeleton point of each tourist under each resource in each operation includes:

[0037] Based on the attention of each cultural relic to all skeleton points of tourists in each operation, the attention of each cultural relic to each skeleton point of tourists in each operation is normalized to obtain the resource allocation weight of each cultural relic to each skeleton point of tourists in each operation;

[0038] The resource allocation weight is used to weight the total amount of each resource of the AR interactive system to obtain the allocation amount of each cultural relic to each skeleton point of the visitor in each operation under each resource.

[0039] Furthermore, a method for obtaining the overall speed difference value of the speed sequences of two trainings in the training pair is a DTW algorithm.

[0040] Furthermore, the method for performing curve fitting on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training for each operation is the least squares method.

[0041] The present invention has the following beneficial effects:

[0042] In an embodiment of the present invention, there are differences in the operating habits of tourists, which may result in a low matching degree of certain skeleton points during the training period, resulting in poor recognition effect of the AR device for each operation. The similarity of the movement of the same skeleton point in any two training sessions of each operation and the movement of the skeleton point are used to analyze the influence of the skeleton point in each operation to obtain the operation influence. In combination with the image features within the local range of the skeleton point, the influence index of the skeleton point is analyzed to measure the influence of the skeleton point on various operations of tourists. The number of times tourists use each operation during the tour period reflects the operating habits of tourists, and the number of times cultural relics perform each operation during the historical exhibition period reflects the characteristics of cultural relics. The recognition index of cultural relics for each operation is analyzed in combination with the operating habits of tourists and the characteristics of cultural relics to measure the accuracy of each cultural relic performing each operation. According to the recognition index and the influence index, each resource of the AR interactive system is dynamically allocated and adjusted to make the resource allocation of the AR interactive system more appropriate, reduce the response delay and interaction error caused by the recognition errors of other skeleton points, and improve the interaction effect of tourists. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A system structure diagram of a digital visual multi-dimensional virtual interactive system for a smart museum provided by one embodiment of the present invention;

[0045] Figure 2 A structural diagram of a skeleton point influence analysis module provided by one embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a computer device for a smart museum digital visual multi-dimensional virtual interactive device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0047] To further illustrate the technical means and effectiveness of the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a smart museum digital visual multi-dimensional virtual interactive system proposed by the present invention. In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0048] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0049] The following describes in detail a specific solution of a smart museum digital visual multi-dimensional virtual interaction system provided by the present invention with reference to the accompanying drawings.

[0050] Example 1:

[0051] See also Figure 1 , which shows a system block diagram of a smart museum digital visual multi-dimensional virtual interaction system provided by an embodiment of the present invention. The system includes: a data acquisition module 110, a skeleton point influence analysis module 120, a cultural relic operation recognition module 130, and a resource allocation module 140.

[0052] The data acquisition module 110 is used to obtain the skeleton points of the RGB image of the hand of the visitor at each moment of each training period of each operation during the interactive learning period, the number of times the visitor uses each operation during the tour period, and the number of times each operation is performed on each cultural relic in the museum during the historical exhibition period.

[0053] To allow visitors to get a closer look at the museum's cultural relics and enhance their experience, when they enter designated viewing areas, staff will provide them with augmented reality (AR) equipment. The AR equipment can render 3D models of the cultural relics on display in the viewing area, allowing visitors to interact and observe the rendered models through gestures.

[0054] Visitors need to wear the AR device on their heads. The AR device is equipped with an RGB camera and a depth camera to capture the movements of the visitors' hands and the spatial depth. The images captured by the RGB camera and the depth camera are recorded as hand RGB images and hand depth images respectively.

[0055] Before using AR devices to observe cultural relics, visitors need to learn simple interactive operations according to the operating prompts of the AR devices. The time period during which visitors learn interactive operations is recorded as the interactive learning period, and the time period during which visitors learn each interactive operation is recorded as the training period for each operation. The learning process of interactive operations is as follows:

[0056] First, the AR device guides visitors to perform simple gestures, such as clicking, grabbing, and rotating, through simple animation examples. Then, visitors imitate the gestures in the animation, and use the RGB camera and depth camera to obtain the RGB image and depth image of the visitor's hand at each moment during the interactive learning period. The two images are combined and an open posture estimation algorithm is used to obtain the skeleton points of the hand RGB image at each moment. Finally, the hand RGB image of each training for each operation is matched with the skeleton points of the animation. When the match is successful, the next interactive operation is guided until all interactive operations provided by the AR device are successfully matched.

[0057] Each training session requires a certain amount of time, that is, each training session has multiple RGB images of the hand at multiple moments, and the animation of each operation contains multiple frames of template images. The matching method is as follows: the RGB images of the hand at all moments in the time period of each training session of each operation and the multiple frames of template images of the animation are recorded as the to-be-matched images and the standard images in turn, and the scale-invariant feature transformation algorithm is used to match the skeleton points of each to-be-matched image with those of each standard image to obtain matching pairs of the skeleton points of each to-be-matched image and each standard image; the mean of the Euclidean distance between the two skeleton points in all matching pairs of the skeleton points of each to-be-matched image and each standard image is normalized to obtain the matching index between each to-be-matched image and each standard image; if the matching index between all the to-be-matched images and any one of the standard images is greater than the preset matching threshold, then the hand RGB images and the skeleton points of the animation are successfully matched within the time period of each training session of each operation; otherwise, the matching fails.

[0058] It should be noted that in this embodiment of the present invention, an open pose estimation algorithm is used to obtain skeleton points in the animation template image; the preset matching threshold is an empirical value of 0.9, which can be set by the implementer according to specific circumstances; the Norm function is used for normalization processing; and the image acquisition frequency of the RGB camera and the depth camera is once every 0.1 seconds, which can be set by the implementer according to specific circumstances. The open pose estimation algorithm and the scale-invariant feature transformation algorithm are well known to those skilled in the art and will not be described in detail here.

[0059] When all the visitor's operations are successfully matched, the visitor can operate and interact with the cultural relics exhibited in the museum through all the operations learned above and observe the rendered cultural relic models. During this process, the skeleton points of the visitor's hand are continuously tracked, and each skeleton point of the visitor's hand can be recognized at all times in the process.

[0060] The time period between the last moment of the interactive learning period and the current moment is recorded as the tourist's tour period, and the number of times the tourist uses each operation during the tour period is counted; the week before the current day is regarded as the historical exhibition period, and tourists perform various operations on the three-dimensional model of the cultural relics through the AR devices they wear. The cultural relics perform each operation to facilitate tourists to understand the cultural relic information, and the number of times each cultural relic performs each operation during the historical exhibition period is counted.

[0061] AR interactions in smart museums primarily rely on matching the visitor's skeleton points with those in their virtual rendering model. During the interaction, the AR device's capture device captures the visitor's hand movements and converts them into model renderings. Poor AR device recognition accuracy and latency in the rendered model's response significantly impact the visitor's interactive experience. Therefore, this solution dynamically adjusts the AR interaction system's resource allocation to skeleton points to optimize interaction.

[0062] The skeleton point influence analysis module 120 is used to obtain the operational influence of each skeleton point of the visitor in each operation based on the similarity of the movement of the same skeleton point in any two training sessions within the training period of each operation, as well as the movement of each skeleton point in each training session; and to obtain the influence index of each skeleton point of the visitor based on the operational influence of each skeleton point of the visitor in all operations and the edge curvature degree of the hand RGB image at each moment in the training period of each operation.

[0063] See also Figure 2 , which shows a structural diagram of a skeleton point influence analysis module provided by an embodiment of the present invention. The skeleton point influence analysis module includes: an importance analysis unit 121, an influence analysis unit 122, and an influence index analysis unit 123.

[0064] The importance analysis unit 121 is used to obtain the running speed and acceleration of each skeleton point of the tourist at each moment of each training in the training period of each operation; record the last training in the training period of each operation as the standard training of each operation; and obtain the importance of each skeleton point of the tourist in each operation based on the movement conditions of each skeleton point of the tourist in the standard training of each operation and the degree of acceleration fluctuation at all moments.

[0065] Inertial sensors are installed at each joint of the visitor's hand to measure the velocity and acceleration of each skeletal point at each moment during the interactive learning period. It should be noted that the inertial sensors and the RGB camera have the same data acquisition frequency, and the velocity refers to the angular velocity.

[0066] Preferably, in some possible implementation methods of the embodiments of the present invention, the method for obtaining the importance includes: obtaining the hand depth image of the tourist at each moment of the standard training of each operation and the duration of the standard training; recording the pixel points at the same position in the hand depth image of the skeleton points of the hand RGB image at each moment as the skeleton points of the hand depth image at each moment, and converting the skeleton points of the hand depth image into point cloud data; performing curve fitting on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training of each operation to obtain the running trajectory of each skeleton point of the tourist in the standard training of each operation; taking the ratio of the length of the running trajectory to the duration as the standard average speed of each skeleton point of the tourist in each operation; obtaining the variance of the acceleration of each skeleton point of the tourist at all moments of the standard training of each operation as the acceleration discrete value of each skeleton point of the tourist in each operation; and obtaining the importance of each skeleton point of the tourist in each operation based on the standard average speed and the acceleration discrete value.

[0067] Different skeleton points of tourists have different degrees of influence on different operations. For example, the grasping operation is mainly identified by identifying the relative movement of fingers and palms. The palm is basically motionless, and the fingers have a larger range of movement. Compared with the palm, the skeleton points at the finger position move faster and the acceleration fluctuation is more obvious. Therefore, the skeleton points at the finger position are more important in the grasping operation.

[0068] When using each operation, visitors need to determine the trajectory of each skeleton point. The faster the skeleton point's trajectory and the more significant the fluctuations in acceleration at all times during each operation, the more significant the angle changes at the joint corresponding to the skeleton point, indicating that the skeleton point is more important in each operation. The standard average speed is the average speed of the skeleton point's trajectory during each operation. Therefore, the standard average speed is positively correlated with the acceleration dispersion and importance.

[0069] In the embodiment of the present invention, the product of the standard average speed of each skeleton point of the visitor in each operation and the acceleration discrete value is normalized to obtain the importance of each skeleton point of the visitor in each operation.

[0070] It should be noted that while the Norm function is used for normalization in the embodiments of the present invention, other normalization methods, such as function transformation, maximum-minimum normalization, and other normalization methods, may also be used, and are not limited here. The least squares method is used to perform curve fitting on the point cloud data of the same skeleton point in the hand depth image at all moments of standard training for each operation. The least squares method and the method for converting pixels in the depth image into point cloud data are both well known to those skilled in the art and will not be described in detail here.

[0071] The influence analysis unit 122 is used to obtain the operational influence of each skeleton point of the visitor in each operation.

[0072] Preferably, in some possible implementations of the embodiments of the present invention, the method for obtaining operational influence includes: arranging the movement speed of each skeleton point of the tourist at all moments of each training in time sequence to obtain the speed sequence of each skeleton point of the tourist in each training; forming a training pair by any two trainings in the training period of each operation, obtaining the mean of the overall speed difference values ​​of the speed sequences of the two trainings of each skeleton point of the tourist in all training pairs of each operation, negatively correlating and normalizing the mean, and obtaining the stability of each skeleton point of the tourist in each operation; obtaining the operational influence of each skeleton point of the tourist in each operation based on the importance and stability, and the importance and stability are both positively correlated with the operational influence.

[0073] Due to differences in tourist behavior, the matching degree of some skeleton points during the training period may be low, resulting in poor AR device recognition of each action. To address this issue, a thorough analysis of the matching of a tourist's skeleton points across multiple training sessions is required. The more similar the skeleton point's motion during each training session, and the more significant the actual motion of the skeleton point during the visit, the more influential the skeleton point is in each action, and the greater the influence of the action.

[0074] Stability reflects the similarity of skeleton point motion across different training sessions within a training period. A higher stability indicates more similar skeleton point motion across different training sessions, more stable skeleton point motion during the tour period, and a higher likelihood of successful matching within that period, making that skeleton point more influential in each operation. Importance analyzes the importance of a skeleton point in each operation based on the motion of the last training session within each operation's training period. Standard training presents the actual motion of a tourist's skeleton point during the tour period. A higher importance indicates a more important actual motion of a skeleton point during the tour period, and a more influential skeleton point in each operation. Therefore, both importance and stability are positively correlated with operation influence.

[0075] It should be noted that this embodiment uses a dynamic time warping (DTW) algorithm to obtain DTW values ​​of the two velocity sequences. The DTW algorithm is well known to those skilled in the art and will not be described in detail here.

[0076] In a specific implementation of the embodiment of the present invention, the operational influence is expressed by the formula:

[0077]

[0078] Where, The influence of the a-th skeleton point of the tourist on the b-th operation; is the DTW value of the velocity sequence of the tourist's a-th skeleton point in the m-th training pair of the b-th operation; For all training pairs of tourists operating in type b; is the stability of the tourist's a-th skeleton point in the b-th operation; is the importance of the a-th skeleton point of the tourist in the b-th operation; exp is an exponential function with a natural constant as the base. It should be noted that this embodiment uses the exp function to Perform negative correlation and normalization.

[0079] The influence index analysis unit 123 is used to obtain the influence index of each skeleton point of the tourist.

[0080] In an embodiment of the present invention, a method for obtaining an influence index is as follows: taking the ratio of the duration of the training period to the interactive learning period of each operation as the learning occupancy of each operation; obtaining the overall recognition of each skeleton point of the visitor in each operation based on the prominence of the same skeleton point in the local range of the hand RGB image at each moment of the standard training of each operation; obtaining the comprehensive influence value of each skeleton point of the visitor in each operation based on the operational influence, overall recognition and learning occupancy of each skeleton point of the visitor in each operation; and normalizing the cumulative sum of the comprehensive influence values ​​of each skeleton point of the visitor in all operations to obtain the influence index of each skeleton point of the visitor.

[0081] Environmental factors may cause occlusion when the camera captures tourist hand movements. During the monitoring of skeleton points during the tour period, some skeleton points may be lost or mismatched, thus interfering with the matching of each operation. This embodiment analyzes the overall recognition of skeleton points by analyzing the image features of the local range of the skeleton points. The specific method for obtaining the overall recognition is as follows:

[0082] For the RGB image of the hand at each moment, a circular area formed by taking each skeleton point of the hand RGB image as the center and half of the minimum distance between each skeleton point and the other skeleton points as the radius is used as the analysis area for each skeleton point in the hand image; corner point detection is performed on the analysis area, and the total number of corner points in the analysis area is counted as the corner point quantity of each skeleton point; edge detection is performed on the analysis area, and the mean curvature of the pixel points on all edges in the analysis area is calculated as the structural prominence value of each skeleton point; the product of the corner point quantity and the structural prominence value of each skeleton point is calculated as the local recognition degree of the corresponding skeleton point; the mean of the local recognition degree of the same skeleton point in the hand RGB images of all moments of standard training for each operation is normalized to obtain the overall recognition degree of each skeleton point of the tourist in each operation.

[0083] This embodiment analyzes the difficulty of identifying skeleton points by the number of corner points and edge curvature characteristics within the analysis area of ​​the skeleton points. If the number of corner points and edge curvature within the analysis area of ​​the skeleton point is larger, it means that the skeleton point is more prominent in the image compared to other skeleton points, and the probability of matching failure of the skeleton point is smaller, then the skeleton point is easier to identify. The structural prominence value reflects the edge curvature within the analysis area of ​​the skeleton point. The larger the structural prominence value, the greater the edge curvature within the analysis area of ​​the skeleton point. Therefore, the product of the number of corner points of the skeleton point and the structural prominence value is used as the local recognition degree to analyze the difficulty of the skeleton point being identified in the hand RGB image.

[0084] It should be noted that this embodiment uses the Harris corner extraction algorithm and the Canny operator to perform corner detection and edge detection on the hand RGB image in sequence. The Harris corner extraction algorithm and the Canny operator are well known to those skilled in the art and will not be described in detail here.

[0085] The greater the operational influence, the higher the accuracy of identifying each tourist operation using skeleton points during the visit, indicating that the skeleton point has a greater influence on each operation, and thus, the greater the influence of the skeleton point on each operation. A higher overall recognition degree indicates that each skeleton point is easier to identify for each operation, and the more precise the skeleton point's position during the visit, the higher the accuracy of the skeleton point's correct identification for each operation, and thus, the greater the influence of the skeleton point on each operation. A higher learning occupancy degree indicates that the longer the training time for each operation, the higher the accuracy of tourists using each operation during the visit. Therefore, operational influence, overall recognition degree, and learning occupancy are all positively correlated with the influence indicator.

[0086] In a specific implementation of the embodiment of the present invention, the influence index of each skeleton point of a visitor is expressed as follows:

[0087]

[0088]

[0089] Where, is the influence index of the a-th skeleton point of the tourist; B is the number of types of operations; The influence of the a-th skeleton point of the tourist on the b-th operation; is the overall recognition degree of the a-th skeleton point of the tourist in the b-th operation; is the learning occupancy of the b-th operation; The total number of moments in the standard training period for type b operation; is the structural prominence value of the a-th skeleton point of the tourist in the n-th moment of the standard training of the b-th operation in the RGB image of the hand; is the local recognition degree of the tourist's a-th skeleton point in the n-th moment of the standard training for the b-th operation in the hand RGB image; is the comprehensive influence value of the a-th skeleton point of the tourist in the b-th operation; Norm is the normalization function.

[0090] The cultural relic operation identification module 130 is used to adjust the number of times tourists use each operation during the tour period and the number of times each cultural relic performs each operation during the historical exhibition period according to the difference in the number of times each cultural relic performs different operations during the historical exhibition period, and obtain the identification index of each cultural relic for each operation.

[0091] During a visit, the order and method of how visitors manipulate different artifacts are influenced and influenced by their habits and the artifact's characteristics. Visitors will develop habits when interacting with AR devices. For example, if a visitor is more accustomed to detailed observation, the AR interaction system will need to pay more attention to "view" or "rotate" operations on artifacts and less attention to other operations. The characteristics of artifacts also influence visitors' actions. For example, visitors may prioritize zooming in on exquisite artifacts to see their details, which will often require grabbing. Visitors may also prioritize observing artwork or historical exhibits from different angles, which will often require rotating artworks.

[0092] The difference in the number of times cultural relics perform different operations during the historical exhibition period reflects the greater the degree of influence of cultural relics on tourists' own operations. The number of times tourists use each operation during the tour period reflects their operating habits, and the number of times cultural relics perform each operation during the historical exhibition period reflects the characteristics of cultural relics. Combining the above factors, the identification index of each cultural relic for each operation is obtained.

[0093] Preferably, in some possible implementation methods of the embodiments of the present invention, the method for obtaining the identification index includes: taking the ratio of the number of times a tourist uses each operation during the tour period to the total number of times all types of operations are used as the tourist's own influence index for each operation; taking the ratio of the number of times each cultural relic performs each operation during the historical exhibition period to the total number of times all types of operations are performed as the cultural relic influence index for each operation; normalizing the standard deviation of the number of times each cultural relic performs all types of operations during the historical exhibition period to obtain the operation influence of each cultural relic; taking the operation influence as the weight of the cultural relic influence index, and the difference between the constant 1 and the operation influence as the weight of the self-influence index, performing weighted summation on the self-influence index and the cultural relic influence index to obtain the identification index of each cultural relic for each operation.

[0094] A greater degree of operational influence indicates a greater variance in the number of different operations performed on each artifact during the historical exhibition period. This suggests that artifact characteristics and the operations performed on artifacts vary significantly between visitors, indicating a greater degree of influence on the visitor's own operations. The self-influence indicator reflects the visitor's own habits, while the artifact influence indicator reflects the artifact's characteristics. A greater degree of operational influence indicates a greater degree of influence on the visitor's own operations, indicating that artifact characteristics have a more significant impact on visitor operations during interaction. Paying more attention to artifact influence indicators can help improve the recognition accuracy of each visitor operation during interaction. Conversely, the greater the influence of visitors' operational habits on artifact operations, the greater the degree of influence. Paying more attention to self-influence indicators can help improve the recognition accuracy of each visitor operation during interaction.

[0095] The identification index of each artifact for each operation is expressed as follows:

[0096]

[0097] Where, is the identification index of each cultural relic for the bth operation; WY is the influence of the operation on each cultural relic; is the cultural relic impact index of each cultural relic on the b-th operation; It is the self-influence indicator of tourists in the b-th operation.

[0098] The resource allocation module 140 is used to adjust the total amount of each resource of the AR interactive system according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the visitor, and obtain the allocation amount of each cultural relic for each skeleton point of the visitor in each operation under each resource.

[0099] In some possible implementations of the present invention, the method for obtaining the distribution amount includes: obtaining the attention of each cultural relic to each skeleton point of the tourist in each operation based on the identification index of each cultural relic for each operation, the importance of each skeleton point of the tourist in each operation, and the influence index of each skeleton point of the tourist; obtaining the total amount of each resource of the AR interaction system; using the attention degree to adjust the total amount of each resource of the AR interaction system, and obtaining the distribution amount of each cultural relic to each skeleton point of the tourist in each operation under each resource.

[0100] The influence index reflects the degree of influence of a skeleton point on various visitor operations. A larger influence index indicates that each skeleton point plays a more important role in the visitor's operations. To ensure smooth and accurate operations, more precise identification of each skeleton point is required, resulting in a higher level of attention to each skeleton point. The recognition index reflects the accuracy of each operation performed by each artifact. A larger recognition index indicates that each artifact recognizes each visitor's operation more accurately during the interaction process. To improve recognition reliability and interaction quality, each artifact needs to perform each operation with greater precision, resulting in a higher level of attention to each operation. The importance index reflects the importance of a skeleton point in each operation. To ensure the accuracy and completeness of skeleton recognition, skeleton points with greater importance are given more attention. Therefore, the recognition index, importance, and influence index are all positively correlated with attention.

[0101] During the interaction process, the resources of the AR interaction system need to be dynamically adjusted based on the attention of the skeleton points. Skeleton points with greater attention should be allocated more resources, thereby improving the recognition accuracy of the skeleton points, ensuring that the AR interaction system can recognize each operation more accurately and enhancing the interactive experience of tourists.

[0102] In this embodiment, the specific method for obtaining the allocation amount is: based on the attention of each cultural relic to all skeleton points of tourists in each operation, the attention of each cultural relic to each skeleton point of tourists in each operation is normalized to obtain the resource allocation weight of each cultural relic to each skeleton point of tourists in each operation; the total amount of each resource of the AR interactive system is weighted by using the resource allocation weight to obtain the allocation amount of each cultural relic to each skeleton point of tourists in each operation under each resource.

[0103] The distribution of each skeleton point of each tourist for each resource and each cultural relic in each operation is expressed as follows:

[0104]

[0105]

[0106] Where, The distribution amount of the a-th skeleton point of each cultural relic to the tourist under each resource in the b-th operation; is the attention degree of each cultural relic to the a-th skeleton point of the tourist in the b-th operation; A is the total number of skeleton points of the tourist; Assign a weight to the resources of the a-th skeleton point of each cultural relic to the tourist in the b-th operation; is the total amount of each resource of the AR interaction system; is the identification index of the bth operation for each artifact; is the importance of the a-th skeleton point of the tourist in the b-th operation; is the influence index of the a-th skeleton point of the tourist.

[0107] It should be noted that in this embodiment of the present invention, the resource types of the AR interaction system include: GPU resources, CPU resources, RAM resources, and network bandwidth. During the tour period, when each skeleton point of the visitor performs each operation on each cultural relic, the allocation amount of each resource of the AR interaction system is obtained using the above content.

[0108] In different interactive scenarios, combined with tourists' own operating habits and cultural relics' characteristics, the influence of skeleton points on tourists' various operations, and the importance of skeleton points in each operation, the allocation of each resource of the AR interactive system to each skeleton point in each operation for each cultural relic is dynamically adjusted to improve the accuracy of the AR interactive system in skeleton point recognition, reduce the interference caused by errors in the recognition of other skeleton points, and thus enhance the tourists' interactive experience.

[0109] So far, the present invention is completed.

[0110] Example 2:

[0111] Figure 3 A schematic diagram of a computer device for a digital visual multi-dimensional virtual interactive device for a smart museum provided by an embodiment of the present invention. For example, Figure 3 As shown, the computer device includes: a memory 201, a processor 202, and a computer program 203 stored in the memory 201 and running on the processor 202, wherein when the processor 202 executes the computer program 203, the computer device can execute any one of the smart museum digital visual multi-dimensional virtual interaction systems introduced above.

[0112] In addition, an embodiment of the present application also protects a device, which may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to execute a smart museum digital visual multi-dimensional virtual interaction system provided by an embodiment of the present application.

[0113] In this embodiment, the device can be divided into functional modules based on the above-described method examples. For example, each functional module can be mapped to a specific functional module, or two or more functions can be integrated into a single processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and represents only a logical functional division. In actual implementation, other division methods may be used.

[0114] It should be understood that the device provided in this embodiment is used to execute the above-mentioned smart museum digital visual multi-dimensional virtual interaction system, and thus can achieve the same effect as the above-mentioned implementation method.

[0115] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a device, the processing module may be used to control and manage the operation of the device. The storage module may be used to support the device in executing mutual program codes, etc.

[0116] The processing module may be a processor or controller that implements or executes the various exemplary logic blocks, modules, and circuits disclosed herein. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processing (DSP) and a microprocessor, and the storage module may be a memory.

[0117] Example 3:

[0118] This embodiment also provides a computer-readable storage medium, which stores computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement a smart museum digital visual multi-dimensional virtual interaction system provided by the above embodiment.

[0119] Among them, the device and computer-readable storage medium provided in this embodiment are used to execute the corresponding system provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding system provided above, and will not be repeated here.

[0120] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0121] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A smart museum digital visual multi-dimensional virtual interactive system, characterized in that: The system includes: The data acquisition module is used to obtain the skeleton points of the RGB images of the hand of the visitor at each moment of each training period for each operation during the interactive learning period, the number of times the visitor uses each operation during the tour period, and the number of times each cultural relic in the museum performs each operation during the historical exhibition period; The skeleton point influence analysis module is used to obtain the operational influence of each skeleton point of the tourist in each operation according to the similarity of the movement of the same skeleton point in any two trainings within the training period of each operation and the movement of each skeleton point in each training; obtain the influence index of each skeleton point of the tourist according to the operational influence of each skeleton point of the tourist in all operations and the edge curvature of the hand RGB image at each moment of the training period of each operation; The cultural relic operation identification module is used to adjust the number of times tourists use each operation during the tour period and the number of times each cultural relic performs each operation during the historical exhibition period according to the difference in the number of times each cultural relic performs different operations during the historical exhibition period, and obtain the identification index of each cultural relic for each operation; The resource allocation module is used to adjust the total amount of each resource of the AR interactive system according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the tourist, so as to obtain the allocation amount of each cultural relic for each skeleton point of the tourist in each operation under each resource.

2. According to claim 1, a smart museum digital visual multi-dimensional virtual interactive system is characterized in that: The obtaining of the operational influence of each skeleton point of the visitor on each operation includes: Obtain the running speed and acceleration of each skeleton point of the tourist at each moment of each training in the training period of each operation; The last training in the training period of each operation is recorded as the standard training of each operation; the importance of each skeleton point of the tourist in each operation is obtained according to the movement of each skeleton point of the tourist in the standard training of each operation and the degree of acceleration fluctuation at all times; Arrange the movement speed of each skeleton point of the tourist at all moments of each training in time sequence to obtain the speed sequence of each skeleton point of the tourist in each training; form a training pair by any two trainings in the training period of each operation, obtain the average of the overall speed difference values ​​of the speed sequence of the two trainings of each skeleton point of the tourist in all training pairs of each operation, negatively correlate and normalize the average, and obtain the stability of each skeleton point of the tourist in each operation; According to the importance and the stability, the operational influence of each skeleton point of the visitor in each operation is obtained, and both the importance and the stability are positively correlated with the operational influence.

3. A smart museum digital visual multi-dimensional virtual interactive system according to claim 2, characterized in that: The obtaining of the importance of each skeleton point of the visitor in each operation includes: Obtain the hand depth images of tourists at each moment of the standard training for each operation and the duration of the standard training; The pixel points at the same position of the skeleton points of the hand RGB image at each moment in the hand depth image are recorded as the skeleton points of the hand depth image at each moment, and the skeleton points of the hand depth image are converted into point cloud data; curve fitting is performed on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training of each operation to obtain the running trajectory of each skeleton point of the tourist in the standard training of each operation; the ratio of the length of the running trajectory to the duration is used as the standard average speed of each skeleton point of the tourist in each operation; Obtain the variance of the acceleration of each skeleton point of the tourist at all moments of the standard training of each operation as the discrete value of the acceleration of each skeleton point of the tourist in each operation; The importance of each skeleton point of the visitor in each operation is obtained according to the standard average speed and the acceleration discrete value, and both the standard average speed and the acceleration discrete value are positively correlated with the importance.

4. The smart museum digital visual multi-dimensional virtual interactive system according to claim 2 is characterized in that: The obtaining of the influence index of each skeleton point of the tourist includes: The ratio of the duration of the training period to the interactive learning period of each operation is taken as the learning occupancy of each operation; According to the prominence degree of the same skeleton point in the local range of the hand RGB image at each moment of the standard training for each operation, the overall recognition degree of each skeleton point of the tourist in each operation is obtained; According to the operational influence of each skeleton point of the tourist in each operation, the overall recognition degree and the learning occupancy degree, the comprehensive influence value of each skeleton point of the tourist in each operation is obtained; the cumulative sum of the comprehensive influence values ​​of each skeleton point of the tourist in all operations is normalized to obtain the influence index of each skeleton point of the tourist.

5. The smart museum digital visual multi-dimensional virtual interactive system according to claim 4 is characterized in that: The obtaining of the overall recognition degree of each skeleton point of the visitor in each operation includes: For the hand RGB image at each moment, a circular area formed by taking each skeleton point of the hand RGB image as the center and half of the minimum distance between each skeleton point and the other skeleton points as the radius is used as the analysis area of ​​each skeleton point of the hand image; Performing corner point detection on the analysis area, counting the total number of corner points in the analysis area as the corner point amount of each skeleton point; performing edge detection on the analysis area, calculating the mean value of the curvature of all pixel points on the edge of the analysis area as the structural prominence value of each skeleton point; The product of the corner point quantity and the structural prominence value of each skeleton point is calculated as the local recognition degree of the corresponding skeleton point; the mean of the local recognition degree of the same skeleton point in the hand RGB images at all moments of the standard training of each operation is normalized to obtain the overall recognition degree of each skeleton point of the tourist in each operation.

6. The smart museum digital visual multi-dimensional virtual interactive system according to claim 1 is characterized in that: The obtaining of identification indicators of each cultural relic for each operation includes: The ratio of the number of times tourists use each operation during their visit to the total number of times they use all operations is used as the tourists' own influence indicator for each operation. The ratio of the number of times each cultural relic performs each operation to the total number of operations of all types during the historical exhibition period is used as the cultural relic impact indicator for each operation; The standard deviation of the number of times each cultural relic performs all kinds of operations during the historical exhibition period is normalized to obtain the operational influence of each cultural relic; The operation influence is used as the weight of the cultural relic influence index, the difference between the constant 1 and the operation influence is used as the weight of the self-influence index, and the self-influence index and the cultural relic influence index are weighted and summed to obtain the identification index of each cultural relic for each operation.

7. The smart museum digital visual multi-dimensional virtual interactive system according to claim 3 is characterized in that: The method of adjusting the total amount of each resource of the AR interactive system according to the recognition index of each cultural relic for each operation and the influence index of each skeleton point of the tourist to obtain the distribution amount of each cultural relic for each skeleton point of the tourist in each operation under each resource includes: According to the recognition index of each cultural relic for each operation, the importance of each skeleton point of the tourist in each operation, and the influence index of each skeleton point of the tourist, the attention degree of each cultural relic to each skeleton point of the tourist in each operation is obtained; the recognition index, the importance and the influence index are all positively correlated with the attention degree; The total amount of each resource of the AR interactive system is obtained; the total amount of each resource of the AR interactive system is adjusted using the attention degree to obtain the distribution amount of each skeleton point of each cultural relic to the tourist in each operation under each resource.

8. The smart museum digital visual multi-dimensional virtual interactive system according to claim 7 is characterized in that: The method of using the attention degree to adjust the total amount of each resource of the AR interactive system to obtain the distribution amount of each skeleton point of each cultural relic to each tourist under each resource in each operation includes: Based on the attention of each cultural relic to all skeleton points of tourists in each operation, the attention of each cultural relic to each skeleton point of tourists in each operation is normalized to obtain the resource allocation weight of each cultural relic to each skeleton point of tourists in each operation; The resource allocation weight is used to weight the total amount of each resource of the AR interactive system, and the allocation amount of each cultural relic to each skeleton point of the tourist in each operation under each resource is obtained.

9. The smart museum digital visual multi-dimensional virtual interactive system according to claim 2 is characterized in that: The method for obtaining the overall speed difference value of the speed sequence of two trainings in the training pair is the DTW algorithm.

10. The smart museum digital visual multi-dimensional virtual interactive system according to claim 3 is characterized in that: The method for performing curve fitting on the point cloud data of the same skeleton point in the hand depth image at all moments of the standard training for each operation is the least squares method.

Images