A museum artifact AR interaction method, system, device and storage medium

Abstract

The application discloses a kind of museum cultural relics AR interaction method, system, equipment and storage medium, it is related to the field of augmented reality and wisdom museum technical field, including collecting cultural relic image information in museum and carrying out three-dimensional reconstruction, obtain virtual cultural relic model;Obtain the photo of cultural relic in museum, call the virtual cultural relic model corresponding to the photo of cultural relic;Real scene is collected, and real scene plane is identified, and virtual cultural relic model is placed on the plane identified, realizes AR placement;Obtain user hand key point information, determine user hand posture information, and display the interactive information of corresponding gesture.The application places virtual cultural relic model on the real scene plane identified in combination with plane detection algorithm, improves the performance of plane detection, promotes the accuracy of virtual cultural relic when placing, better realizes the degree of fusion of virtual cultural relic and real space scene, improves the authenticity of visitor and cultural relic interaction.

Description

Technical Field

[0001] This invention relates to the fields of augmented reality and smart museum technology, and in particular to an AR interactive method, system, device and storage medium for museum artifacts. Background Technology

[0002] Augmented Reality (AR) is a technology designed to add digital information resources to the real physical world (hereinafter referred to as AR technology) to enhance human perception and cognition in entirely new ways. With the rapid development of mobile information technology and smart devices, AR technology has been widely applied in various fields in recent years. AR technology has advantages such as real-time interaction, virtual-real integration, and strong entertainment value, which can bring users a more realistic experience.

[0003] Since the 1980s, the National Museum of China has been undertaking digitization, embarking on a process of informatization, digitalization, and smartification. Currently, museums across China are actively building smart museums, encompassing smart management, smart conservation, and smart services. In content display services, most museums utilize AR technology to dynamically showcase resources, allowing users to clearly view and interact with high-precision 3D models of artifacts. This demonstrates that AR is a primary technological tool for the current and future digitization of cultural relics and the construction of smart museums.

[0004] In 2018, the Holocaust Memorial Museum in Washington, D.C., showcased an AR application that used AR to depict the lives of the townspeople in Eisiskes. Zhang Shaobo et al. proposed an augmented reality museum display system based on an object 6D pose estimation algorithm, which can meet the timeliness requirements of AR applications and effectively improve the way users interact with cultural relics.

[0005] However, when existing museums use AR technology to display cultural relics, they simply present the relics virtually, describing their general appearance and placing them arbitrarily in the scene. This results in a lack of integration between the virtual relics and the actual exhibition space, making the AR exhibition method monotonous and unrealistic. Consequently, the visitors have a poor experience, which is detrimental to the inheritance of my country's agricultural civilization. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by providing a method, system, device, and storage medium for AR interaction of museum artifacts. This solves the problem that in the prior art, virtual artifacts are placed arbitrarily in the scene, resulting in a lack of integration with the spatial scene during the exhibition, leading to a monotonous and unrealistic AR exhibition method and a poor viewing experience for visitors.

[0007] This invention specifically provides the following technical solution: an AR interaction method for museum artifacts, comprising the following steps: Collect image information of various cultural relics in the museum, and perform three-dimensional reconstruction based on the image information of various cultural relics to obtain virtual cultural relic models; After the user terminal obtains photos of cultural relics in the museum, it retrieves the virtual cultural relic model corresponding to the photos; the user terminal collects real-time data of the real scene, identifies the plane of the real scene, and places the virtual cultural relic model on the plane of the real scene to perform AR placement. Users can interact with virtual artifact models placed using AR by using various hand gestures. The user terminal acquires key point information of the user's hand, identifies the user's hand posture information, and displays the corresponding gestures and interaction information with the virtual cultural relic model through the user terminal.

[0008] Preferably, the step of collecting various types of cultural relic image information from the museum and performing three-dimensional reconstruction based on these images to obtain a virtual cultural relic model includes the following steps: Multiple artifacts were obtained from the museum exhibition hall, and photographed from different angles using imaging equipment to obtain an overview image of each artifact. The distance measurement app was used to measure the artifacts in the display case and obtain the length, width, and height structural data of each artifact. The overview map and measured structural data of each artifact are input into the modeling software to perform three-dimensional reconstruction of the artifact, obtain a virtual artifact model, and export it in OBJ format. Use a format conversion tool to convert all the virtual cultural relic models in OBJ format to GLTF format to create a digital resource library of cultural relics; The images of cultural relics used for AR recognition and virtual models of cultural relics in the digital resource library are stored on a cloud server.

[0009] Preferably, the retrieval of the virtual artifact model corresponding to the artifact photograph includes AR image recognition and AR road sign recognition, specifically including the following steps: The AR image recognition includes the following steps: Create an AR image recognition component, based on the WeChat Mini Program xr-frame framework, set the image recognition mode, and set the scene lighting and camera follow basic settings; The images of cultural relics in the digital resource database are used as the image source for AR recognition; After scanning the image of a cultural relic using a user terminal, a virtual model of the corresponding cultural relic appears, and the user terminal can control the scaling and rotation of the virtual cultural relic model. The AR road sign recognition includes the following steps: Create an AR road sign recognition component, set the OSD recognition mode according to the WeChat Mini Program xr-frame framework, and set the scene lighting and camera follow settings; Use real photos of road signs in the museum as the feature recognition source, and add child node location information; Scan the museum's specific landmarks for AR recognition and display AR instructions.

[0010] Preferably, the scaling and rotation control of the virtual cultural relic model via the user terminal includes the following steps: Use event.touches.length to determine whether it is a single-finger or two-finger operation; If the operation is determined to be a single-finger operation, the virtual artifact model will be rotated during the single-finger operation. The specific operation includes the following steps: Get current canvas information and Determine the radius of rotation The solution formula is as follows: ; Get the click position of the last triggered event and the current tap position of the finger Rotation speed Set it to a fixed value, and set the angle change difference in the x-direction according to the rotation radius. The solution formula is as follows: ; The angle change difference in the y direction is obtained in the same way as above, and the rotation result of the virtual cultural relic model is determined based on the angle changes in the x and y directions; If the operation is determined to be a two-finger gesture, the virtual artifact model will be scaled during the two-finger gesture. The specific operation includes the following steps: Obtain the current position information of the two fingers in the x and y directions, calculate the difference between the two fingers in the x and y directions, and calculate the distance between the two fingers. The solution formula is as follows: ; Get the distance between the two fingers when the last zoom event occurred. Calculate the scaling ratio The solution formula is as follows: ; in and This is a scaling parameter used to convert the distance difference into a scaling ratio; The scaling is achieved by multiplying the model values ​​in the x, y, and z directions of the virtual artifact model by the scaling factor.

[0011] Preferably, the virtual artifact model is placed on the real-world scene plane for AR placement, including the following steps: We used WebRTC to acquire a video stream of the museum's real-world scene and then used FFmpeg to extract keyframes from the acquired video. Define the space surrounding all keyframes as Fi, and use 2D CNN to extract image features from the keyframes; Image features are back-projected onto Fi to obtain voxel features and 3D feature volumes; Use average pooling to aggregate features of the same voxel from different keyframes; 3D sparse convolution is used to process the three-dimensional feature volume, construct the three-dimensional sparse feature volume, and determine the voxel occupancy of the plane. Based on the 3D sparse feature volume, extract the normal parameters and plane offsets belonging to the planar voxels; Calculate the displacement between the voxel center and the centroid of the plane, and move the voxel center to the centroid of the plane; The Mean-Shift clustering algorithm was used to determine the final planar parameters after the voxel centers were moved to the plane centroid. The virtual artifact model is placed on the detected plane and then placed using AR.

[0012] Preferably, the user terminal acquires key point information of the user's hand, identifies the user's hand posture information, and displays the corresponding gesture and interaction information between the virtual cultural relic model through the user terminal, including the following steps: Create a gesture recognition component and set the gesture recognition mode according to the WeChat Mini Program xr-frame framework; The user's hand joint information is captured by the camera to determine hand posture information; Customize basic gesture information and display interactive information corresponding to the virtual cultural relic model based on the recognized hand posture information.

[0013] Preferably, it also includes connecting the user client to the digital resource database of cultural relics in the cloud server to enable interaction between multiple users and cultural relics, including the following steps: User-authorized personal information is stored in a MySQL database on a cloud server; Select the message interaction module to establish a connection with the MySQL database, view other users' messages, and enter your own message. New messages are then synchronously updated in the database.

[0014] Preferably, the present invention also includes a museum artifact AR interactive system, comprising: The data management module is used to collect image information of various cultural relics in the museum, and to perform three-dimensional reconstruction based on the image information of various cultural relics to obtain virtual cultural relic models; The AR function module is used to retrieve virtual artifact models corresponding to the artifact photos after the user terminal obtains photos of artifacts in the museum; the user terminal collects real-world scenes in real time, identifies the plane of the real-world scene, and places the virtual artifact models on the plane of the real-world scene for AR placement. The interaction module is used to allow users to interact with virtual artifact models placed in AR through various gestures. The gesture recognition module is used to acquire key point information of the user's hand on the user terminal, recognize the user's hand posture information, and display the corresponding gesture and interaction information between the virtual cultural relic model on the user terminal.

[0015] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the aforementioned AR interactive method for museum artifacts.

[0016] The present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the AR interactive method for museum artifacts.

[0017] Compared with the prior art, the present invention has the following significant advantages: This invention uses collected cultural relic information to perform 3D reconstruction on each type of cultural relic, obtaining corresponding virtual cultural relic models. This effectively improves the real detail information of the cultural relics observed later and reduces the pressure on the user's scanning and recognition. At the same time, during AR recognition, by acquiring a photo of a certain type of cultural relic, the corresponding virtual cultural relic is directly obtained, and the real scene is captured in real time. The virtual cultural relic model is placed on the recognized real scene plane through corresponding algorithms, improving the performance of plane detection, promoting the accuracy of virtual cultural relic placement, better realizing the integration of virtual cultural relics with the real space scene, and improving the realism of visitors' interaction with cultural relics. Attached Figure Description

[0018] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 The images are three-dimensional models of some agricultural historical artifacts; (a) is a stone axe; (b) is a garlic-shaped pot; (c) is a curved plow; (d) is a measuring cup; (e) is a small agricultural tool - a small hook; and (f) is a barley fork. Figure 3 This is a flowchart of the AR image recognition operation. Figure 4The images show the results of AR image recognition, where (a) is the image after AR recognition, (b) is the image of the virtual cultural relic model scaled up, and (c) is the image of the virtual cultural relic model rotated. Figure 5 Flowchart for AR road sign recognition operation; Figure 6 This is an AR road sign rendering; Figure 7 Flowchart of AR placement operation; Figure 8 This is an image showing the AR placement effect. Figure 9 Here is a flowchart of the AR gesture recognition operation. Figure 10 This is an image illustrating AR gesture effects. Figure 11 Flowchart of multi-user message interaction operation; Figure 12 This is a screenshot showing the interactive effect of multiple users leaving comments. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] This invention proposes an AR interaction method, system, device, and storage medium for museum artifacts, specifically as follows: AR technology can provide a more intuitive and vivid way of displaying cultural relics, better preserve and display their original appearance. It can not only allow visitors to virtually view the exhibition, but also allow visitors to interact with the cultural relic models in various ways, thereby effectively protecting and inheriting agricultural cultural relics.

[0021] In conclusion, using AR technology to build a Chinese agricultural history museum can not only effectively protect and preserve agricultural cultural relics and enhance the visitor experience, but also provide diversified display methods and strengthen communication and interaction among visitors. This is of great significance for promoting the dissemination and development of agricultural culture and enhancing public awareness and understanding of agricultural history. To help understand the features and advantages of this invention, the following will provide a more detailed description of the invention in conjunction with a framework flowchart.

[0022] refer to Figure 1 The embodiments of this application provide an AR interaction method for museum artifacts, including the following steps: Step S1: Collect image information of various cultural relics in the museum, and perform three-dimensional reconstruction based on the image information of various cultural relics to obtain virtual cultural relic models.

[0023] The museum is the Museum of Chinese Agricultural History.

[0024] This step involves collecting image information of various cultural relics in the museum, and then performing 3D reconstruction based on this image information to obtain virtual cultural relic models. The specific steps include: Step S11: Obtain multiple (100 representative agricultural history) artifacts from the six exhibition halls of the museum, and use a camera to take pictures of the artifacts from different dimensions (use a mobile phone to take pictures from the top, middle and bottom in 360°) to obtain an overview image of each artifact.

[0025] Step S12: Use a distance measurement app to measure the artifacts in the display case and obtain structural data such as length, width, and height for each artifact.

[0026] Step S13: Use 3ds Max modeling software to perform 3D reconstruction of the measured cultural relic, obtain a 3D model (virtual cultural relic model) and export it in OBJ format.

[0027] To ensure that the 3D models of cultural relics can be displayed correctly in the system, the obj2gltf format conversion tool is used to convert the OBJ format cultural relic models to GLTF format. This ensures that users can perform various AR interactions with the 3D models of cultural relics throughout the entire experience, thus guaranteeing a smooth interactive experience.

[0028] The images and 3D models of cultural relics used for AR recognition are stored on Tencent Cloud servers for unified management, facilitating the subsequent use of these images and models by various AR functions. The 3D model display effect of some agricultural historical relics is shown below. Figure 2 As shown. This step involves creating a digital resource library of cultural relics and uploading it to the cloud server, including the following steps:

[0029] Step S14: Use a format conversion tool to convert all 3D models in OBJ format to GLTF format to create a digital resource library of cultural relics.

[0030] Step S15: Store the images of cultural relics used for AR recognition and virtual cultural relic models in the digital cultural relic resource database on a cloud server for unified management.

[0031] To facilitate unified data management, user personal information, including nicknames and IDs, is stored in a Tencent Cloud MySQL database to store various user data and facilitate subsequent interactions between users.

[0032] Step S2: After the user terminal obtains photos of cultural relics in the museum, it retrieves the virtual cultural relic model corresponding to the photos; the user terminal collects real-time data of the real scene, identifies the plane of the real scene, and places the virtual cultural relic model on the plane of the real scene for AR placement.

[0033] This step involves retrieving the virtual artifact model corresponding to the artifact photograph, including AR image recognition, specifically as follows: Figure 3 As shown, the specific steps include the following: Step S21: Create an AR image recognition component. Based on the WeChat Mini Program xr-frame framework, set the image recognition mode, scene lighting, camera tracking, and other basic settings.

[0034] Step S22: Use images of cultural relics from the digital resource database as the image source for AR recognition.

[0035] The process involves obtaining the selected images of cultural relics from the page and transmitting them to the AR image recognition component; the AR image recognition component then identifies the feature images and modules that need to be recognized.

[0036] Step S23: After scanning the image of the cultural relic with a mobile phone, a 3D model of the corresponding cultural relic (virtual cultural relic model) appears, which is used to control the scaling and rotation of the cultural relic.

[0037] This involves scanning images of cultural relics in real life, identifying and tracking the 3D model of the relics, rotating, zooming in or out of the 3D model, and determining whether to change the image selection information. If so, the image is re-acquired; otherwise, AR recognition is completed.

[0038] The process for scaling and rotating cultural relics includes the following steps: The event.touches.length is used to determine whether it is a single-finger or two-finger operation.

[0039] If the operation is determined to be a single-finger gesture, the model will be rotated during the single-finger gesture. The specific operation includes the following steps: Get current canvas information and Determine the radius of rotation The solution formula is as follows: ; Get the click position of the last triggered event and the current tap position of the finger Rotation speed Set it to a fixed value, and set the angle change difference in the x-direction according to the rotation radius. The solution formula is as follows: ; The angle change difference in the y direction is obtained in the same way as above, and the rotation result of the three-dimensional model of the cultural relic is determined based on the angle changes in the x and y directions.

[0040] If the operation is determined to be a two-finger gesture, the model will be scaled up or down during the two-finger gesture. The specific operation includes the following steps: Obtain the current position information of the two fingers in the x and y directions, calculate the difference between the two fingers in the x and y directions, and calculate the distance between the two fingers. The solution formula is as follows: ; Get the distance between the two fingers when the last zoom event occurred. Calculate the scaling ratio The solution formula is as follows: ; in and This is a scaling parameter used to convert distance differences into a scaling ratio.

[0041] The scaling is achieved by multiplying the model values ​​in the x, y, and z directions of the 3D model of the cultural relic (virtual cultural relic model) by the scaling ratio.

[0042] The AR image recognition effect, as well as the effect of rotating and scaling virtual cultural relic models, are shown in the following figures. Figure 4 As shown in (a), (b) and (c).

[0043] This invention incorporates an AR road sign recognition function, specifically as follows: Figure 5 As shown in the image. Users can use their phone's rear camera to scan specific signs with AR markers within the China Agricultural History Museum. Once a scan is successful, AR directions will be displayed, allowing users to choose their preferred viewing route. The AR sign recognition effect is as follows: Figure 6 As shown.

[0044] This step involves retrieving the virtual artifact model corresponding to the artifact photograph, including AR road sign recognition, and specifically includes the following steps: Step S24: Create an AR road sign recognition component. Based on the WeChat Mini Program xr-frame framework, set the OSD recognition mode, scene lighting, and camera follow settings.

[0045] Step S25: Use real photos of the signs in the museum as the feature recognition source, and add the location information of the child nodes.

[0046] Step S26: Scan the specific road signs of the museum in real life for AR recognition and display AR guidance information.

[0047] After scanning a specific sign in the museum, it determines whether it matches the recognition source in the AR sign recognition component. If it does, it calculates the child node location information and displays the sign information; otherwise, the recognition fails.

[0048] In this step, the virtual artifact model is placed on the identified real-world scene plane using AR placement, specifically as follows: Figure 7 As shown. It includes the following steps:

[0049] When a user scans a real-life scene using their phone's rear camera, WebRTC is used to obtain a video stream of the museum's real-life scene, and the video stream from the phone's camera is then pushed to Tencent Cloud servers.

[0050] (Using a plane detection algorithm to process real-world scenes) FFmpeg is used to extract keyframes from the captured video. The space surrounding all keyframes is defined as Fi. 2D CNN is used to extract image features from the keyframes. The image features are back-projected into (local features overall) Fi to obtain voxel features and a 3D feature volume. Average pooling is used to aggregate features of the same voxel from different keyframes. 3D sparse convolution is used to process the 3D feature volume. (It is determined whether the distance between the voxel and the surface is less than λ. If it is, the voxel belongs to the plane (where greater than λ means the voxel does not belong to the plane). It is also determined whether the distance between the voxel and the surface is less than θ. If it is, the voxel does not belong to the plane. Otherwise, a 3D sparse feature volume is constructed.) The 3D sparse feature volume is constructed to determine the voxel occupancy of the plane.

[0051] Based on the 3D sparse feature volume, extract the normal parameters and plane offset (estimate) of the voxels belonging to the plane, calculate the displacement (estimate) between the voxel center and the centroid of the plane, and move the voxel center to the centroid of the plane.

[0052] The Mean-Shift clustering algorithm is used to determine the final planar parameters. The virtual model of the artifact is then placed on the detected plane (the plane in Fi is detected, planar data information is returned, and the virtual artifact is placed at the center of the plane). AR placement is then performed to achieve the desired effect. Figure 8 As shown.

[0053] This invention incorporates a gesture recognition function, specifically as follows: Figure 9 As shown in the image, users can use their phone's rear camera to scan a single-hand gesture according to the on-screen instructions. The camera captures the user's hand joint information, determines the hand posture, and displays corresponding interactive information, increasing the user's enjoyment when viewing cultural relics. The gesture recognition effect is as follows: Figure 10As shown. Currently, the basic custom gestures mainly include six interactive gestures: heart, thumbs up, OK, number one, number five, and number six. More gesture types can be added later as needed. This invention is not limited by the user's terminal system and requires no download; any user can start using it by scanning a QR code, improving the convenience of user operation.

[0054] Step S3: The user interacts with the virtual artifact model placed by AR using various gestures.

[0055] Step S4: Use the user terminal to obtain key point information of the user's hand, recognize the user's hand posture information, and display the interaction information between the corresponding gesture and the virtual cultural relic model.

[0056] This step specifically includes the following steps: Step S41: Create a gesture recognition component and set the gesture recognition mode according to the WeChat Mini Program xr-frame framework.

[0057] Step S42: Capture the user's hand joint information using a camera to determine hand posture information.

[0058] Step S43: Define basic gesture information and display corresponding interactive information based on the recognized hand gesture information.

[0059] This invention incorporates a multi-user message interaction function, specifically as follows: Figure 11 As shown in the image, while browsing the museum, users can select the message board to view other users' comments and experiences. Users can also authorize the system to enter their own messages, which are then updated in the database and displayed on the message board, facilitating communication among all museum users. The effect is as follows: Figure 12 As shown.

[0060] It also includes step S5: using the user terminal to connect with the digital resource database of cultural relics in the cloud server to conduct interaction between multiple users and cultural relics.

[0061] This step specifically includes the following steps: Step S51: Store the user's authorized personal information in the MySQL database of the cloud server.

[0062] Step S52: Select the message interaction module, establish a connection with the MySQL database, view other users' message content, and enter your own message content. New message content is synchronously updated in the database.

[0063] Based on the above methods and statements, the present invention also includes a museum artifact AR interactive system, comprising: a data management module, an AR function module, a gesture recognition module, and a multi-user interaction module.

[0064] The system comprises several modules: a data management module for collecting images of various cultural relics from the museum and performing 3D reconstruction to obtain virtual relic models; an AR module for retrieving corresponding virtual relic models from photos of artifacts in Chinese on the user's device; a real-time scene capture module for the user's device and scene plane recognition; and a gesture recognition module for acquiring key hand points, recognizing hand postures, and displaying corresponding gesture interactions. A multi-user interaction module connects the user's device to a digital resource library of cultural relics on a cloud server, enabling multi-user interaction with the artifacts. The data management module includes images and 3D models of cultural relics from the China Agricultural History Museum.

[0065] This invention creates a digital resource database of Chinese agricultural historical relics, enabling users to view the details of relics up close using AR image recognition in Chinese agricultural history museums. The AR navigation function allows users to customize browsing routes, and the AR placement function and gesture recognition function enable diverse interactions with the relics, bringing users closer to the relics and allowing them to experience the unique charm of the relics.

[0066] The present invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a museum artifact AR interaction method.

[0067] The present invention also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of a museum artifact AR interactive method.

[0068] The above description, in conjunction with specific preferred embodiments, provides a more detailed explanation of the present invention. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for AR interaction of museum artifacts, characterized in that, Includes the following steps: Collecting image information of various cultural relics in museums, and performing 3D reconstruction based on the image information to obtain virtual cultural relic models; including the following steps: Multiple artifacts were obtained from the museum exhibition hall. The artifacts were photographed from different dimensions using a camera to obtain an overview image of each artifact. A distance measuring app was used to measure the artifacts in the display case to obtain the length, width, and height structural data of each artifact. The overview image and the measured structural data of each artifact were input into the modeling software to perform three-dimensional reconstruction of the artifacts, obtain a virtual artifact model, and export it in OBJ format. Using a format conversion tool, all virtual cultural relic models in OBJ format are converted to GLTF format to create a digital resource library of cultural relics; the cultural relic images and virtual cultural relic models used for AR recognition in the digital resource library are stored on a cloud server; After the user client obtains photos of cultural relics in the museum, it retrieves the virtual cultural relic model corresponding to the photos; the user client then captures real-world scene data in real time, identifies the plane of the real-world scene, and places the virtual cultural relic model on the plane of the real-world scene, performing AR placement; this includes the following steps: WebRTC was used to acquire a real-world video stream of the museum, and FFmpeg was used to extract keyframes from the acquired video. The space enclosing all keyframes was defined as Fi, and 2D CNN was used to extract image features from the keyframes. The image features were back-projected onto Fi to obtain voxel features and a 3D feature volume. Average pooling was used to aggregate features of the same voxels from different keyframes. 3D sparse convolution was used to process the 3D feature volume, constructing a 3D sparse feature volume to determine the voxel occupancy of planes. Based on the 3D sparse feature volume, the normal parameters and plane offsets of plane voxels were extracted. The displacement between the voxel center and the centroid of the plane was calculated, and the voxel center was moved to the plane centroid. Mean-Shift clustering was used to determine the final plane parameters after the voxel center was moved to the plane centroid. Virtual artifact models were placed on the detected plane for AR placement. Users can interact with virtual artifact models placed using AR by using various hand gestures. The user terminal obtains key point information of the user's hand, identifies the user's hand posture information, and displays the corresponding gestures and interaction information with the virtual cultural relic model through the user terminal.

2. The AR interactive method for museum artifacts as described in claim 1, characterized in that, The process of retrieving the virtual artifact model corresponding to the artifact photograph includes AR image recognition and AR road sign recognition, specifically including the following steps: The AR image recognition includes the following steps: Create an AR image recognition component, based on the WeChat Mini Program xr-frame framework, set the image recognition mode, and set the scene lighting and camera follow basic settings; The images of cultural relics in the digital resource database are used as the image source for AR recognition; After scanning the image of a cultural relic using a user terminal, a virtual model of the corresponding cultural relic appears, and the user terminal can control the scaling and rotation of the virtual cultural relic model. The AR road sign recognition includes the following steps: Create an AR road sign recognition component, set the OSD recognition mode according to the WeChat Mini Program xr-frame framework, and set the scene lighting and camera follow settings; Use real photos of road signs in the museum as the feature recognition source, and add child node location information; Scan the museum's specific landmarks for AR recognition and display AR instructions.

3. The AR interactive method for museum artifacts as described in claim 2, characterized in that, The scaling and rotation control of the virtual cultural relic model via the user terminal includes the following steps: Use event.touches.length to determine whether it is a single-finger or two-finger operation; If the operation is determined to be a single-finger operation, the virtual artifact model will be rotated during the single-finger operation. The specific operation includes the following steps: Get current canvas information and Determine the radius of rotation The solution formula is as follows: ； Get the click position of the last triggered event and the current tap position of the finger Rotation speed Set it to a fixed value, and set the angle change difference in the x-direction according to the rotation radius. The solution formula is as follows: ； The angle change difference in the y direction is obtained in the same way as above, and the rotation result of the virtual cultural relic model is determined based on the angle changes in the x and y directions; If the operation is determined to be a two-finger gesture, the virtual artifact model will be scaled during the two-finger gesture. The specific operation includes the following steps: Obtain the current position information of the two fingers in the x and y directions, calculate the difference between the two fingers in the x and y directions, and calculate the distance between the two fingers. The solution formula is as follows: ； Get the distance between the two fingers when the last zoom event occurred. Calculate the scaling ratio The solution formula is as follows: ； in and This is a scaling parameter used to convert the distance difference into a scaling ratio; The scaling is achieved by multiplying the model values ​​in the x, y, and z directions of the virtual artifact model by the scaling factor.

4. The AR interactive method for museum artifacts as described in claim 1, characterized in that, The user terminal acquires key point information of the user's hand, recognizes the user's hand posture information, and displays the corresponding gestures and interaction information with the virtual cultural relic model through the user terminal, including the following steps: Create a gesture recognition component and set the gesture recognition mode according to the WeChat Mini Program xr-frame framework; The user's hand joint information is captured by the camera to determine hand posture information; Customize basic gesture information and display interactive information corresponding to the virtual cultural relic model based on the recognized hand posture information.

5. The AR interactive method for museum artifacts as described in claim 2, characterized in that, It also includes connecting the user client to a digital resource database of cultural relics on a cloud server to facilitate interaction between multiple users and cultural relics, including the following steps: User-authorized personal information is stored in a MySQL database on a cloud server; Select the message interaction module to establish a connection with the MySQL database, view other users' messages, and enter your own message. New messages are then synchronously updated in the database.

6. An AR interactive system for museum artifacts, characterized in that, include: The data management module is used to collect image information of various cultural relics in the museum, and to perform 3D reconstruction based on the image information to obtain virtual cultural relic models; it includes the following steps: Multiple artifacts were obtained from the museum exhibition hall. The artifacts were photographed from different dimensions using a camera to obtain an overview image of each artifact. A distance measuring app was used to measure the artifacts in the display case to obtain the length, width, and height structural data of each artifact. The overview image and the measured structural data of each artifact were input into the modeling software to perform three-dimensional reconstruction of the artifacts, obtain a virtual artifact model, and export it in OBJ format. Using a format conversion tool, all virtual cultural relic models in OBJ format are converted to GLTF format to create a digital resource library of cultural relics; the cultural relic images and virtual cultural relic models used for AR recognition in the digital resource library are stored on a cloud server; The AR (Augmented Reality) module is used to retrieve virtual artifact models corresponding to photos of artifacts in museums after the user terminal acquires the photos; it also performs AR placement by capturing real-world scene data in real time, identifying the plane of the real-world scene, and placing the virtual artifact models on the plane of the real-world scene; the steps include: WebRTC was used to acquire a real-world video stream of the museum, and FFmpeg was used to extract keyframes from the acquired video. The space enclosing all keyframes was defined as Fi, and 2D CNN was used to extract image features from the keyframes. The image features were back-projected onto Fi to obtain voxel features and a 3D feature volume. Average pooling was used to aggregate features of the same voxels from different keyframes. 3D sparse convolution was used to process the 3D feature volume, constructing a 3D sparse feature volume to determine the voxel occupancy of planes. Based on the 3D sparse feature volume, the normal parameters and plane offsets of plane voxels were extracted. The displacement between the voxel center and the centroid of the plane was calculated, and the voxel center was moved to the plane centroid. Mean-Shift clustering was used to determine the final plane parameters after the voxel center was moved to the plane centroid. Virtual artifact models were placed on the detected plane for AR placement. The interaction module is used to allow users to interact with virtual artifact models placed in AR through various gestures. The gesture recognition module is used to acquire key point information of the user's hand on the user terminal, recognize the user's hand posture information, and display the corresponding gesture and interaction information between the virtual cultural relic model on the user terminal.

7. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of a museum artifact AR interaction method as described in any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the AR interactive method for museum artifacts as described in any one of claims 1 to 5.

Images