VR-based tourism industry fusion method

By establishing a cultural relics data database and real-time adjustment of virtual models, the problem of lack of interactivity in traditional tourism experience is solved, the personalized interaction between users and cultural relics is realized, immersion and education are enhanced, and the digital transformation of the tourism industry is promoted.

CN119048720APending Publication Date: 2024-11-29JIANGXI UNIVERSITY OF FINANCE AND ECONOMICS
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411541064.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-31
Publication Date
2024-11-29

Smart Images

  • Figure CN119048720A_ABST
    Figure CN119048720A_ABST
Patent Text Reader

Abstract

The invention is suitable for the field of tourism industry, and provides a VR-based tourism industry fusion method, and the method comprises the steps: building a virtual model for each kind of cultural relics, collecting the real-time image information of a camera device of a user, and achieving the real-time fusion of the cultural relics through the precise posture recognition and the dynamic adaptation of the virtual model. The user can experience cultural relic display conforming to the own body characteristics in the virtual environment, and the immersion and reality sense are enhanced. According to the method, the posture of each user is unique, each user can obtain personalized experience by dynamically adjusting the virtual model, the method combines advanced technologies such as computer vision, virtual reality and human body posture recognition, digital transformation of the tourism industry is promoted, and wide possibility is provided for future application development.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of tourism industry, and in particular relates to a tourism industry integration method based on VR. Background Art

[0002] The integration of virtual reality (VR) technology in the tourism industry is rapidly changing the way tourists experience the world. By creating a fully immersive virtual environment, VR transforms the tourism experience from traditional passive viewing to active participation and interaction. Visitors can enter a virtual world by wearing a head-mounted display (HMD) to explore historical sites, museum exhibitions, and even participate in virtual tours without leaving home. This technology not only provides visualization, but also enhances immersion through sound and touch.

[0003] In this regard, augmented reality (AR) technology also plays an important auxiliary role, by superimposing virtual information on the real world to enhance visitors' actual experience. For example, when visitors scan exhibits in a museum with a smartphone or tablet, they can see richer background information, animations, or interactive content with cultural relics. This interactivity allows visitors to increase their personal sense of participation while learning about history and culture.

[0004] However, many current traditional tourism exhibitions and museum displays lack interactivity. Visitors can only obtain information through a single method such as seeing and hearing, and the tourists' experience is often passive. Even with the use of VR equipment, many traditional tourism experiences cannot provide sufficient immersion, causing tourists to feel a sense of distance when viewing cultural relics. At the same time, when faced with a variety of historical costumes and accessories, they can only view them from a distance and cannot interact. Summary of the invention

[0005] The purpose of the present invention is to provide a tourism industry integration method based on VR, aiming to solve the technical problems existing in the prior art identified in the background technology.

[0006] The present invention is implemented as follows: a tourism industry integration method based on VR, the method comprising: Establish a cultural relic data information database and store all the cultural relic information existing locally, and establish a virtual model for each cultural relic based on the cultural relic information, and bind the use location to the virtual model of each cultural relic; Obtain the image information uploaded by the user and the marked position of the user in the image, identify the item information existing in the marked position, search it in the cultural relic data information database, identify and extract the type of cultural relic at the marked position and the corresponding virtual model; Collect real-time image information from the user's camera device, identify portrait information, analyze and obtain the body shape data and body proportion data of the person, and dynamically adjust the virtual model of the cultural relic selected by the user based on this; Identify the position and posture of the person in the current image, and overlay the selected virtual model to the corresponding position of the person in the image according to the usage position of the virtual model; Identify the current portrait posture and adjust the virtual model accordingly.

[0007] As a further solution of the present invention, the establishment of a cultural relic data information database, storing all the cultural relic information existing locally, establishing a virtual model for each cultural relic based on the cultural relic information, and binding a usage location for the virtual model of each cultural relic specifically includes: Collect local cultural relics information, including images of cultural relics and cultural relics attributes, where the cultural relics attributes include information, type, quality, and size, and classify the collected cultural relics according to type; Based on the collected cultural relic image information, 3D modeling is performed for each cultural relic to generate a 3D virtual model of the cultural relic; Bind the corresponding usage position to the virtual model of each artifact, including the wearing position and the wearing position; A cultural relics data information database will be established, and the created virtual models and information of cultural relics will be stored in the cultural relics data information database.

[0008] As a further solution of the present invention, the object information existing at the identified marked position is searched in a cultural relic data information database to identify and extract the type of cultural relic at the marked position and the corresponding virtual model, specifically including: Get the image information uploaded by the user and read the marked position in the image; Perform edge detection on the objects at the marked position, crop the specific object image, and perform feature detection on the object image to identify the shape, color and texture information of the object; Based on the results of feature detection, they are compared and matched with the information in the cultural relics data database to identify the type of cultural relics to which the item at the marked position belongs, and extract the virtual model of the cultural relic.

[0009] As a further solution of the present invention, the real-time image information of the user's camera device is collected, and the portrait information is recognized, and the body shape data and body proportion data of the person are analyzed and obtained, and based on this, the virtual model of the cultural relic selected by the user is dynamically adjusted, specifically including: Establish a connection with the user's camera device and obtain real-time images captured by the camera device; Use computer vision to identify the user's personal features, including face and body outline, from real-time images; Analyze the identified person to obtain the user's body data, including the height, width, depth and relative positions of various body parts; The body proportion information is extracted from the user's body shape data, and the extracted virtual model of the cultural relic is proportionally adjusted accordingly, including the overall length, local length, and length-to-width ratio.

[0010] As a further solution of the present invention, the identifying the position and posture of the portrait in the current image and covering the selected virtual model to the corresponding position of the portrait in the image according to the use position of the virtual model specifically includes: Analyze the recognized portrait, perform posture estimation, and identify the user's body parts and the corresponding positions in space; According to the extracted posture information, the use position of the virtual model is matched, and the corresponding parts of the virtual model are matched with the actual body parts of the user; According to the matching results, the virtual model is positioned at the corresponding actual position in the user's image, and the coordinates of the virtual model are fused with the user's position and posture information.

[0011] As a further solution of the present invention, the identifying the current portrait posture and adjusting the virtual model synchronously accordingly specifically includes: Identify the nodes of the portrait and use vector operations to calculate the angle based on the three-dimensional coordinates of the joints; Extract the specific positions of the user's hands, legs, and head in the image, and extract the corresponding three-dimensional position data by obtaining the coordinates; Based on the recognized posture data, the virtual model is adjusted in real time to keep it consistent with the user's movements; The calculated joint angles and positions are applied to the virtual model, and the user's movements are monitored in real time to update the coordinates of the virtual model in real time.

[0012] The beneficial effects of the present invention are: Through precise posture recognition and dynamic adaptation of virtual models, users can experience the display of cultural relics that match their own physical characteristics in the virtual environment, enhancing the sense of immersion and reality. Each user has a unique body shape. By dynamically adjusting the virtual model, each user can get a personalized experience. This personalization not only improves user satisfaction, but also promotes the emotional connection between users and cultural relics. At the same time, the process emphasizes real-time, and can quickly adjust the position and proportion of the virtual model according to the user's movements and posture changes to achieve a natural and smooth interactive experience. Combining virtual cultural relics with the user's actual images not only enhances the fun and educational nature of cultural relics display, but also provides an innovative platform for cultural communication and attracts more users to participate. Ultimately, this method combines advanced technologies such as computer vision, virtual reality, and human posture recognition to promote the digital transformation of the tourism industry and provide broad possibilities for future application development. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flow chart of a tourism industry integration method based on VR provided by an embodiment of the present invention; Figure 2 A flowchart of establishing a virtual model for each type of cultural relic based on cultural relic information provided by an embodiment of the present invention; Figure 3 A flowchart for identifying and extracting the types of cultural relics at marked locations and the corresponding virtual models provided by an embodiment of the present invention; Figure 4 A flow chart of dynamically adjusting a virtual model of a cultural relic selected by a user provided in an embodiment of the present invention; Figure 5 A flow chart of overlaying a selected virtual model to a corresponding position of a portrait in an image according to a use position of the virtual model provided by an embodiment of the present invention; Figure 6 A flowchart of an embodiment of the present invention for identifying the position and posture of a portrait in a current image, and overlaying the selected virtual model to the corresponding position of the portrait in the image according to the use position of the virtual model. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0015] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0016] Figure 1 A flow chart of a tourism industry integration method based on VR provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes: S100, establishing a cultural relic data information database, storing all cultural relic information existing locally, establishing a virtual model for each cultural relic based on the cultural relic information, and binding a usage location for the virtual model of each cultural relic; like Figure 2 As shown, the establishment of a cultural relic data information database, storing all the cultural relic information existing locally, establishing a virtual model for each cultural relic based on the cultural relic information, and binding a usage location for the virtual model of each cultural relic, specifically includes: S110, collecting local cultural relics information, including cultural relics images and cultural relics attributes, wherein the cultural relics attributes include information, type, quality, and size, and classifying the collected cultural relics according to the type; S120, performing 3D modeling for each type of cultural relic based on the collected cultural relic image information to generate a 3D virtual model of the cultural relic; S130, binding a corresponding usage position to the virtual model of each cultural relic, including a wearing position and a wearing position; S140, establishing a cultural relic data information database, and storing the created cultural relic virtual model and information in the cultural relic data information database.

[0017] Based on the collected cultural relic image information, 3D modeling is performed for each cultural relic to generate a 3D virtual model of the cultural relic, specifically including: Preprocess the cultural relic images and input the preprocessed cultural relic images into the pre-trained ResNet50 model to obtain image features that represent the key visual information of the images; Construct a 3D mesh of a regular icosahedron based on a sphere; Create a texture map for the 3D mesh, apply the color and detail features of the original image to the 3D mesh, and obtain the initial 3D model of the artifact; Based on Laplace smoothing, the Laplace smoothing loss function is constructed based on the position difference between the vertex and the adjacent vertices in the 3D mesh; Sampling points are obtained by sampling from the initial 3D model of the cultural relic. Based on the Chamfer distance, the Chamfer distance loss function is constructed by taking the mean value between the sampling points and the point cloud of the adjacent image features, and between the point cloud of the image features and the adjacent sampling points; The Laplace smoothing loss function and the Chamfer distance loss function are combined in a weighted sum manner to form a total loss function; The initial 3D model of the cultural relic is iteratively optimized by minimizing the total loss function to gradually adjust the vertex positions of the mesh so that the shape gradually approaches the real form of the cultural relic, and the 3D model of the cultural relic is obtained.

[0018] This scheme is mainly aimed at clothing and accessories. This step first collects cultural relics information, using field visits, museum archives, photography, scanning and other methods to ensure that high-definition images and detailed attribute information of local cultural relics are obtained, including the name of the cultural relics, historical background, function, material, size, type, etc. The accuracy and comprehensiveness of the data are ensured through questionnaires, interviews and literature collection. Then, the collected cultural relics are systematically classified by type (such as ceramics, metals, fabrics, etc.), era (such as ancient, modern), and purpose (such as decoration, daily use). Subsequently, professional modeling software is used to create high-quality 3D virtual models based on the collected cultural relic images to ensure that they are as close to real cultural relics as possible in shape, color and texture; for complex cultural relics, 3D scanning technology is considered to directly generate high-precision models. In this process, the use position of each cultural relic is clarified, and its wearing position and wearing position in user interaction are bound, such as the wearing position of jewelry and the wearing position of clothing, and this information is bound to the virtual model. Finally, a structured database system is developed to store the information and virtual models of all cultural relics in the cultural relics data information base to support efficient data retrieval and management, while ensuring that the database has a good indexing mechanism to facilitate quick queries.

[0019] Collect cultural relic information through various channels and means to ensure that the attributes and image data of the cultural relics are comprehensive and accurate, so that subsequent virtual displays and interactions are more credible, thereby improving the user experience. After binding the usage location, users can truly feel the use effect of the cultural relics in the virtual environment, increase immersion and interactivity, and improve the overall travel experience. High-quality 3D modeling provides users with a vivid visual experience, showing the details and beauty of cultural relics, allowing users to feel the history and cultural connotations of cultural relics in a VR environment. S200, obtain the image information uploaded by the user, as well as the marked position of the user in the image, and identify the item information at the marked position, search it in the cultural relic data information library, identify and extract the type of cultural relic at the marked position and the corresponding virtual model; like Figure 3 As shown, the item information existing at the identified marked position is searched in the cultural relic data information database to identify and extract the type of cultural relic at the marked position and the corresponding virtual model, specifically including: S210, obtaining the image information uploaded by the user, and reading the marked position in the image; S220, performing edge detection on the object at the marked position, cropping a specific object image, and performing feature detection on the object image to identify the shape, color, and texture information of the object; S230, based on the result of feature detection, compare and match it with the information in the cultural relic data information database, identify the type of cultural relic to which the object at the marked position belongs, and extract the virtual model of the cultural relic.

[0020] Perform edge detection on the objects at the marked position, crop the specific object image, and perform feature detection on the object image to identify the shape, color and texture information of the object, including: Convert the image uploaded by the user into a grayscale image, use the Canny algorithm to perform edge detection on the grayscale image, and generate a binary image; The contour detection algorithm is used to obtain all independent contours in the binary image, and the area and perimeter of each contour are calculated to obtain the shape information of the object; Fill the inside of the outline with white and the rest with black to get the mask image; The mask image is used to extract the pixels of the contour area in the color image, and the average color of the pixels in the contour area is calculated to identify the color information of the object; The mask image is used to obtain the contour area in the grayscale image, and the texture of the contour area in the grayscale image is calculated using the local binary pattern to obtain the local texture of the image; the grayscale co-occurrence matrix of the contour area in the grayscale image is calculated to obtain the texture characteristics in the image; and the texture information of the object is constructed based on the local texture of the image and the texture characteristics in the image.

[0021] This step first obtains the image information uploaded by the user, which can be captured in real time or uploaded later through the user's camera device, such as a smartphone or camera. Next, the system reads the marked positions in the image, such as the specific objects or areas specified by the user in the image, for subsequent analysis. On this basis, the edge detection of the object is performed, and the specific object area in the image is identified and cropped through image processing algorithms. After cropping, the system will perform feature detection on the object image, and form a feature description of the object by extracting the shape, color and texture information of the object.

[0022] Based on the results of feature detection, the system compares and matches the extracted features with the information in the cultural relic data information database. This process involves the use of machine learning models or image recognition algorithms to determine whether the items at the marked locations match the cultural relics in the database. After successful identification, the system will extract the type of cultural relic and obtain the corresponding virtual model for subsequent interactive use. This process ensures that users can interact with actual cultural relics in a virtual environment.

[0023] By acquiring user image information and accurately identifying the location of the mark, a personalized and customized user experience can be achieved, allowing users to interact more closely with cultural relics. The combination of edge detection and feature detection ensures efficient and accurate extraction of artifact information, thereby improving the accuracy and speed of recognition. In addition, when performing feature comparison, the system can quickly match relevant information in the cultural relic data information library, greatly reducing the time cost of users looking for relevant cultural relic information and improving efficiency. By using machine learning and deep learning algorithms, the system can continuously learn and optimize the recognition model, thereby improving future recognition accuracy and efficiency and adapting to cultural relics of different types and styles.

[0024] S300, collecting real-time image information of the user's camera device, identifying portrait information, analyzing and obtaining body shape data and body proportion data of the person, and dynamically adjusting the virtual model of the cultural relic selected by the user based on the data; like Figure 4 As shown, the real-time image information of the user's camera device is collected, and the portrait information is recognized, and the body shape data and body proportion data of the person are analyzed and obtained, and based on this, the virtual model of the cultural relic selected by the user is dynamically adjusted, specifically including: S310, establishing a connection with a user's camera device and acquiring real-time images captured by the camera device in real time; S320, using computer vision to identify the user's personal features from real-time images, including face and body contours; S330, analyzing the identified person to obtain the user's body data, including the height, width, depth and relative positions of various parts of the body; S340, extracting body proportion information from the user's body shape data, and adjusting the proportions of the extracted virtual model of the cultural relic accordingly, including overall length, local length, and length-to-width ratio.

[0025] This step requires establishing a connection with the user's camera device, which can be achieved through an application or software interface to ensure real-time acquisition of image data collected by the camera device. This process requires the user to authorize camera access rights so that the system can capture the user's real-time image information.

[0026] Next, computer vision technology is used to identify the user's physical features, including the face and body contour, from the real-time image. To this end, the system can use face detection algorithms to locate the face and use posture estimation technology to identify key points of various parts of the body to build a body model of the user. This step aims to obtain the user's overall appearance and posture information to provide basic data for subsequent analysis.

[0027] Then, the recognized person is analyzed to extract the user's body data, including the height, width, depth and relative position of each body part. The system calculates the size and proportion of each body part in order to make necessary adjustments to the virtual model. These body data can reflect the user's body characteristics, such as body shape, weight, height, etc., and convert them into values ​​that can be used to adjust the virtual model.

[0028] After extracting body proportion information from the user's body data, the system will adjust the proportions of the extracted virtual model of the cultural relic based on this data. This includes dynamic adjustment of the overall length, local length, and length-to-width ratio of the virtual model to ensure that the display of the virtual cultural relic on the user is more natural and realistic. For example, if the user is taller, the system will adjust the size of the virtual model accordingly to make it look more in line with the user's body proportions, thereby improving immersion.

[0029] S400, identifying the position and posture of the portrait in the current image, and covering the selected virtual model to the corresponding position of the portrait in the image according to the use position of the virtual model; like Figure 5 As shown, the identifying the position and posture of the portrait in the current image and covering the selected virtual model to the corresponding position of the portrait in the image according to the use position of the virtual model specifically includes: S410, analyzing the recognized portrait, performing posture estimation, and identifying the user's body parts and corresponding positions in space; S420, matching the use position of the virtual model according to the extracted posture information, and making the corresponding parts of the virtual model correspond to the actual body parts of the user; S430, based on the matching result, the virtual model is positioned at the corresponding actual position in the user image, and the coordinates of the virtual model are integrated with the position and posture information of the user.

[0030] The analyzing the recognized portrait, performing posture estimation, and identifying the user's body parts and corresponding positions in space specifically include: Convert the input image into an RGB image and input the RGB image into the MediaPipe pose estimation model for pose estimation to obtain the key points of the human body. Obtain the three-dimensional coordinate data of each key point to obtain the position of the body part in space; Combine the three-dimensional coordinate data of all key points into a two-dimensional array; The two-dimensional array is subjected to principal component analysis using PCA to obtain the rotation quaternion used to describe the rotation and orientation of the body posture.

[0031] The step of matching the use position of the virtual model according to the extracted posture information and making the corresponding parts of the virtual model correspond to the actual body parts of the user specifically includes: Convert the rotation quaternion to generate a 3x3 rotation matrix; The offset from the current position of the virtual model to the three-dimensional coordinates of the key point is used as the translation vector; Construct a transformation matrix using the translation vector and rotation matrix. The first three rows and the first three columns of the transformation matrix are used to store the rotation matrix, the last column of the first three rows is used to store the translation vector, and the last row is [0, 0, 0, 1] to maintain the form of homogeneous coordinates. The current position coordinates of the virtual model are converted into homogeneous coordinates and multiplied with the transformation matrix to obtain transformed homogeneous coordinates; The three-dimensional position coordinates are extracted from the transformed homogeneous coordinates, the position of the virtual model is updated, and the rotation quaternion is applied to the virtual model after the updated position, so that the virtual model matches the position and posture information of the user.

[0032] The overall process of this step focuses on accurately matching and integrating the recognized user posture with the virtual cultural relic model, so as to achieve the natural display of virtual cultural relics in the user's real-time image. The system first analyzes the recognized portrait information and uses the posture estimation algorithm to identify the main parts of the user's body (such as head, shoulders, elbows, knees, etc.) and their specific positions in three-dimensional space. The goal of this stage is to obtain the user's body contour and key points, thereby providing the necessary information for subsequent virtual model matching. Once the user's body parts and positions are obtained, the system will match the use position of the virtual model (such as wearing position, wearing position) with the user's actual body parts based on the extracted posture information. At this point, the system needs to consider the design details of the virtual model to ensure that it can match the shape and movement of the user's body.

[0033] In the virtual model, a coordinate system is defined to describe the positions of each node of the model. At the same time, the user's coordinate system also needs to be defined in order to merge the virtual model with the user's real-time image.

[0034] The system maps the transformed virtual model coordinates to the user's actual image to ensure the correct position of the virtual model in the user's field of view, and uses rendering technology to render the positioned virtual model into the user's real-time image, allowing the user to see virtual artifacts that perfectly match their body in the VR environment.

[0035] Through precise posture recognition and dynamic adaptation of virtual models, users can experience the display of cultural relics that match their own physical characteristics in the virtual environment, enhancing the sense of immersion and reality. Each user has a unique body shape. By dynamically adjusting the virtual model, each user can get a personalized experience. This personalization not only improves user satisfaction, but also promotes the emotional connection between users and cultural relics. At the same time, the process emphasizes real-time, and can quickly adjust the position and proportion of the virtual model according to the user's movements and posture changes to achieve a natural and smooth interactive experience. Combining virtual cultural relics with the user's actual images not only enhances the fun and educational nature of cultural relics display, but also provides an innovative platform for cultural communication and attracts more users to participate. Ultimately, this method combines advanced technologies such as computer vision, virtual reality, and human posture recognition to promote the digital transformation of the tourism industry and provide broad possibilities for future application development.

[0036] S500, identifying the current portrait posture, and synchronously adjusting the virtual model accordingly.

[0037] like Figure 6 As shown, the identification of the current portrait posture and the synchronous adjustment of the virtual model accordingly specifically include: S510, identifying each node of the portrait, and based on this, using vector operations to calculate the angle according to the three-dimensional coordinates of the joints; S520, extracting specific positions of the user's hands, legs, and head in the image, and extracting corresponding three-dimensional position data by obtaining coordinates; S530, adjusting the virtual model in real time according to the recognized posture data so as to make it consistent with the user's movements; S540, applying the calculated joint angles and positions to the virtual model, monitoring the user's movements in real time, and updating the coordinates of the virtual model in real time.

[0038] The main task of this step is to recognize the user's portrait posture, including portrait angle, hand position, leg position, head position and joint bending angle, and adjust the virtual model accordingly to ensure that the virtual artifact matches the user's actual movement. The specific process can be described in detail as follows: First, the system will accurately analyze the recognized portrait and locate the key nodes of the user's body. These key nodes include but are not limited to the head, shoulders, elbows, wrists, knees and ankles. Through computer vision technology, the system can capture the three-dimensional coordinates of these nodes in space in real time and establish a dynamic posture model of the user.

[0039] Next, the system uses vector operations to calculate the angle of the joint. By identifying the relative positions of the various nodes, the system can determine the degree and direction of the joint bending. For example, when the user bends the knee, the system can calculate the relative positions of the three key nodes of the knee, ankle and hip to obtain the angle of the knee joint. This information is crucial for the subsequent adjustment of the virtual model.

[0040] In order to achieve real-time adjustment of the virtual model, the system will also extract the specific positions of the user's hands, legs, and head in the image and obtain their three-dimensional coordinate data. These data include the precise positions of the user's wrists, elbows, shoulders, knees, ankles, etc., thus providing the necessary basis for the adaptation of the virtual model.

[0041] With the recognized posture data, the system can make real-time adjustments to the virtual model. This process involves applying the calculated joint angles and positions to the virtual model to keep it consistent with the user's movements. The system sets the joint points of the virtual model to match the corresponding key nodes of the user, so that the virtual artifact can naturally integrate into the user's movements. For example, when the user raises his arm, the corresponding part of the virtual model will also rise with it, maintaining dynamic consistency.

[0042] In addition, the system monitors the user's movements in real time and continuously updates the coordinates of the virtual model to reflect the user's posture changes. This real-time feedback mechanism ensures that the virtual artifacts are visually highly coordinated with the user's movements and postures, providing a smooth interactive experience.

[0043] In the embodiment of the present invention, positioning the virtual model at the corresponding actual position in the user image according to the matching result, and fusing the coordinates of the virtual model with the user's position and posture information, specifically includes: Taking the midpoint of the image as the origin, extract the user's current position coordinates and posture information, define the coordinate system of the virtual model, and determine the coordinates of each node of the model in its own coordinate system; Extract the initial homogeneous coordinates of the virtual model:

[0044] in, , , is the coordinate position of the virtual model; Define the coordinates of each node of the user:

[0045] in, , , It is the central location for users; Construct a transformation matrix based on the user's posture information, including: Translation matrix: adjusts the spatial position of the virtual model according to the user's position. Specifically:

[0046] in, , ,

[0047] Rotation matrix: adjusts the orientation of the virtual model based on the user's posture. Specifically: ; Scaling Matrix: Sets the scaling factor based on the user's body proportions , specifically:

[0048] Combine the translation matrix, rotation matrix and scaling matrix to form a transformation matrix M

[0049] And apply the composite transformation matrix M to the initial homogeneous coordinates V of the virtual model to obtain the new virtual model coordinates : ; The transformed virtual model coordinates are mapped to the actual image, and the positioned virtual model is rendered into the user's real-time image.

[0050] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0051] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0052] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A tourism industry integration method based on VR, characterized in that: The method comprises the following steps: Establish a cultural relic data information database and store all the cultural relic information existing locally, and establish a virtual model for each cultural relic based on the cultural relic information, and bind the use location to the virtual model of each cultural relic; Obtain the image information uploaded by the user and the marked position of the user in the image, identify the item information existing in the marked position, search it in the cultural relic data information database, identify and extract the type of cultural relic at the marked position and the corresponding virtual model; Collect real-time image information from the user's camera device, identify portrait information, analyze and obtain the body shape data and body proportion data of the person, and dynamically adjust the virtual model of the cultural relic selected by the user based on this; Identify the position and posture of the person in the current image, and overlay the selected virtual model to the corresponding position of the person in the image according to the usage position of the virtual model; Identify the current portrait posture and adjust the virtual model accordingly; The establishment of a cultural relic data information database, storing all the cultural relic information existing locally, establishing a virtual model for each cultural relic based on the cultural relic information, and binding a usage location for the virtual model of each cultural relic specifically includes: Collect local cultural relics information, including images of cultural relics and cultural relics attributes, where the cultural relics attributes include information, type, quality, and size, and classify the collected cultural relics according to type; Based on the collected cultural relic image information, 3D modeling is performed for each cultural relic to generate a 3D virtual model of the cultural relic; Bind the corresponding usage position to the virtual model of each artifact, including the wearing position and the wearing position; A cultural relics data information database is established, and the created virtual models and information of cultural relics are stored in the cultural relics data information database.

2. According to the VR-based tourism industry integration method of claim 1, it is characterized in that: Based on the collected cultural relic image information, 3D modeling is performed for each cultural relic to generate a 3D virtual model of the cultural relic, specifically including: Preprocess the cultural relic images and input the preprocessed cultural relic images into the pre-trained ResNet50 model to obtain image features that represent the key visual information of the images; Construct a 3D mesh of a regular icosahedron based on a sphere; Create a texture map for the 3D mesh, apply the color and detail features of the original image to the 3D mesh, and obtain the initial 3D model of the artifact; Based on Laplace smoothing, the Laplace smoothing loss function is constructed based on the position difference between the vertex and the adjacent vertices in the 3D mesh; Sampling points are obtained by sampling from the initial 3D model of the cultural relic. Based on the Chamfer distance, the Chamfer distance loss function is constructed by taking the mean value between the sampling points and the point cloud of the adjacent image features, and between the point cloud of the image features and the adjacent sampling points; The Laplace smoothing loss function and the Chamfer distance loss function are combined in a weighted sum manner to form a total loss function; The initial 3D model of the cultural relic is iteratively optimized by minimizing the total loss function to gradually adjust the vertex positions of the mesh so that the shape gradually approaches the real form of the cultural relic, and the 3D model of the cultural relic is obtained.

3. The tourism industry integration method based on VR according to claim 2 is characterized in that: The identification of the object information at the marked position, searching the object information in the cultural relic data information database, identifying and extracting the type of cultural relic at the marked position and the corresponding virtual model specifically includes: Get the image information uploaded by the user and read the marked position in the image; Perform edge detection on the objects at the marked position, crop the specific object image, and perform feature detection on the object image to identify the shape, color and texture information of the object; Based on the results of feature detection, they are compared and matched with the information in the cultural relics data database to identify the type of cultural relics to which the item at the marked position belongs, and extract the virtual model of the cultural relic.

4. The VR-based tourism industry integration method according to claim 3 is characterized in that: Perform edge detection on the objects at the marked position, crop the specific object image, and perform feature detection on the object image to identify the shape, color and texture information of the object, including: Convert the image uploaded by the user into a grayscale image, use the Canny algorithm to perform edge detection on the grayscale image, and generate a binary image; The contour detection algorithm is used to obtain all independent contours in the binary image, and the area and perimeter of each contour are calculated to obtain the shape information of the object; Fill the inside of the outline with white and the rest with black to get the mask image; The mask image is used to extract the pixels of the contour area in the color image, and the average color of the pixels in the contour area is calculated to identify the color information of the object; The mask image is used to obtain the contour area in the grayscale image, and the texture of the contour area in the grayscale image is calculated using the local binary pattern to obtain the local texture of the image; the grayscale co-occurrence matrix of the contour area in the grayscale image is calculated to obtain the texture characteristics in the image; and the texture information of the object is constructed based on the local texture of the image and the texture characteristics in the image.

5. The VR-based tourism industry integration method according to claim 4 is characterized in that: The real-time image information of the user's camera device is collected, and the portrait information is recognized, and the body shape data and body proportion data of the person are analyzed and obtained, and based on this, the virtual model of the cultural relic selected by the user is dynamically adjusted, specifically including: Establish a connection with the user's camera device and obtain real-time images captured by the camera device; Use computer vision to identify the user's personal features, including face and body outline, from real-time images; Analyze the identified person to obtain the user's body data, including the height, width, depth and relative positions of various body parts; The body proportion information is extracted from the user's body shape data, and the extracted virtual model of the cultural relic is proportionally adjusted accordingly, including the overall length, local length, and length-to-width ratio.

6. The method for integrating tourism industry based on VR according to claim 5, characterized in that: The identifying the position and posture of the portrait in the current image and covering the selected virtual model to the corresponding position of the portrait in the image according to the use position of the virtual model specifically includes: Analyze the recognized portrait, perform posture estimation, and identify the user's body parts and the corresponding positions in space; According to the extracted posture information, the use position of the virtual model is matched, and the corresponding parts of the virtual model are matched with the actual body parts of the user; According to the matching results, the virtual model is positioned at the corresponding actual position in the user's image, and the coordinates of the virtual model are fused with the user's position and posture information.

7. The method for integrating tourism industry based on VR according to claim 6, characterized in that: The analyzing the recognized portrait, performing posture estimation, and identifying the user's body parts and corresponding positions in space specifically include: Convert the input image into an RGB image and input the RGB image into the MediaPipe pose estimation model for pose estimation to obtain the key points of the human body. Obtain the three-dimensional coordinate data of each key point to obtain the position of the body part in space; Combine the three-dimensional coordinate data of all key points into a two-dimensional array; The two-dimensional array is subjected to principal component analysis using PCA to obtain the rotation quaternion used to describe the rotation and orientation of the body posture.

8. The VR-based tourism industry integration method according to claim 7 is characterized in that: The method of positioning the virtual model at the corresponding actual position in the user image according to the matching result and fusing the coordinates of the virtual model with the user's position and posture information specifically includes: Taking the midpoint of the image as the origin, extract the user's current position coordinates and posture information, define the coordinate system of the virtual model, and determine the coordinates of each node of the model in its own coordinate system; Extract the initial homogeneous coordinates of the virtual model: ; in, , , is the coordinate position of the virtual model; Define the coordinates of each node of the user: ; in, , , It is the central location for users; Construct a transformation matrix based on the user's posture information, including: Translation matrix: adjusts the spatial position of the virtual model according to the user's position. Specifically: ; in, , , ; Rotation matrix: adjusts the orientation of the virtual model based on the user's posture. Specifically: ; Scaling Matrix: Sets the scaling factor based on the user's body proportions , specifically: ; Combine the translation matrix, rotation matrix, and scaling matrix to form a composite transformation matrix M: ; And apply the composite transformation matrix M to the initial homogeneous coordinates V of the virtual model to obtain the new virtual model coordinates : ; The transformed virtual model coordinates are mapped to the actual image, and the positioned virtual model is rendered into the user's real-time image.

9. The VR-based tourism industry integration method according to claim 8 is characterized in that: The identifying the current portrait posture and adjusting the virtual model synchronously accordingly specifically includes: Identify the nodes of the portrait and use vector operations to calculate the angle based on the three-dimensional coordinates of the joints; Extract the specific positions of the user's hands, legs, and head in the image, and extract the corresponding three-dimensional position data by obtaining the coordinates; Based on the recognized posture data, the virtual model is adjusted in real time to keep it consistent with the user's movements; The calculated joint angles and positions are applied to the virtual model, and the user's movements are monitored in real time to update the coordinates of the virtual model in real time.

Citation Information

Patent Citations

  • System and method for realizing virtual disassembly and guidance of cultural relics based on VR technology, and medium

    CN109696961A

  • Digital vow tablet device

    CN115454256A

  • Museum cultural relic AR interaction method, system and device and storage medium

    CN117930978A

  • Metaverse museum system and visitor management method of metaverse museum

    KR1020240077433A

  • Vitamin-added fruit jelly

    KR1020240101746A