Method for realizing element universe online exhibition hall interaction based on VR technology

Through VR technology, the head-mounted display device and pre-trained recognition model are used in the Metaverse online exhibition hall, which improves the efficiency of exhibit information query and user interaction experience, solves the problem of poor user experience, and realizes the adjustment of exhibits based on user preferences Personalized display of location.

CN118915907AActive Publication Date: 2024-11-08SEPU TECH CO LTD

Patent Information

Application Number
CN202410943164.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-08
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

In the Metaverse Online Exhibition Hall, users are inefficient when querying exhibit information, unable to interact with other users or virtual users, and fail to consider the user's attention to exhibits, resulting in a poor user experience.

Method used

Through VR technology, in response to the start operation of the head-mounted display device, obtain the virtual exhibition hall space, determine the user's starting coordinates and generate the user's virtual model, collect the exhibit selection operation, enter the exhibit image to the pre-trained identification model, obtain the exhibit information, and Adjust the exhibit location according to user's historical behavior prediction preferences to achieve real-time interaction and information query.

Benefits of technology

It improves the efficiency of querying exhibit information, enhances the interactive experience between users and virtual users, adjusts the exhibit location according to user preferences, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915907A_ABST
    Figure CN118915907A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a method for realizing element universe online exhibition hall interaction based on a VR technology. A specific embodiment of the method comprises the steps of obtaining a virtual exhibition hall space in response to detection of a starting operation acting on the head-mounted display device; determining a user starting coordinate corresponding to the target user; generating a user virtual model corresponding to the target user, and loading the user virtual model at the user starting coordinate; in response to a selection operation of a target user acting on any virtual exhibit model in the virtual exhibition hall space acquired by the action acquisition equipment, intercepting an exhibit image of the virtual exhibit model; inputting the exhibit image into a pre-trained exhibit identification model to obtain exhibit identification information; and according to the exhibit identification information, determining exhibit information, and displaying the exhibit information in the head-mounted display device. According to the embodiment, the time for querying the exhibit information is shortened, and the query efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a method for realizing the interaction of a metaverse online exhibition hall based on VR technology. Background Art

[0002] The Metaverse is an online virtual space that can interact with the real world. The Metaverse is a physical reality enhanced by virtual reality. It is a 3D virtual space based on the future Internet with connection perception and sharing characteristics, showing convergence and physical persistence. Common virtual reality devices include smartphones, smart glasses, virtual reality head-mounted displays, etc. Users can interact with other users through the Metaverse exhibition hall. At present, when users browse exhibits in the Metaverse exhibition hall, they usually use the following method: obtain the exhibit catalog and check the exhibit information they need to query one by one from the exhibit catalog.

[0003] However, it is found in practice that when browsing exhibits in the above manner, the following technical problems often occur: when checking the exhibit information to be queried one by one, it takes a long time to determine the exhibit information to be queried, and the query efficiency is low.

[0004] In the process of adopting technical solutions to solve the above technical problem one, the following problem often occurs: Technical problem two: users cannot interact with other users or virtual users (for example, NPCs) in the virtual space, and the user experience is poor. For these problems in the above technical problem two, the conventional solution is generally to interact with other users or virtual users in the form of text. However, the above conventional solution still has the following problems: when interacting in the form of text, it is necessary to use an associated text input device, resulting in poor practicality and convenience of the device, and it takes a long time to input text through text interaction, and the interaction efficiency is low.

[0005] In the process of adopting technical solutions to solve the above technical problem 1, the following problems often occur: Technical problem 3: Users have different levels of attention to different exhibits, and the user's attention to virtual exhibits is not taken into account, resulting in a poor user experience. For these problems in the above technical problem 3, the conventional solution is generally to determine the user's preferences and determine the exhibit model based on the preferences. However, the above conventional solution still has the following problems: It does not take into account that user preferences will change over time, resulting in the determined exhibit model not matching the user's preferences, resulting in a poor user experience.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose methods, devices, electronic devices and computer-readable media for implementing Metaverse Online Exhibition Hall interaction based on VR technology to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a method for realizing Metaverse online exhibition hall interaction based on VR technology, the method comprising: in response to detecting a startup operation acting on the above-mentioned head-mounted display device, acquiring a virtual exhibition hall space; determining the user starting coordinates corresponding to the target user; generating a user virtual model corresponding to the above-mentioned target user, and loading the above-mentioned user virtual model at the above-mentioned user starting coordinates; in response to the above-mentioned action acquisition device capturing the selection operation of the above-mentioned target user acting on any virtual exhibit model in the above-mentioned virtual exhibition hall space, intercepting an exhibit image of the above-mentioned virtual exhibit model; inputting the above-mentioned exhibit image into a pre-trained exhibit recognition model to obtain exhibit recognition information; determining the exhibit information based on the above-mentioned exhibit recognition information, and displaying the above-mentioned exhibit information on the above-mentioned head-mounted display device.

[0010] In a second aspect, some embodiments of the present disclosure provide a device for realizing Metaverse online exhibition hall interaction based on VR technology, the device comprising: an acquisition unit, configured to acquire a virtual exhibition hall space in response to detecting a startup operation acting on the above-mentioned head-mounted display device; a first determination unit, configured to determine the user starting coordinates corresponding to the target user; a generation unit, configured to generate a user virtual model corresponding to the above-mentioned target user, and load the above-mentioned user virtual model at the above-mentioned user starting coordinates; an interception unit, configured to intercept an exhibit image of the above-mentioned virtual exhibit model in response to the above-mentioned action acquisition device capturing a selection operation of the above-mentioned target user acting on any virtual exhibit model in the above-mentioned virtual exhibition hall space; an input unit, configured to input the above-mentioned exhibit image into a pre-trained exhibit recognition model to obtain exhibit recognition information; a second determination unit, configured to determine the exhibit information based on the above-mentioned exhibit recognition information, and display the above-mentioned exhibit information in the above-mentioned head-mounted display device.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0013] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the method of realizing the interaction of the Metaverse online exhibition hall based on VR technology in some embodiments of the present disclosure, the time for querying exhibit information is reduced and the query efficiency is improved. Specifically, the reason why it takes a long time to determine the exhibit information to be queried and the query efficiency is low is that when viewing the exhibit information to be queried one by one, it takes a long time to determine the exhibit information to be queried, and the query efficiency is low. Based on this, in some embodiments of the present disclosure, the method of realizing the interaction of the Metaverse online exhibition hall based on VR technology, first, in response to detecting the start operation acting on the above-mentioned head-mounted display device, obtain the virtual exhibition hall space. Thus, the Metaverse exhibition hall can be obtained. Secondly, determine the user starting coordinates corresponding to the target user. Thus, the user's generation coordinates can be determined. Then, generate the user virtual model corresponding to the above-mentioned target user, and load the above-mentioned user virtual model at the above-mentioned user starting coordinates. Thus, the user model can be loaded. Afterwards, in response to the above-mentioned action acquisition device collecting the selection operation of the above-mentioned target user acting on any virtual exhibit model in the above-mentioned virtual exhibition hall space, intercept the exhibit image of the above-mentioned virtual exhibit model. Thus, the image of the exhibit to be queried can be determined. Then, the exhibit image is input into the pre-trained exhibit recognition model to obtain the exhibit recognition information. Thus, the exhibit information of the required exhibit can be determined. Finally, the exhibit information is determined based on the exhibit recognition information, and the exhibit information is displayed in the head-mounted display device. Thus, the exhibit is identified, the time for querying the exhibit information is reduced, and the query efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 It is a flowchart of some embodiments of the method for realizing the interaction of the Metaverse Online Exhibition Hall based on VR technology according to the present disclosure;

[0016] Figure 2 It is a structural schematic diagram of some embodiments of the device for realizing the interaction of the Metaverse Online Exhibition Hall based on VR technology according to the present disclosure;

[0017] Figure 3It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 The process 100 of some embodiments of the method for implementing the interaction of the Metaverse Online Exhibition Hall based on VR technology according to the present disclosure is shown. The method for implementing the interaction of the Metaverse Online Exhibition Hall based on VR technology includes the following steps:

[0025] Step 101: in response to detecting a start-up operation acting on the head mounted display device, a virtual exhibition hall space is acquired.

[0026] In some embodiments, the execution subject (e.g., head-mounted display device) of the method for implementing the interaction of the Metaverse Online Exhibition Hall based on VR technology can obtain a virtual exhibition hall space in response to detecting a startup operation acting on the above-mentioned head-mounted display device. Among them, the above-mentioned method for implementing the interaction of the Metaverse Online Exhibition Hall based on VR technology is applied to a head-mounted display device. The above-mentioned head-mounted display device is connected to a motion acquisition device. The above-mentioned head-mounted display device may be a VR glasses. The above-mentioned motion acquisition device may be a device for collecting the user's behavior and actions. The above-mentioned head-mounted display device may include: a sound receiving device and a camera. The above-mentioned sound receiving device may be used to collect the user's speech. The above-mentioned camera may be used to collect the user's human eye information (e.g., pupil image). The above-mentioned virtual exhibition hall space may be a pre-generated virtual space for displaying virtual exhibits.

[0027] In practice, the virtual exhibition hall space can be generated through the following steps:

[0028] The first step is to obtain an initial virtual space and a set of virtual exhibit models. The initial virtual space may be a pre-generated virtual space. The virtual exhibit models in the set of virtual exhibit models may be pre-set virtual models of exhibits for exhibition.

[0029] In the second step, for each virtual exhibit model in the above virtual exhibit model set, the following processing steps are performed:

[0030] The first processing step is to determine the exhibit location information corresponding to the virtual exhibit model, wherein the exhibit location information may be the preset coordinate information of the virtual exhibit model in the initial virtual space.

[0031] The second processing step is to generate a virtual booth at a position corresponding to the exhibit location information in the initial virtual space, wherein the virtual booth may be a virtual booth for displaying a virtual exhibit model.

[0032] The third processing step is to load the virtual exhibit model into the virtual exhibition stand;

[0033] The third step is to determine the initial virtual space loaded with each virtual exhibit model as the virtual exhibition hall space.

[0034] Considering the problems of the above conventional solutions, facing the above technical problem three: users pay different attention to different exhibits, and fail to consider the user's attention to virtual exhibits, resulting in poor user experience. Combined with the current technical status, we can decide to adopt the following solution:

[0035] Optionally, after step 101, the following steps are further included:

[0036] The first step is to control the camera included in the head mounted display device to perform pupil recognition processing on the target user to generate a pupil recognition result.

[0037] In some embodiments, the execution entity may control the camera included in the head mounted display device to perform pupil recognition processing on the target user to generate a pupil recognition result.

[0038] The second step is to obtain a user history behavior information set of the target user in response to the pupil recognition result satisfying a second preset condition.

[0039] In some embodiments, the execution subject may obtain the user history behavior information set of the target user in response to the pupil recognition result satisfying the second preset condition. The user history behavior information in the user history behavior information set includes the time point of the history behavior. The second preset condition may indicate that the target user has history access data.

[0040] The third step is to perform user preference prediction processing on the target user for each user historical behavior information in the user historical behavior information set according to the user historical behavior information to obtain a user preference tag group.

[0041] In some embodiments, the execution subject may perform user preference prediction processing on the target user based on each user historical behavior information in the user historical behavior information set to obtain a user preference tag group. The user preference tags in the user preference tag group represent the target user's preference for the virtual exhibit model. In practice, the execution subject may use a collaborative filtering algorithm to perform user preference prediction processing on the user historical behavior information.

[0042] The fourth step is to merge the obtained user preference tag groups to generate a merged user preference tag group.

[0043] In some embodiments, the execution entity may merge the obtained user preference tag groups to generate a merged user preference tag group.

[0044] In the fifth step, deduplication processing is performed on each merged user preference tag included in the merged user preference tag group to generate a deduplication user preference tag group.

[0045] In some embodiments, the execution entity may perform deduplication processing on each merged user preference tag included in the merged user preference tag group to generate a deduplication user preference tag group.

[0046] Step 6: For each deduplicated user preference tag in the deduplicated user preference tag group, perform the following determination steps:

[0047] The first determination step is to determine the target time point and tag frequency corresponding to the above-mentioned user preference tags after deduplication.

[0048] In some embodiments, the execution entity may determine the target time point and label frequency corresponding to the deduplicated user preference label. The target time point may be the historical behavior time point included in the user historical behavior information corresponding to the deduplicated user preference label. The label frequency may be the number of times the deduplicated user preference label appears in the merged user preference label group.

[0049] The second determination step is to determine the preference tag weight corresponding to the above-mentioned user preference tag after deduplication according to the above-mentioned target time point and tag frequency.

[0050] In some embodiments, the execution entity may determine the preference tag weight corresponding to the user preference tag after deduplication according to the target time point and tag frequency.

[0051] In the seventh step, the deduplicated user preference tags included in the deduplicated user preference tag group are sorted according to the determined weights of the preference tags to generate a deduplicated user preference tag sequence.

[0052] In some embodiments, the execution subject may sort the deduplicated user preference tags included in the deduplicated user preference tag group according to the determined weights of the preference tags to generate a deduplicated user preference tag sequence. The sorting may be to sort the deduplicated user preference tags in descending order according to the corresponding preference tag weights.

[0053] In the eighth step, the virtual exhibit models in the virtual exhibition hall space are swapped according to the deduplicated user preference tag sequence to generate a swapped virtual exhibition hall space as the virtual exhibition hall space.

[0054] In some embodiments, the execution entity may perform position swap processing on each virtual exhibit model in the virtual exhibition hall space according to the deduplicated user preference tag sequence to generate a swapped virtual exhibition hall space as the virtual exhibition hall space.

[0055] The first step to the eighth step are an inventive point of an embodiment of the present disclosure, which solves the technical problem three mentioned in the background technology, "users have different attention to different exhibits, and the user's attention to virtual exhibits is not taken into account, resulting in a poor user experience." The reasons for the poor user experience are as follows: users have different attention to different exhibits, and the user's attention to virtual exhibits is not taken into account, resulting in a poor user experience. If the above factors are solved, the effect of improving the user experience can be achieved. In order to achieve this effect, the present disclosure first controls the camera included in the head-mounted display device to perform pupil recognition processing on the target user to generate a pupil recognition result. Thus, it can be determined whether the target user has entered the virtual exhibit space. Second, in response to the pupil recognition result satisfying the second preset condition, the user history behavior information set of the target user is obtained. Thus, the historical behavior of the target user can be obtained. Third, for each user history behavior information in the user history behavior information set, the user preference prediction processing is performed on the target user according to the user history behavior information to obtain a user preference tag group. Thus, the user preference of the target user can be determined. Fourth, the obtained user preference tag groups are merged to generate a merged user preference tag group; the merged user preference tags included in the merged user preference tag group are deduplicated to generate a deduplicated user preference tag group. Thus, each preference tag can be deduplicated. Fifth, for each deduplicated user preference tag in the deduplicated user preference tag group, the following determination steps are performed: determine the target time point and tag frequency corresponding to the deduplicated user preference tag; determine the preference tag weight corresponding to the deduplicated user preference tag according to the target time point and tag frequency. Thus, the weight corresponding to each preference tag can be determined. Sixth, according to the determined weights of each preference tag, sort the deduplicated user preference tags included in the deduplicated user preference tag group to generate a deduplicated user preference tag sequence; according to the deduplicated user preference tag sequence, perform position swapping on each virtual exhibit model in the virtual exhibition hall space to generate a swapped virtual exhibition hall space as the virtual exhibition hall space. Therefore, the positions of the virtual exhibit models on display can be exchanged through the preference tags and the corresponding weights, so that users can watch the virtual exhibit models that have a high degree of user attention, thereby improving the user experience.

[0056] Step 102: Determine the user starting coordinates corresponding to the target user.

[0057] In some embodiments, the execution entity may determine the user starting coordinates corresponding to the target user. The user starting coordinates may be the coordinates loaded by the target user into the virtual exhibition hall space. The target user may be a user wearing the head mounted display device.

[0058] In practice, the above execution entity can determine the user starting coordinates corresponding to the target user through the following steps:

[0059] The first step is to obtain virtual space configuration information corresponding to the initial virtual space, wherein the virtual space configuration information may be parameter information for generating the initial virtual space.

[0060] The second step is to parse the above virtual space configuration information to generate a parsing result.

[0061] The third step is to determine the virtual space starting coordinates included in the initial virtual space according to the above analysis results. The virtual space starting coordinates may be the starting coordinates set in the initial virtual space.

[0062] The fourth step is to map the user position coordinates of the target user to the initial virtual space to obtain mapping coordinates.

[0063] The fifth step is to match the virtual space starting coordinates with the mapping coordinates to determine the virtual space coordinates as the user starting coordinates. The matching process may be to overlap the virtual space starting coordinates with the mapping coordinates to generate the user starting coordinates.

[0064] Step 103: Generate a user virtual model corresponding to the target user, and load the user virtual model at the user's starting coordinates.

[0065] In some embodiments, the execution entity may generate a user virtual model corresponding to the target user, and load the user virtual model at the user starting coordinates. The user virtual model may be used to characterize the target user in the virtual exhibit space. In practice, the execution entity may place the user virtual model at the user starting coordinates.

[0066] Optionally, after step 103, the following steps are further included:

[0067] The first step is to control the above-mentioned action collection device to collect the user behavior information of the above-mentioned target user in real time.

[0068] In some embodiments, the execution entity may control the action acquisition device to acquire user behavior information of the target user in real time.

[0069] The second step is to perform behavior recognition processing on the above user behavior information to generate user behavior recognition information.

[0070] In some embodiments, the execution subject may perform behavior recognition processing on the user behavior information to generate user behavior recognition information. In practice, the user behavior information may be input into a pre-trained behavior recognition model to obtain the user behavior recognition information.

[0071] The third step is to select a preset user behavior model corresponding to the above user behavior identification information from the preset user behavior model library.

[0072] In some embodiments, the execution entity may select a preset user behavior model corresponding to the user behavior identification information from a preset user behavior model library.

[0073] The fourth step is to adjust the behavior of the user virtual model according to the preset user behavior model.

[0074] In some embodiments, the execution entity may perform behavior adjustment processing on the user virtual model according to the preset user behavior model.

[0075] Step 104 : in response to the action acquisition device acquiring the selection operation of the target user on any virtual exhibit model in the virtual exhibition hall space, capturing an exhibit image of the virtual exhibit model.

[0076] In some embodiments, the execution subject may capture an exhibit image of the virtual exhibit model in response to the action acquisition device acquiring a selection operation of the target user on any virtual exhibit model in the virtual exhibition hall space. The selection operation may include but is not limited to a click operation. The exhibit image may be an image used to display the virtual exhibit model.

[0077] Step 105 , input the exhibit image into a pre-trained exhibit recognition model to obtain exhibit recognition information.

[0078] In some embodiments, the execution subject may input the exhibit image into a pre-trained exhibit recognition model to obtain exhibit recognition information. The exhibit recognition model may be a pre-trained classification model that takes the exhibit image as input and the exhibit recognition information as output. The exhibit recognition information may uniquely represent a virtual exhibit model. For example, the exhibit recognition information may be the encoding of the virtual exhibit model.

[0079] Optionally, the exhibit recognition model can be trained through the following steps:

[0080] The first step is to obtain a sample set.

[0081] In some embodiments, the execution subject may obtain a sample set, wherein the samples in the sample set include sample exhibit images and sample exhibit identification information corresponding to the sample exhibit images.

[0082] The second step is to select samples from the above sample set.

[0083] In some embodiments, the execution entity may select a sample from the sample set. Here, the execution entity may randomly select a sample from the sample set.

[0084] The third step is to input the above samples into the initial network model to obtain the exhibit identification information corresponding to the above samples.

[0085] In some embodiments, the execution subject may input the sample into an initial network model to obtain exhibit identification information corresponding to the sample. The initial neural network may be a classification model that can obtain exhibit identification information based on the exhibit image. The initial neural network may be a classification model.

[0086] The fourth step is to determine the loss value between the exhibit identification information and the sample exhibit identification information included in the sample.

[0087] In some embodiments, the execution subject may determine a loss value between the exhibit identification information and the sample exhibit identification information included in the sample. In practice, the loss value between the exhibit identification information and the sample exhibit identification information included in the sample may be determined based on a preset loss function. For example, the preset loss function may be a cross entropy loss function.

[0088] In the fifth step, in response to the loss value being greater than or equal to a preset threshold, the network parameters of the initial network model are adjusted.

[0089] In some embodiments, the execution subject may adjust the network parameters of the initial network model in response to the loss value being greater than or equal to a preset threshold. Here, there is no restriction on the setting of the preset threshold. For example, the difference between the loss value and the preset threshold may be calculated to obtain the loss difference. On this basis, the error value is passed forward from the last layer of the model using back propagation, stochastic gradient descent and other methods to adjust the parameters of each layer. Of course, as needed, the network freezing (dropout) method may also be used to keep the network parameters of some layers unchanged and not adjust them. There is no limitation on this.

[0090] Optionally, in response to the loss value being less than the preset threshold, the initial network model is determined as an exhibit recognition model.

[0091] In some embodiments, the execution entity may determine the initial network model as an exhibit recognition model in response to the loss value being less than the preset threshold.

[0092] Step 106 , determining the exhibit information according to the exhibit identification information, and displaying the exhibit information on the head mounted display device.

[0093] In some embodiments, the execution subject may determine the exhibit information based on the exhibit identification information, and display the exhibit information in the head mounted display device. In practice, the exhibit information corresponding to the exhibit identification information may be obtained from a target database. The target database may be a database storing exhibit information. The exhibit information may be information used to describe a virtual exhibit model.

[0094] Considering the problems of the above conventional solutions, facing the above technical problem 2: users cannot interact with other users or virtual users in the virtual space. Combined with the current technical status, we can decide to adopt the following solution:

[0095] Optionally, after step 106, the following steps are further included:

[0096] The first step is to detect in real time whether the user virtual model corresponding to the target user meets the first preset condition.

[0097] In some embodiments, the execution subject may detect in real time whether the user virtual model corresponding to the target user satisfies a first preset condition, wherein the first preset condition may be that the user virtual model is within a preset range centered on the location of the virtual user.

[0098] In the second step, in response to detecting that the user virtual model satisfies the first preset condition, the sound receiving device included in the head mounted display device is controlled to collect the audio data of the target user in real time.

[0099] In some embodiments, the execution entity may, in response to detecting that the user virtual model satisfies the first preset condition, control the sound receiving device included in the head mounted display device to collect audio data of the target user in real time.

[0100] The third step is to pre-process the audio data to generate pre-processed audio data, wherein the pre-processing may be noise reduction processing on the audio data.

[0101] The fourth step is to perform audio recognition processing on the pre-processed audio data to generate audio recognition information.

[0102] In some embodiments, the execution subject may perform audio recognition processing on the pre-processed audio data to generate audio recognition information. The audio recognition processing may be inputting the pre-processed audio data into a pre-trained audio recognition model to obtain the audio recognition information.

[0103] The fifth step is to input the above audio recognition information into a pre-trained large language model to obtain audio response information.

[0104] In some embodiments, the execution subject may input the audio recognition information into a pre-trained large language model to obtain audio response information, wherein the large language model may be a large language model deployed in a local server.

[0105] The sixth step is to input the above audio response information into the pre-trained speech generation model to obtain the response audio.

[0106] In some embodiments, the above-mentioned execution entity can input the above-mentioned audio response information into a pre-trained speech generation model to obtain a response audio.

[0107] The seventh step is to play the above reply audio and control the preset user model to match the above reply audio.

[0108] In some embodiments, the execution subject may play the reply audio and control the preset user model to match the reply audio. In practice, the face model of the preset user model may be matched according to the reply audio.

[0109] The first step to the seventh step are an inventive point of an embodiment of the present disclosure, which solves the technical problem three mentioned in the background technology, "users cannot interact with other users or virtual users in the virtual space, and the user experience is poor". The reasons for the poor user experience are as follows: users cannot interact with other users or virtual users in the virtual space, and the user experience is poor. If the above factors are solved, the effect of improving the user experience can be achieved. In order to achieve this effect, the present disclosure first detects in real time whether the user virtual model corresponding to the target user meets the first preset condition. Thus, it can be determined whether the user is within the conversation range of the virtual user. Second, in response to detecting that the user virtual model meets the first preset condition, the sound receiving device included in the head-mounted display device is controlled to collect the audio data of the target user in real time. Thus, the user's conversation audio can be recorded. Third, the audio data is preprocessed to generate preprocessed audio data. Thus, it is easy to recognize the user's conversation audio. Fourth, the preprocessed audio data is subjected to audio recognition processing to generate audio recognition information; the audio recognition information is input into a pre-trained large language model to obtain audio reply information. Thus, the reply content of the virtual user can be determined. Fifth, the above audio reply information is input into the pre-trained speech generation model to obtain the reply audio. Thus, the reply audio can be generated. Sixth, the above reply audio is played and processed, and the preset user model is controlled to match the above reply audio. Thus, the dialogue with the target user and the virtual user is completed, and the user experience is improved.

[0110] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the method of realizing the interaction of the Metaverse online exhibition hall based on VR technology in some embodiments of the present disclosure, the time for querying exhibit information is reduced and the query efficiency is improved. Specifically, the reason why it takes a long time to determine the exhibit information to be queried and the query efficiency is low is that when viewing the exhibit information to be queried one by one, it takes a long time to determine the exhibit information to be queried, and the query efficiency is low. Based on this, in some embodiments of the present disclosure, the method of realizing the interaction of the Metaverse online exhibition hall based on VR technology, first, in response to detecting the start operation acting on the above-mentioned head-mounted display device, obtain the virtual exhibition hall space. Thus, the Metaverse exhibition hall can be obtained. Secondly, determine the user starting coordinates corresponding to the target user. Thus, the user's generation coordinates can be determined. Then, generate the user virtual model corresponding to the above-mentioned target user, and load the above-mentioned user virtual model at the above-mentioned user starting coordinates. Thus, the user model can be loaded. Afterwards, in response to the above-mentioned action acquisition device collecting the selection operation of the above-mentioned target user acting on any virtual exhibit model in the above-mentioned virtual exhibition hall space, intercept the exhibit image of the above-mentioned virtual exhibit model. Thus, the image of the exhibit to be queried can be determined. Then, the exhibit image is input into the pre-trained exhibit recognition model to obtain the exhibit recognition information. Thus, the exhibit information of the required exhibit can be determined. Finally, the exhibit information is determined based on the exhibit recognition information, and the exhibit information is displayed in the head-mounted display device. Thus, the exhibit is identified, the time for querying the exhibit information is reduced, and the query efficiency is improved.

[0111] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a device for implementing the interaction of the Metaverse Online Exhibition Hall based on VR technology. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device for realizing the interaction of the Metaverse online exhibition hall based on VR technology can be specifically applied to various electronic devices.

[0112] like Figure 2As shown, in some embodiments, the device 200 for realizing the interaction of the Metaverse online exhibition hall based on VR technology includes: an acquisition unit 201, a first determination unit 202, a generation unit 203, an interception unit 204, an input unit 205 and a second determination unit 206. Among them, the acquisition unit 201 is configured to acquire the virtual exhibition hall space in response to detecting a startup operation acting on the above-mentioned head-mounted display device; the first determination unit 202 is configured to determine the user starting coordinates corresponding to the target user; the generation unit 203 is configured to generate a user virtual model corresponding to the above-mentioned target user, and load the above-mentioned user virtual model at the above-mentioned user starting coordinates; the interception unit 204 is configured to intercept the exhibit image of the above-mentioned virtual exhibit model in response to the above-mentioned action acquisition device capturing the selection operation of the above-mentioned target user acting on any virtual exhibit model in the above-mentioned virtual exhibition hall space; the input unit 205 is configured to input the above-mentioned exhibit image into a pre-trained exhibit recognition model to obtain exhibit recognition information; the second determination unit 206 is configured to determine the exhibit information according to the above-mentioned exhibit recognition information, and display the above-mentioned exhibit information in the above-mentioned head-mounted display device.

[0113] It is understandable that the units recorded in the device 200 for realizing the interaction of the Metaverse Online Exhibition Hall based on VR technology are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 for realizing the interaction of the Metaverse Online Exhibition Hall based on VR technology and the units contained therein, and will not be repeated here.

[0114] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0115] like Figure 3As shown, the electronic device 300 may include a processing device 301 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0116] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0117] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0118] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0119] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0120] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to detecting a startup operation acting on the head-mounted display device, obtains a virtual exhibition hall space. Determines the user starting coordinates corresponding to the target user. Generates a user virtual model corresponding to the target user, and loads the user virtual model at the user starting coordinates. In response to the action acquisition device capturing the selection operation of the target user acting on any virtual exhibit model in the virtual exhibition hall space, captures the exhibit image of the virtual exhibit model. Inputs the exhibit image into a pre-trained exhibit recognition model to obtain exhibit recognition information. Determines the exhibit information based on the exhibit recognition information, and displays the exhibit information in the head-mounted display device.

[0121] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising an acquisition unit, a first determination unit, a generation unit, an interception unit, an input unit, and a second determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as "a unit for acquiring a virtual exhibition hall space in response to detecting a startup operation acting on the above-mentioned head-mounted display device".

[0124] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0125] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for realizing the interaction of Metaverse online exhibition hall based on VR technology, applied to a head-mounted display device, wherein: The head mounted display device is connected to a motion acquisition device, and the method comprises: In response to detecting a start-up operation acting on the head mounted display device, acquiring a virtual exhibition hall space; Determine the user starting coordinates corresponding to the target user; Generate a user virtual model corresponding to the target user, and load the user virtual model at the user starting coordinates; In response to the action acquisition device acquiring a selection operation of the target user on any virtual exhibit model in the virtual exhibition hall space, capturing an exhibit image of the virtual exhibit model; Inputting the exhibit image into a pre-trained exhibit recognition model to obtain exhibit recognition information; The exhibit information is determined according to the exhibit identification information, and the exhibit information is displayed on the head mounted display device.

2. The method according to claim 1, wherein: The virtual exhibition hall space is generated by the following steps: Obtaining an initial set of virtual space and virtual exhibit models; For each virtual exhibit model in the virtual exhibit model set, the following processing steps are performed: Determining exhibit location information corresponding to the virtual exhibit model; Generating a virtual exhibition stand at a position corresponding to the exhibit location information in the initial virtual space; Loading the virtual exhibit model into the virtual exhibition stand; The initial virtual space loaded with each virtual exhibit model is determined as the virtual exhibition hall space.

3. The method according to claim 1, wherein: The determining of the user starting coordinates corresponding to the target user includes: Acquire virtual space configuration information corresponding to the initial virtual space; Parsing the virtual space configuration information to generate a parsing result; Determining the virtual space starting coordinates included in the initial virtual space according to the analysis result; Mapping the user position coordinates of the target user into the initial virtual space to obtain mapping coordinates; The virtual space starting coordinates and the mapping coordinates are matched to determine the virtual space coordinates as the user starting coordinates.

4. The method according to claim 1, wherein: After generating the user virtual model corresponding to the target user and loading the user virtual model at the user starting coordinates, the method further includes: Controlling the action acquisition device to acquire user behavior information of the target user in real time; Performing behavior recognition processing on the user behavior information to generate user behavior recognition information; Selecting a preset user behavior model corresponding to the user behavior identification information from a preset user behavior model library; According to the preset user behavior model, behavior adjustment processing is performed on the user virtual model.

5. The method according to claim 1, wherein: The exhibit recognition model is trained by the following steps: Acquire a sample set, wherein the samples in the sample set include sample exhibit images and sample exhibit identification information corresponding to the sample exhibit images; Selecting a sample from the sample set; Inputting the sample into the initial network model to obtain exhibit identification information corresponding to the sample; determining a loss value between the exhibit identification information corresponding to the sample and the sample exhibit identification information included in the sample; In response to the loss value being greater than or equal to a preset threshold, adjusting the network parameters of the initial network model.

6. The method according to claim 5, wherein: The method further comprises: In response to the loss value being less than the preset threshold, the initial network model is determined as an exhibit recognition model.

7. A device for realizing the interaction of Metaverse online exhibition hall based on VR technology, comprising: an acquisition unit, configured to acquire a virtual exhibition hall space in response to detecting a start-up operation acting on the head mounted display device; A first determining unit is configured to determine a user starting coordinate corresponding to a target user; A generating unit, configured to generate a user virtual model corresponding to the target user, and load the user virtual model at the user starting coordinates; a capture unit configured to capture an exhibit image of the virtual exhibit model in response to the action capture device capturing a selection operation of the target user on any virtual exhibit model in the virtual exhibition hall space; An input unit, configured to input the exhibit image into a pre-trained exhibit recognition model to obtain exhibit recognition information; The second determining unit is configured to determine the exhibit information according to the exhibit identification information, and display the exhibit information in the head mounted display device.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Exhibition information watching system and method in virtual reality environment

    CN114511229A

  • Exhibition system based on meta universe, interaction method and electronic equipment

    CN115857704A

  • Interaction method based on network virtual exhibition hall

    CN115981516A

  • Virtual customer service display method and device based on meta universe, equipment and medium

    CN116704162A

  • Exhibit information display method and device based on cloud platform, equipment and medium

    CN117235395A

Cited By

  • Data synchronization method, collaboration system, head-mounted display device, medium and product

    CN119917308A