Mixed reality space interaction method, mixed reality terminal and head-mounted display equipment

Abstract

The embodiment of the invention discloses a mixed reality space interaction method, a mixed reality end and head-mounted display equipment. A specific embodiment of the method comprises the following steps: in response to a detected selection operation of a user on a scanning button, displaying a scanning interface on a head-mounted display device, and performing graph code scanning; loading a mixed reality scene corresponding to the scanned graph code in response to determining that the graph code is successfully scanned; in response to determining that the mixed reality scene is successfully loaded, determining whether the real space scene is matched with the mixed reality scene, and displaying a matching interface on the head-mounted display device; and in response to determining that the real space scene is matched with the mixed reality scene, displaying the mixed reality scene on the head-mounted display equipment. According to the embodiment, the interaction modes of the user and the mixed reality space are enriched, and immersive interaction experience can be provided for the user.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of mixed reality technology, and in particular to a mixed reality space interaction method, a mixed reality terminal, and a head-mounted display device. Background Art

[0002] Mixed reality (MR) space is a mixture of physical space (or real space) and digital space. It links digital space and physical space in some form. Users can browse the mixed reality space through mixed reality devices and observe the physical space and the digital content associated with it at the same time. Mixed reality space can be used in mixed reality guided tours (MR guided tours), for example, it can be applied to digital exhibition halls, and users can experience the combined virtual and real guided tour effect while visiting the exhibition hall.

[0003] However, in the existing MR guided tour process, the feature points of the physical space are identified through the camera of the mixed reality device (for example, MR glasses), and after matching the feature information, a piece of audio and video material is played through the mixed reality device. However, this type of guided tour provides limited interaction methods, and there is no clear connection between the digital space and the physical space, which cannot give users an immersive experience. At present, there is a lack of a dedicated method and tool to achieve immersive interaction between browsing users and MR space.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a mixed reality space interaction method, a mixed reality terminal, and a head-mounted display device to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a mixed reality space interaction method, which is applied to a mixed reality terminal, wherein the mixed reality terminal includes a head-mounted display device, the head-mounted display device includes a processor and two display modules, corresponding to the left and right eyes of a user respectively, and the processor is integrated with a mixed reality capability module, the method comprising: in response to detecting a user's selection operation on a scan button, displaying a scanning interface on the head-mounted display device, and performing a code scan, wherein the content displayed on the scanning interface has a first preset number of degrees of freedom and a first preset display distance; in response to determining that the code scan is successful, loading a mixed reality scene corresponding to the scanned code; in response to determining that the mixed reality scene is loaded successfully, determining whether the real space scene matches the mixed reality scene, and displaying a matching interface on the head-mounted display device, wherein the matching interface displays a dynamic space model and matching guidance information, and the dynamic space model has a first preset number of degrees of freedom and a first preset display distance; in response to determining that the real space scene matches the mixed reality scene, displaying the mixed reality scene on the head-mounted display device, wherein the degrees of freedom and display distance of the displayed content of the head-mounted display device are generated by the mixed reality capability module of the mixed reality terminal.

[0008] Optionally, before the above-mentioned response to detecting the user's selection operation on the scan button, the method also includes: in response to detecting that the target application is turned on, displaying the initial page of the above-mentioned target application on the above-mentioned head-mounted display device, wherein the initial page of the above-mentioned target application displays the three-dimensional model and the scan button of the above-mentioned target application, the three-dimensional model and the scan button of the above-mentioned target application have a second preset number of degrees of freedom, the three-dimensional model of the above-mentioned target application corresponds to a first preset display distance, and the above-mentioned scan button corresponds to a second preset display distance.

[0009] Optionally, after the above response to determining that the code scan is successful, the method also includes: displaying a data loading interface on the above head-mounted display device, wherein the content displayed on the above data loading interface has a first preset number of degrees of freedom, and the content displayed on the above data loading interface corresponds to a first preset display distance.

[0010] Optionally, the above-mentioned loading of the mixed reality scene corresponding to the scanned code includes: requesting the mixed reality scene corresponding to the scanned code from an associated server; in response to determining that the request for the above-mentioned mixed reality scene is successful, determining whether to turn on spatial occlusion; in response to determining that spatial occlusion is turned on, determining whether standardized spatial map resources of the above-mentioned mixed reality scene are downloaded; in response to determining that the standardized spatial map resources of the above-mentioned mixed reality scene have been downloaded, determining that the above-mentioned mixed reality scene has been loaded successfully.

[0011] Optionally, the above-mentioned loading of the mixed reality scene corresponding to the scanned code also includes: in response to determining that the standardized spatial map resources of the above-mentioned mixed reality scene have not been downloaded, downloading the standardized spatial map resources of the above-mentioned mixed reality scene; in response to determining that the standardized spatial map resources of the above-mentioned mixed reality scene have been downloaded successfully, determining that the above-mentioned mixed reality scene has been loaded successfully; in response to determining that the standardized spatial map resources of the above-mentioned mixed reality scene have failed to be downloaded, displaying a re-download prompt message on the above-mentioned head-mounted display device.

[0012] Optionally, the above-mentioned determination of whether the real space scene matches the above-mentioned mixed reality scene includes: in response to determining that spatial occlusion is turned on, determining whether the position information of the above-mentioned mixed reality end matches the position configuration information of the above-mentioned mixed reality scene; in response to determining that the position information of the above-mentioned mixed reality end matches the position configuration information of the above-mentioned mixed reality scene, determining that the real space scene matches the above-mentioned mixed reality scene.

[0013] Optionally, the above-mentioned determination of whether the real space scene matches the above-mentioned mixed reality scene also includes: in response to determining that spatial occlusion is not turned on, executing the following steps: determining whether the above-mentioned mixed reality scene is configured with an anchor map; in response to determining that the above-mentioned mixed reality scene is configured with an anchor map, shrinking the anchor map corresponding to the above-mentioned mixed reality scene from the center position of the above-mentioned matching interface and displaying it at a preset position of the above-mentioned matching interface, wherein the above-mentioned anchor map corresponds to a second preset display distance; determining each feature point of the real space scene, and determining each feature point of the anchor map corresponding to the above-mentioned mixed reality scene; in response to determining that each feature point of the real space scene matches each feature point of the above-mentioned anchor map, determining that the real space scene matches the above-mentioned mixed reality scene.

[0014] Optionally, after the above-mentioned response to determining that the above-mentioned mixed reality scene is loaded successfully, the method also includes: in response to detecting that the user's interaction information satisfies the preset trigger condition of any scene element in the selected mixed reality scene, determining whether the above-mentioned any scene element is cached; in response to determining that the above-mentioned any scene element is cached, triggering the media information corresponding to the above-mentioned any scene element in the above-mentioned mixed reality scene; in response to determining that the above-mentioned any scene element is not cached, performing the following steps: downloading the above-mentioned any scene element; in response to determining that the above-mentioned any scene element is downloaded successfully, triggering the media information corresponding to the above-mentioned any scene element in the above-mentioned mixed reality scene; in response to determining that the download of the above-mentioned any scene element fails, displaying re-download guidance information on the above-mentioned head-mounted display device.

[0015] Optionally, the method also includes: in response to determining that the above-mentioned real space scene does not match the above-mentioned mixed reality scene, determining the reason for the matching failure, and displaying a matching failure interface, wherein the above-mentioned matching failure interface displays a positioning picture, matching failure prompt information and a matching retry button, and the content displayed on the above-mentioned matching failure interface has a first preset number of degrees of freedom, the above-mentioned positioning picture corresponds to a first preset display distance, and the above-mentioned matching retry button corresponds to a second preset display distance.

[0016] Optionally, after the above response to determining that the real space scene matches the above mixed reality scene, the method also includes: displaying a successful matching interface on the above head-mounted display device, and playing a successful matching prompt message, wherein the successful matching interface displays a successful matching prompt message, and the successful matching prompt message has a first preset value of degrees of freedom and a first preset display distance.

[0017] In a second aspect, some embodiments of the present disclosure provide a mixed reality terminal, comprising: a head-mounted display device for allowing a user to browse a mixed reality scene and media information; a processor, the processor being connected to the head-mounted display device; the processor being configured to execute the method described in any implementation of the first aspect.

[0018] In a third aspect, some embodiments of the present disclosure provide a head-mounted display device for browsing a mixed reality space, comprising: a frame, the frame having a glasses structure or a headband structure, and having a space for accommodating a circuit structure; one or more display modules, the display module being embedded in the glasses frame, the display module comprising a semi-transparent and semi-reflective display unit, so that the user can see the real environment and the display content at the same time, wherein the real environment and the display content constitute a mixed reality space; a posture sensor, the posture sensor being embedded in the circuit structure and configured to obtain posture data of the user's head; a grayscale camera, the grayscale camera being integrated in the circuit structure and used to collect images of environmental data; a processor, the processor being integrated in the circuit structure or separately arranged on the host of the head-mounted display device, the processor being configured to configure different degrees of freedom, display distances and interaction methods according to the display page requirements of the mixed reality space, the interaction methods including gesture interaction, voice interaction, and button interaction.

[0019] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0020] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the mixed reality space interaction method of some embodiments of the present disclosure, the interaction mode between the user and the mixed reality space is enriched, and an immersive interaction experience can be provided for the user. Specifically, the reason for not being able to provide the user with an immersive interaction experience is that in the existing MR navigation process, the feature points of the physical space are identified by the camera of the mixed reality device (for example, MR glasses), and after matching the feature information, a section of audio and video data is played through the mixed reality device. However, this type of navigation method provides limited interaction methods, and there is no clear connection between the digital space and the physical space, which cannot give the user an immersive experience. At present, there is still a lack of a dedicated method and tool to realize the immersive interaction between the browsing user and the MR space. Based on this, the mixed reality space interaction method of some embodiments of the present disclosure, first, in response to detecting the user's selection operation of the scan button, the scanning interface is displayed on the head-mounted display device, and the code scanning is performed, wherein the content displayed on the scanning interface has a first preset number of degrees of freedom and a first preset display distance. Thus, after the user selects the scan button, the code displayed in the scanning interface can be identified. Then, in response to determining that the code scan is successful, the mixed reality scene corresponding to the scanned code is loaded. Thus, after the code is scanned successfully, the mixed reality scene corresponding to the code can be obtained. Afterwards, in response to determining that the mixed reality scene is loaded successfully, it is determined whether the real space scene matches the mixed reality scene, and the matching interface is displayed on the head-mounted display device, wherein the matching interface displays a dynamic space model and matching guidance information, and the dynamic space model has a first preset number of degrees of freedom and a first preset display distance. Thus, it can be determined whether the real space scene matches the mixed reality scene. Finally, in response to determining that the real space scene matches the mixed reality scene, the mixed reality scene is displayed on the head-mounted display device, wherein the degrees of freedom and display distance of the display content of the head-mounted display device are generated by the mixed reality capability module of the mixed reality end. Thus, the user can browse the mixed reality scene. Also because of the implemented mixed reality space interaction method, different degrees of freedom and display distances can be used in different interaction interfaces and different display contents. Thereby, the interaction mode between the user and the mixed reality space is enriched, and an immersive interactive experience can be provided for the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0022] Figure 1is a flow chart of some embodiments of the mixed reality space interaction method according to the present disclosure;

[0023] Figure 2 are flow charts of other embodiments of the mixed reality space interaction method according to the present disclosure;

[0024] Figure 3 is a schematic diagram of the structure of some embodiments of the mixed reality terminal according to the present disclosure;

[0025] Figure 4 It is a schematic diagram of the hardware structure of a head-mounted display device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0028] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0031] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0032] Figure 1 The process 100 of some embodiments of the mixed reality space interaction method according to the present disclosure is shown. The mixed reality space interaction method comprises the following steps:

[0033] Step 101, in response to detecting a user's selection operation on a scan button, displaying a scan interface on a head mounted display device and performing a code scan.

[0034] In some embodiments, the execution subject (e.g., head-mounted display device) of the mixed reality space interaction method may respond to detecting the user's selection operation of the scan button, display a scanning interface on the head-mounted display device, and perform a code scan. Among them, the mixed reality space interaction method can be applied to a mixed reality end. The mixed reality end may include a head-mounted display device. The head-mounted display device may include a processor and two display modules. The two display modules may correspond to the left and right eyes of the user, respectively. The processor may be integrated with a mixed reality capability module. The mixed reality end may be a device for loading a mixed reality scene. The head-mounted display device may be a device for loading a mixed reality scene. For example, the head-mounted display device may be MR glasses. The mixed reality capability module may be a software development kit that integrates some AR capabilities. The mixed reality capability module may be integrated into the software system running on the head-mounted display device as an SDK, and when the mixed reality capability module can call various AR capabilities in the operating system layer and / or hardware layer of the head-mounted display device through an API interface, different display content and interaction methods are provided for the mixed reality scene. The AR capabilities may include, but are not limited to: 2D / 3D display, gesture operation, and head tracking. The content displayed on the scanning interface has a first preset number of degrees of freedom and a first preset display distance. The first preset number may be 0. The first preset display distance may be 1m. The selection operation may include but is not limited to at least one of the following: click, slide, voice control, gesture control, head posture control. The content displayed on the scanning interface may include but is not limited to: scanning screen, scanning prompt information. The scanning prompt information may be prompt information that guides the user to scan the code according to the prompt. Here, the scanning prompt information may be displayed in the scanning interface in text form. The scanning screen may be a screen uploaded by the camera. In practice, in response to detecting the user's selection operation on the scan button, the execution subject may display the scanning interface on the head-mounted display device and identify the code in the scanning screen. For example, the code may be a QR code. Thus, after successfully identifying the code, the mixed reality scene corresponding to the scanned code can be determined.

[0035] Optionally, before step 101, the execution subject may also display the initial page of the target application on the head mounted display device in response to detecting that the target application is turned on. The initial page of the target application displays the three-dimensional model and the scan button of the target application. The three-dimensional model and the scan button of the target application have a second preset number of degrees of freedom. The three-dimensional model of the target application may correspond to a first preset display distance. The scan button may correspond to a second preset display distance. The scan button may be a button for the user to select to scan a code. The second preset number may be 3. The first preset display distance may be 1m. The second preset display distance may be 0.5m.

[0036] Step 102, in response to determining that the code scan is successful, loading the mixed reality scene corresponding to the scanned code.

[0037] In some embodiments, in response to determining that the code scan is successful, the execution subject may load the mixed reality scene corresponding to the scanned code. The mixed reality scene may include various scene elements pre-edited in the mixed reality scene. The scene elements in the various scene elements correspond to media information. The media information may include but is not limited to at least one of the following types of data: image, video, audio, text. In one or more embodiments, during the loading process, the head-mounted display device may display a loading interface. The content displayed on the loading interface may include but is not limited to: a three-dimensional model of the target application and a loading status icon. The target application may be an opened application. The content displayed on the loading interface has a second preset value of freedom. The second preset value may be 3. The display distance of the three-dimensional model of the target application may be 1m. The display distance of the loading status icon may be 1m. The display distance may be the distance from the display interface to the camera position of the mixed reality space. In one or more embodiments, loading means loading the mixed reality scene into the running memory of the head-mounted display device so as to render, display, edit, interact with, and perform other operations on the mixed reality scene.

[0038] In some optional implementations of some embodiments, in response to determining that the code scan is successful, the execution subject may load the mixed reality scene corresponding to the scanned code through the following steps:

[0039] The first step is to request the mixed reality scene corresponding to the scanned code from the associated server. The above-mentioned server can be a mobile device connected to the above-mentioned head-mounted display device. For example, the mobile device can be a mobile phone. In practice, the above-mentioned execution entity can send request information for the above-mentioned mixed reality scene to the above-mentioned server. The above-mentioned execution entity and the above-mentioned server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication (for example, communication network) of any form or medium.

[0040] The second step is to determine whether to enable spatial occlusion in response to determining that the mixed reality scene request is successful. The enabling of spatial occlusion can indicate that there is a virtual wall at the current location, and the user cannot see the media information behind the virtual wall. In practice, in response to determining that the mixed reality scene data sent by the server is received, the execution subject can determine that the mixed reality scene request is successful.

[0041] The third step is to determine whether the standardized spatial map resources of the mixed reality scene are downloaded in response to determining to turn on spatial occlusion. The standardized spatial map resources can represent location information. The location information can be information related to the location of the real space. For example, the location information can be geographic location information. The geographic location information can include longitude and latitude. The location information can also be location coordinates determined by Bluetooth technology. In practice, in response to determining to turn on spatial occlusion, the execution entity can determine whether the standardized spatial map resources of the mixed reality scene are loaded into the running memory of the head-mounted display device.

[0042] In step 4, in response to determining that the standardized spatial map resource of the mixed reality scene has been downloaded, it is determined that the mixed reality scene has been loaded successfully. Thus, after confirming that the standardized spatial map resource of the mixed reality scene has been downloaded successfully, it can be confirmed that the mixed reality scene has been loaded successfully.

[0043] In some optional implementations of some embodiments, in response to determining that the code scan is successful, the execution subject may also download the standardized spatial map resource of the mixed reality scene in response to determining that the standardized spatial map resource of the mixed reality scene has not been downloaded. In practice, the execution subject may download the standardized spatial map resource of the mixed reality scene from an associated server.

[0044] Then, in response to determining that the standardized space map resource of the mixed reality scene is downloaded successfully, it is determined that the mixed reality scene is loaded successfully.

[0045] Finally, in response to determining that the download of the standardized spatial map resource of the mixed reality scene has failed, a re-download prompt message is displayed on the head mounted display device. The re-download prompt message may be a prompt message that prompts the user that the download has failed and guides the user to re-download. Here, the re-download prompt message may be displayed in the loading interface in text form. Thus, the user can re-download the standardized spatial map resource after receiving the re-download prompt message.

[0046] In some optional implementations of some embodiments, in response to determining that the image code scan is successful, the execution entity may also display a data loading interface on the head-mounted display device. The content displayed on the data loading interface may include but is not limited to: a three-dimensional model of the target application and loading prompt information. The loading prompt information may be a prompt information that prompts the user that data is being loaded. The content displayed on the data loading interface has a first preset number of degrees of freedom. The content displayed on the data loading interface corresponds to a first preset display distance. The first preset number may be 0. The first preset display distance may be 1m.

[0047] Step 103, in response to determining that the mixed reality scene is loaded successfully, determining whether the real space scene matches the mixed reality scene, and displaying a matching interface on the head mounted display device.

[0048] In some embodiments, in response to determining that the mixed reality scene is loaded successfully, the execution entity may determine whether the real space scene matches the mixed reality scene, and display a matching interface on the head mounted display device. The matching interface displays a dynamic space model and matching guidance information. The dynamic space model may represent a room of the loaded mixed reality scene. The dynamic space model may be displayed with an animation effect. The dynamic space model has a first preset number of degrees of freedom and a first preset display distance. The first preset number may be 0. The first preset display distance may be 1m.

[0049] In some optional implementations of some embodiments, the execution subject may determine whether the real space scene matches the mixed reality scene by following the steps below:

[0050] The first step is to determine whether the position information of the mixed reality terminal matches the position configuration information of the mixed reality scene in response to determining to turn on spatial occlusion. The position configuration information may be position information configured according to the real space corresponding to the mixed reality scene. The position information may be GPS positioning coordinates or coordinates determined by Bluetooth technology. In practice, the execution entity may determine whether the position information is within a preset buffer corresponding to the position configuration information to determine whether the position information of the mixed reality terminal matches the position configuration information of the mixed reality scene.

[0051] The second step is to determine that the real space scene matches the mixed reality scene in response to determining that the position information of the mixed reality terminal matches the position configuration information of the mixed reality scene. In practice, the execution subject can determine that the position information matches the position configuration information in response to determining that the position information is within a preset buffer corresponding to the position configuration information.

[0052] In some optional implementations of some embodiments, in response to determining that spatial occlusion is not enabled, the execution subject may further perform the following steps:

[0053] The first step is to determine whether the mixed reality scene is configured with an anchor graph. The anchor graph may be a preset image of the real space corresponding to the mixed reality scene. In practice, in response to determining that the mixed reality scene has uploaded a preset image of the corresponding real space, the execution subject may determine that the mixed reality scene is configured with an anchor graph.

[0054] In the second step, in response to determining that the above-mentioned mixed reality scene is configured with an anchoring graph, the anchoring graph corresponding to the above-mentioned mixed reality scene is reduced from the center position of the above-mentioned matching interface and displayed at a preset position of the above-mentioned matching interface. Among them, the above-mentioned anchoring graph corresponds to a second preset display distance. The above-mentioned second preset display distance can be 0.5m. The above-mentioned preset position can be a position at a preset distance below the dynamic space model displayed on the above-mentioned matching interface. Here, the specific setting of the above-mentioned preset distance is not limited.

[0055] The third step is to determine the various feature points of the real space scene, and determine the various feature points of the anchor graph corresponding to the above mixed reality scene. The various feature points of the above real space scene can characterize the characteristics of the real space scene. The various feature points of the above anchor graph can characterize the characteristics of the anchor graph. In practice, the above execution entity can use a feature point detection algorithm to determine the various feature points of the real space scene, and determine the various feature points of the anchor graph corresponding to the above mixed reality scene. The above feature point detection algorithm can be but is not limited to: Harris algorithm, SIFT algorithm.

[0056] In step 4, in response to determining that each feature point of the real space scene matches each feature point of the anchor graph, determining that the real space scene matches the mixed reality scene. In practice, the execution subject may determine that each feature point of each feature point of the anchor graph matches a feature point of each feature point of the real space scene, and determine that the real space scene matches the mixed reality scene.

[0057] It should be noted that when there are feature points in the anchor graph that do not match the feature points in the real space scene, in some embodiments, the execution subject may display an anchor failure prompt message and re-anchor guidance information on the matching interface. The anchor failure prompt message may be a prompt message that prompts the user that the anchor graph does not match the real space scene. The re-anchor guidance information may be information that guides the user to re-anchor.

[0058] In some optional implementations of some embodiments, in response to determining that the mixed reality scene is loaded successfully, the execution subject may further perform the following steps:

[0059] The first step is to determine whether any of the above scene elements are cached in response to detecting that the user's interaction information meets the preset trigger conditions of any scene element in the selected mixed reality scene. The preset trigger condition may be set for an event of any of the above scene elements. The preset trigger condition may be a condition that needs to be met to trigger an event. The preset trigger condition may be configured when configuring the mixed reality space. There is no limitation on the specific setting of the preset trigger condition. In practice, in response to detecting that the user's interaction information meets the preset trigger condition of any scene element in the selected mixed reality scene, the above execution subject may detect that any of the above scene elements are loaded into the running memory of the head-mounted display device, and may determine that any of the above scene elements are cached.

[0060] In the second step, in response to determining that any of the above scene elements is cached, the media information corresponding to any of the above scene elements is triggered in the above mixed reality scene. In practice, the above execution subject may trigger the media information corresponding to any of the above scene elements in the above mixed reality scene according to the event configuration information of any of the above scene elements. The event configuration information may include but is not limited to: event type, preset trigger conditions. The above event type may be the type to which the event of the scene element belongs. For example, the event type may be but is not limited to one of the following: media information loading, media information hiding, media information playing. In one or more embodiments, the triggering can be completed through the triggering area, that is, when the user is in the triggering area, the triggering operation is performed. The triggering area may refer to a fixed interactive area in the scene element for collision detection.

[0061] In step 3, in response to determining that any of the above scene elements is not cached, the following steps are performed:

[0062] The first sub-step is to download any of the above scene elements. In practice, the above execution subject may download any of the above scene elements from an associated server.

[0063] The second sub-step is, in response to determining that the above-mentioned arbitrary scene elements are downloaded successfully, triggering the media resource information corresponding to the above-mentioned arbitrary scene elements in the above-mentioned mixed reality scene.

[0064] In a third sub-step, in response to determining that the download of any of the scene elements fails, the head mounted display device displays re-download guidance information. The re-download guidance information may be guidance information for guiding the user to re-download. Here, the re-download guidance information may be displayed in a text form in a current display interface of the head mounted display device.

[0065] Step 104: In response to determining that the real space scene matches the mixed reality scene, displaying the mixed reality scene on a head mounted display device.

[0066] In some embodiments, in response to determining that the real space scene matches the mixed reality scene, the execution subject may display the mixed reality scene on the head mounted display device, wherein the degree of freedom and display distance of the display content of the head mounted display device may be generated by the mixed reality capability module of the mixed reality terminal.

[0067] In some optional implementations of some embodiments, in response to determining that the real space scene matches the mixed reality scene, the execution subject may also display a successful matching interface on the head-mounted display device, and play a successful matching prompt message. The successful matching interface displays a successful matching prompt message, and the successful matching prompt message has a first preset value of degrees of freedom and a first preset display distance. The successful matching prompt message may be a prompt message that prompts the user that the match is successful. Here, the successful matching prompt message may be displayed in a text pop-up window in the successful matching interface.

[0068] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the mixed reality space interaction method of some embodiments of the present disclosure, the interaction mode between the user and the mixed reality space is enriched, and an immersive interaction experience can be provided for the user. Specifically, the reason for not being able to provide the user with an immersive interaction experience is that in the existing MR navigation process, the feature points of the physical space are identified by the camera of the mixed reality device (for example, MR glasses), and after matching the feature information, a section of audio and video data is played through the mixed reality device. However, this type of navigation method provides limited interaction methods, and there is no clear connection between the digital space and the physical space, which cannot give the user an immersive experience. At present, there is still a lack of a dedicated method and tool to realize the immersive interaction between the browsing user and the MR space. Based on this, the mixed reality space interaction method of some embodiments of the present disclosure, first, in response to detecting the user's selection operation of the scan button, the scanning interface is displayed on the head-mounted display device, and the code scanning is performed, wherein the content displayed on the scanning interface has a first preset number of degrees of freedom and a first preset display distance. Thus, after the user selects the scan button, the code displayed in the scanning interface can be identified. Then, in response to determining that the code scan is successful, the mixed reality scene corresponding to the scanned code is loaded. Thus, after the code is scanned successfully, the mixed reality scene corresponding to the code can be obtained. Afterwards, in response to determining that the mixed reality scene is loaded successfully, it is determined whether the real space scene matches the mixed reality scene, and the matching interface is displayed on the head-mounted display device, wherein the matching interface displays a dynamic space model and matching guidance information, and the dynamic space model has a first preset number of degrees of freedom and a first preset display distance. Thus, it can be determined whether the real space scene matches the mixed reality scene. Finally, in response to determining that the real space scene matches the mixed reality scene, the mixed reality scene is displayed on the head-mounted display device, wherein the degrees of freedom and display distance of the display content of the head-mounted display device are generated by the mixed reality capability module of the mixed reality end. Thus, the user can browse the mixed reality scene. Also because of the implemented mixed reality space interaction method, different degrees of freedom and display distances can be used in different interaction interfaces and different display contents. Thereby, the interaction mode between the user and the mixed reality space is enriched, and an immersive interactive experience can be provided for the user.

[0069] Further references Figure 2 , which shows a process 200 of another embodiment of a mixed reality space interaction method. The process 200 of the mixed reality space interaction method includes the following steps:

[0070] Step 201, in response to detecting a user's selection operation on a scan button, displaying a scan interface on a head mounted display device and performing a code scan.

[0071] Step 202: In response to determining that the code scan is successful, loading the mixed reality scene corresponding to the scanned code.

[0072] Step 203, in response to determining that the mixed reality scene is loaded successfully, determining whether the real space scene matches the mixed reality scene, and displaying a matching interface on the head mounted display device.

[0073] Step 204: In response to determining that the real space scene matches the mixed reality scene, displaying the mixed reality scene on a head mounted display device.

[0074] In some embodiments, the specific implementation of steps 201-204 and the technical effects brought about can be referred to Figure 1 The corresponding steps 101-104 in the embodiments are not described in detail here.

[0075] Step 205, in response to determining that the real space scene does not match the mixed reality scene, determining the reason for the matching failure, and displaying a matching failure interface.

[0076] In some embodiments, in response to determining that the real space scene does not match the mixed reality scene, the execution entity may determine the reason for the match failure and display a match failure interface. The content displayed on the match failure interface may include but is not limited to: positioning pictures, match failure prompt information and match retry buttons. The content displayed on the match failure interface has a first preset number of degrees of freedom. The positioning picture corresponds to a first preset display distance. The match retry button corresponds to a second preset display distance. The first preset number may be 0. The first preset display distance may be 1m. The second preset display distance may be 0.5m. The positioning picture may be an icon representing the positioning failure. The match failure prompt information may be prompt information that prompts the user that the match failed. In one or more embodiments, different match failure prompt information may be displayed for different match failure reasons. The match failure reason may be but is not limited to: insufficient feature points, concurrency reaching the upper limit, and network abnormality. The match success prompt information may be displayed in the match failure interface in text form. The match retry button may be a button that can be selected by the user for rematching. In one or more embodiments, in response to determining that the concurrency reaches the upper limit and the matching result cannot be obtained, the execution subject may determine that the reason for the matching failure is that the concurrency reaches the upper limit. In response to determining that the network request fails and the matching result cannot be obtained, the execution subject may determine that the reason for the matching failure is a network anomaly. In response to determining that the feature points identified from the real space scene do not match the feature points identified from the anchor graph, the execution subject may determine that the reason for the matching failure is insufficient feature points.

[0077] from Figure 2It can be seen that Figure 1 Compared with the description of some corresponding embodiments, Figure 2 The process 200 of the mixed reality space interaction method in some corresponding embodiments reflects the operation after the real space scene does not match the above mixed reality scene. Therefore, the scheme described in these embodiments can prompt the user the reason for the matching failure and the re-matching button when the real space scene does not match the above mixed reality scene, so that the user can choose the time to re-match according to the reason for the matching failure, thereby enriching the interaction mode between the user and the mixed reality space and providing the user with an immersive interactive experience.

[0078] Continue to refer Figure 3 , which shows a mixed reality terminal 300 suitable for implementing some embodiments of the present disclosure. The mixed reality terminal includes a head-mounted display device 301 and a processor 302. The head-mounted display device 301 is used for allowing users to browse mixed reality scenes and media information. The processor 302 may be a mobile device without a display function. The mobile device may be a mobile phone or a tablet computer. The processor 302 is configured to execute the following Figure 1 or Figure 2 The processor 302 is connected to the head mounted display device 301. Here, the connection between the processor 302 and the head mounted display device 301 may be a wired connection or a wireless connection. In some embodiments, the head mounted display device 301 may be used as an external display device of the processor 302 and does not have data processing capabilities.

[0079] Reference below Figure 4 , which shows a hardware structure diagram of a head-mounted display device 400 for browsing a mixed reality space. Figure 4 The hardware structure diagram shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0080] like Figure 4As shown, the head-mounted display device 400 includes a frame 401, a display module 402, a posture sensor (not shown in the figure), a grayscale camera 403, and a processor (CPU) (not shown in the figure). Among them, the display module 402, the posture sensor (not shown in the figure), the grayscale camera 403 and the processor (CPU) (not shown in the figure) are connected to each other through a bus. The above-mentioned frame 401 can be a glasses structure or a head ring structure, and can have a space for accommodating a circuit structure. The above-mentioned display module 402 can be one or more. The above-mentioned display module 402 can be embedded in the frame 401. The above-mentioned display module 402 can include a semi-transparent and semi-reflective display unit, so that the user can see the real environment and the display content at the same time. Among them, the above-mentioned real environment and the display content constitute a mixed reality space. The above-mentioned posture sensor (not shown in the figure) can be embedded in the above-mentioned circuit structure and is configured to obtain the posture data of the user's head. The above-mentioned grayscale camera 403 can be integrated in the above-mentioned circuit structure for collecting images of environmental data. The above-mentioned processor (CPU) (not shown in the figure) can be integrated in the above-mentioned circuit structure or be separately set on the host of the head-mounted display device. The processor (CPU) is configured to configure different degrees of freedom, display distances, and interaction methods according to the display page requirements of the mixed reality space. In some embodiments, the interaction methods may include but are not limited to: gesture interaction, voice interaction, and key interaction.

[0081] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0082] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0083] The computer-readable medium may be included in the head-mounted display device; or may exist independently without being assembled into the head-mounted display device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the mixed reality end, the mixed reality end: in response to detecting a user's selection operation on a scan button, displays a scan interface on the head-mounted display device, and performs a code scan, wherein the content displayed on the scan interface has a first preset number of degrees of freedom and a first preset display distance; in response to determining that the code scan is successful, loads a mixed reality scene corresponding to the scanned code; in response to determining that the mixed reality scene is loaded successfully, determines whether the real space scene matches the mixed reality scene, and displays a matching interface on the head-mounted display device, wherein the matching interface displays a dynamic space model and matching guidance information, and the dynamic space model has a first preset number of degrees of freedom and a first preset display distance; in response to determining that the real space scene matches the mixed reality scene, displays the mixed reality scene on the head-mounted display device, wherein the degrees of freedom and display distance of the displayed content of the head-mounted display device are generated by the mixed reality capability module of the mixed reality end.

[0084] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0085] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0087] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A mixed reality space interaction method, applied to a mixed reality terminal, the mixed reality terminal includes a head-mounted display device, the head-mounted display device includes a processor and two display modules, corresponding to the left and right eyes of a user respectively, the processor is integrated with a mixed reality capability module, the method includes: In response to detecting a user's selection operation of a scan button, displaying a scan interface on the head mounted display device, and performing a barcode scan, wherein the content displayed on the scan interface has a first preset number of degrees of freedom and a first preset display distance; In response to determining that the code scan is successful, loading a mixed reality scene corresponding to the scanned code; In response to determining that the mixed reality scene is loaded successfully, determining whether the real space scene matches the mixed reality scene, and displaying a matching interface on the head mounted display device, wherein the matching interface displays a dynamic space model and matching guide information, the dynamic space model having a first preset number of degrees of freedom and a first preset display distance; In response to determining that the real space scene matches the mixed reality scene, the mixed reality scene is displayed on the head-mounted display device, wherein the degrees of freedom and display distance of the display content of the head-mounted display device are generated by the mixed reality capability module of the mixed reality end.

2. The method according to claim 1, wherein: Before responding to detecting a user's selection operation on a scan button, the method further includes: In response to detecting that a target application is turned on, an initial page of the target application is displayed on the head-mounted display device, wherein the initial page of the target application displays a three-dimensional model and a scan button of the target application, the three-dimensional model and the scan button of the target application have a second preset number of degrees of freedom, the three-dimensional model of the target application corresponds to a first preset display distance, and the scan button corresponds to a second preset display distance.

3. The method according to claim 2, wherein: After the response determines that the code scan is successful, the method further includes: A data loading interface is displayed on the head mounted display device, wherein content displayed on the data loading interface has a first preset number of degrees of freedom, and the content displayed on the data loading interface corresponds to a first preset display distance.

4. The method according to claim 1, wherein: The loading of the mixed reality scene corresponding to the scanned code includes: Requesting the associated server for the mixed reality scene corresponding to the scanned code; In response to determining that the mixed reality scene is requested successfully, determining whether to enable spatial occlusion; In response to determining that spatial occlusion is enabled, determining whether a standardized spatial map resource of the mixed reality scene is downloaded; In response to determining that the standardized spatial map resource of the mixed reality scene has been downloaded, it is determined that the mixed reality scene has been loaded successfully.

5. The method according to claim 4, wherein: The loading of the mixed reality scene corresponding to the scanned code also includes: In response to determining that the standardized spatial map resource of the mixed reality scene has not been downloaded, downloading the standardized spatial map resource of the mixed reality scene; In response to determining that the standardized spatial map resource of the mixed reality scene is downloaded successfully, determining that the mixed reality scene is loaded successfully; In response to determining that the download of the standardized space map resource of the mixed reality scene fails, re-download prompt information is displayed on the head-mounted display device.

6. The method according to claim 5, wherein: The determining whether the real space scene matches the mixed reality scene includes: In response to determining to enable spatial occlusion, determining whether the position information of the mixed reality terminal matches the position configuration information of the mixed reality scene; In response to determining that the position information of the mixed reality terminal matches the position configuration information of the mixed reality scene, it is determined that the real space scene matches the mixed reality scene.

7. The method according to claim 6, wherein: The determining whether the real space scene matches the mixed reality scene also includes: In response to determining that spatial occlusion is not enabled, the following steps are performed: Determining whether the mixed reality scene is configured with an anchor graph; In response to determining that the mixed reality scene is configured with an anchoring graph, shrinking the anchoring graph corresponding to the mixed reality scene from the center position of the matching interface and displaying it at a preset position of the matching interface, wherein the anchoring graph corresponds to a second preset display distance; Determining each feature point of the real space scene, and determining each feature point of the anchor graph corresponding to the mixed reality scene; In response to determining that each feature point of the real space scene matches each feature point of the anchor graph, it is determined that the real space scene matches the mixed reality scene.

8. The method according to claim 1, wherein: After the response determines that the mixed reality scene is loaded successfully, the method further includes: In response to detecting that the user's interaction information satisfies a preset trigger condition of any scene element in the selected mixed reality scene, determining whether the any scene element is cached; In response to determining that the arbitrary scene element is cached, triggering media asset information corresponding to the arbitrary scene element in the mixed reality scene; In response to determining that the arbitrary scene element has no cache, the following steps are performed: Downloading any scene element; In response to determining that the arbitrary scene element is downloaded successfully, triggering media asset information corresponding to the arbitrary scene element in the mixed reality scene; In response to determining that the download of any scene element fails, re-download guidance information is displayed on the head mounted display device.

9. The method according to claim 1, wherein: The method further comprises: In response to determining that the real space scene does not match the mixed reality scene, the reason for the matching failure is determined, and a matching failure interface is displayed, wherein the matching failure interface displays a positioning picture, matching failure prompt information and a matching retry button, and the content displayed on the matching failure interface has a first preset number of degrees of freedom, the positioning picture corresponds to a first preset display distance, and the matching retry button corresponds to a second preset display distance.

10. The method according to claim 1, wherein: After the response to determining that the real space scene matches the mixed reality scene, the method further includes: A matching success interface is displayed on the head mounted display device, and matching success prompt information is played, wherein the matching success interface displays matching success prompt information, and the matching success prompt information has a degree of freedom of a first preset value and a first preset display distance.

11. A mixed reality terminal, comprising: Head-mounted display device, used for users to browse mixed reality scenes and media information; A processor connected to the head mounted display device; The processor is configured to perform the method as claimed in any one of claims 1-10.

12. A head-mounted display device for browsing a mixed reality space, comprising: A frame, wherein the frame has a glasses structure or a headband structure and has a space for accommodating a circuit structure; One or more display modules, wherein the display modules are embedded in the glasses frame, and the display modules include a semi-transparent and semi-reflective display unit, so that the user can see the real environment and the display content at the same time, wherein the real environment and the display content constitute a mixed reality space; A posture sensor, the posture sensor is embedded in the circuit structure and is configured to obtain posture data of the user's head; A grayscale camera, which is integrated into the circuit structure and is used to collect images of environmental data; A processor, wherein the processor is integrated into the circuit structure or is separately arranged on a host of a head-mounted display device, and the processor is configured to configure different degrees of freedom, display distances, and interaction modes according to display page requirements of the mixed reality space, wherein the interaction modes include gesture interaction, voice interaction, and button interaction.

Images