Man-machine interaction system, method and device for mixed reality

Through the codeless human-computer interaction system, users can design and display virtual content in 3D space, solving the lack of a professional development team in the existing technology, and achieving efficient and low-cost virtual resources in commercial and industrial scenarios show.

CN119006750APending Publication Date: 2024-11-22SUZHOU FIREFLY MIXED REALITY TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410934390.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In commercial and industrial scenarios, developing high-quality mixed reality (MR), augmented reality (AR), and virtual reality (VR) head-mounted display software requires a professional software development team and programming knowledge, resulting in the end users facing a long development cycle. , high budget and difficult project management.

Method used

Provide a codeless human-computer interaction system, through display process design, content layout and display usage modules, allowing users to design and display virtual content in 3D space, including text, pictures, videos, 3D models and animations, and use head wear Display devices and handheld mobile devices place and display virtual content, and trigger conditions include time, location, user input, etc.

Benefits of technology

It realizes that virtual content is edited, generated and used without relevant software development experience to meet customer needs, simplifies the display process of virtual resources, and reduces development complexity and cost.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006750A_ABST
    Figure CN119006750A_ABST
Patent Text Reader

Abstract

The invention relates to a man-machine interaction system, method and device for mixed reality, and the method comprises the steps: display process design: designing at least one display step which comprises at least one virtual content needing to be displayed; designing a trigger condition of each display step; content arrangement: setting attributes of the virtual content, the attributes at least comprising placement poses of the virtual content in a 3D space; and displaying and using: displaying the corresponding virtual content based on the set attributes in each displaying step. According to the method, a code-free means is adopted, various virtual contents are edited, generated and used under the condition that customers do not need to have related software development experience, and finally the purpose of displaying corresponding virtual resources according to the requirements of the customers in the real world is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of mixed reality devices, and in particular to a human-computer interaction system, method and device for mixed reality. Background Art

[0002] At present, customizing and developing software for mixed reality (MR), augmented reality (AR), and virtual reality (VR) (hereinafter collectively referred to as XR) head-mounted display devices often requires a large amount of professional human resources. The development team needs to master professional editing tools, such as the use of Unreal Engine, Unity and other software, be familiar with programming languages ​​such as C++ and C#, and have rich experience in software design, development, debugging and project management in order to develop high-quality XR software products. However, in many commercial and industrial XR usage scenarios, end users often do not have software development teams that meet the above requirements, and temporarily hiring outsourcing teams makes end users face the challenges of long development cycles, high budgets, and difficult project management.

[0003] At present, in commercial and industrial scenarios, common XR technology usage requirements include: displaying corresponding virtual resources in the real world according to customer needs. How to use code-free means to help users edit, generate and use virtual content without relevant software development experience is a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0004] The present invention provides a human-computer interaction system, method and device for mixed reality to solve the above technical problems.

[0005] In order to solve the above technical problems, the present invention provides a human-computer interaction method for mixed reality, comprising:

[0006] The display process design includes: designing at least one display step, wherein the display step includes at least one virtual content to be displayed; and designing triggering conditions for each display step;

[0007] Content arrangement, including: setting properties of the virtual content, the properties at least including the placement position of the virtual content in the 3D space; and

[0008] The display and use comprises: in each of the display steps, displaying corresponding virtual content based on the set attributes.

[0009] In some embodiments, the virtual content includes at least one or more of the following combinations: text, picture, video, audio, 3D model, and animation.

[0010] In some embodiments, multiple display steps are displayed sequentially in a fixed order.

[0011] In some embodiments, multiple display steps are displayed according to trigger conditions.

[0012] In some embodiments, the trigger condition includes a time condition, a location condition, a direction condition, a user input event, a pre-stored program script, or a custom program script.

[0013] In some embodiments, the user input event includes pressing a physical key or inputting an indication signal.

[0014] In some embodiments, the trigger condition is a combination of multiple trigger conditions after logical operation.

[0015] In some embodiments, the attributes of the virtual content further include: size, color, animation behavior, play speed or volume of the virtual content.

[0016] In some embodiments, the setting method of placing the virtual content in the 3D space includes:

[0017] Obtain reference positioning based on 3D space;

[0018] The virtual content is moved to a position to be placed by an anchor bound to the virtual content, a relationship between the anchor and the reference position is calculated and recorded, and the placed position is defined as the anchor position;

[0019] The placement position of the virtual content is determined based on the anchor position.

[0020] In some embodiments, the reference position is obtained by identifying and positioning a reference object in 3D space.

[0021] In some embodiments, the reference includes at least an environment, an object, or a logo.

[0022] In some embodiments, the anchor is a handheld mobile device.

[0023] In some embodiments, the anchor has a positioning pattern, and the anchor position is obtained by identifying and locating the positioning pattern.

[0024] In some embodiments, the positioning pattern includes a picture, a QR code, a barcode, or a specific graphic.

[0025] In some embodiments, an image acquisition device mounted on a head mounted display device is used to identify and locate the reference object and the positioning pattern.

[0026] In some embodiments, the anchor position is determined as a placement posture of the virtual content.

[0027] In some embodiments, the anchor position is subjected to mathematical calculation to determine the placement position of the virtual content.

[0028] A second aspect of the present invention provides a human-computer interaction system for mixed reality, comprising:

[0029] A display process design module, used to design at least one display step, wherein the display step includes at least one virtual content to be displayed; and to design trigger conditions for each display step;

[0030] A content placement module, configured to set properties of the virtual content, wherein the properties at least include a placement position of the virtual content in a 3D space; and

[0031] The display and use module is used to display the corresponding virtual content based on the set attributes in each of the display steps.

[0032] In some embodiments, the virtual content includes at least one or more of the following combinations: text, picture, video, audio, 3D model, and animation.

[0033] In some embodiments, multiple display steps are displayed sequentially in a fixed order.

[0034] In some embodiments, multiple display steps are displayed according to trigger conditions.

[0035] In some embodiments, the trigger condition includes a time condition, a location condition, a direction condition, a user input event, a pre-stored program script, or a custom program script.

[0036] In some embodiments, the user input event includes pressing a physical key or inputting an indication signal.

[0037] In some embodiments, the trigger condition is a combination of multiple trigger conditions after logical operation.

[0038] In some embodiments, the attributes of the virtual content further include: size, color, animation behavior, play speed or volume of the virtual content.

[0039] In some embodiments, the setting method of placing the virtual content in the 3D space includes:

[0040] Obtain reference positioning based on 3D space;

[0041] The virtual content is moved to a position to be placed by an anchor bound to the virtual content, a relationship between the anchor and the reference position is calculated and recorded, and the placed position is defined as the anchor position;

[0042] The placement position of the virtual content is determined based on the anchor position.

[0043] In some embodiments, the reference position is obtained by identifying and positioning a reference object in 3D space.

[0044] In some embodiments, the reference includes at least an environment, an object, or a logo.

[0045] In some embodiments, the anchor is a handheld mobile device.

[0046] In some embodiments, the anchor has a positioning pattern, and the anchor position is obtained by identifying and locating the positioning pattern.

[0047] In some embodiments, the positioning pattern includes a picture, a QR code, a barcode, or a specific graphic.

[0048] In some embodiments, an image acquisition device mounted on a head mounted display device is used to identify and locate the reference object and the positioning pattern.

[0049] In some embodiments, the anchor position is determined as a placement posture of the virtual content.

[0050] In some embodiments, the anchor position is subjected to mathematical calculation to determine the placement position of the virtual content.

[0051] A third aspect of the present invention further provides a human-computer interaction device for mixed reality, which is applied to the method described above, wherein the device comprises at least one processor, at least one handheld mobile device and at least one head-mounted display device.

[0052] The processor is configured to execute the display process design step;

[0053] The handheld mobile device is configured to cooperate with the head mounted display device to perform a content arrangement step;

[0054] The head mounted display device is configured to perform a demonstration use step.

[0055] In some embodiments, the processor is integrated into the head mounted display device.

[0056] In some embodiments, the processor is integrated into the handheld mobile device.

[0057] Compared with the prior art, the human-computer interaction system, method and device for mixed reality provided by the present invention adopt a code-free approach to edit, generate and use various virtual contents without the need for customers to have relevant software development experience, and ultimately achieve the purpose of displaying corresponding virtual resources in the real world according to customer needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a block diagram of a human-computer interaction system for mixed reality in a specific embodiment of the present invention (content arrangement stage);

[0059] Figure 2 A block diagram of a human-computer interaction system for mixed reality in a specific embodiment of the present invention (showing the use phase);

[0060] Figure 3 A flowchart showing the process design phase and the content arrangement phase in a specific embodiment of the present invention;

[0061] Figure 4 A flowchart showing the process design phase and the content arrangement phase in another specific embodiment of the present invention;

[0062] Figure 5 A flowchart showing the use phase in a specific embodiment of the present invention;

[0063] Figures 6 to 9 They are respectively schematic diagrams of the corresponding relationships between the triggering conditions and the display steps in a specific implementation manner of the present invention.

[0064] In the figure: 10-reference object, 11-reference coordinate system, 20-handheld mobile device, 21-positioning pattern, 22-virtual content, 30-head mounted display device. DETAILED DESCRIPTION

[0065] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0066] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0067] Although the present application makes various references to certain modules in the system according to embodiments of the present application, any number of different modules may be used and run on the client and / or server of the mixed reality device. The modules are illustrative only, and different aspects of the system and method may use different modules.

[0068] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0069] The embodiments of the present application can be applied to different usage scenarios, for example: 1) building a mixed virtual and real immersive exhibition experience in exhibitions and exhibition halls, and matching virtual special effects near the physical goods on display; 2) using mixed reality to display step-by-step operation prompts in employee skills training; 3) quickly providing customers with a virtual equipment layout preview in the sales process of equipment suppliers, etc.

[0070] Please refer to Figures 1 to 9 , the human-computer interaction system for mixed reality provided by the present invention includes: a display process design module, used to design at least one display step, wherein the display step includes at least one virtual content 22 to be displayed; and design trigger conditions for each display step;

[0071] A content placement module, used to set the properties of the virtual content 22, wherein the properties at least include the placement position of the virtual content 22 in the 3D space; and

[0072] The display and use module is used to display the corresponding virtual content 22 based on the set attributes in each of the display steps.

[0073] In some embodiments, the display process design module, the content arrangement module and the display use module can be interconnected through at least one server-side software for data communication and synchronization at each stage.

[0074] In some embodiments, the virtual content 22 includes at least one or more of the following combinations: text, picture, video, audio, 3D model, and animation.

[0075] In some embodiments, multiple display steps may be displayed sequentially in a fixed order.

[0076] In some embodiments, multiple display steps may be displayed according to trigger conditions.

[0077] In some embodiments, the trigger condition may include a time condition, a location condition, a direction condition, a user input event, a pre-stored program script, or a custom program script.

[0078] In some embodiments, the user input event may include pressing a physical key or inputting an indication signal.

[0079] In some embodiments, the trigger condition may be a combination of multiple trigger conditions that have undergone logical operations.

[0080] In some embodiments, the attributes of the virtual content 22 may further include: size, color, animation behavior, play speed or volume of the virtual content 22 .

[0081] In some embodiments, the placement of the virtual content 22 in the 3D space may be configured as follows:

[0082] Obtain reference positioning based on 3D space;

[0083] The virtual content 22 is moved to a position to be placed by an anchor (such as a handheld mobile device 20) bound to the virtual content 22, and the relationship between the anchor and the reference position is calculated and recorded, and the placed position is defined as the anchor position;

[0084] The placement position of the virtual content 22 is determined based on the anchor position.

[0085] In some embodiments, the reference positioning may be obtained by identifying and positioning a reference object 10 in a 3D space.

[0086] In some embodiments, the reference may include at least an environment, an object, or a logo.

[0087] In some embodiments, the anchor has a positioning pattern 21 , and the anchor position is acquired by identifying and locating the positioning pattern 21 .

[0088] In some embodiments, the positioning pattern 21 may include a picture, a QR code, a barcode or a specific graphic.

[0089] In some embodiments, an image acquisition device mounted on the head mounted display device 30 is used to identify and locate the reference object 10 and the positioning pattern 21 .

[0090] In some embodiments, the anchor position is determined as the placement posture of the virtual content 22 .

[0091] In some embodiments, the placement position of the virtual content 22 is determined after mathematical operation is performed on the anchor position.

[0092] It should be understood that the above-mentioned system and its modules can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art will understand that the above-mentioned methods and systems can be implemented using computer executable instructions and / or included in a processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of the present application can not only be implemented by hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software such as executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware and software.

[0093] It should be noted that the above description of the system and its modules is only for the convenience of description, and the present application cannot be limited to the scope of the embodiments cited. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the various modules, or form a subsystem to connect with other modules without deviating from this principle. For example, in some embodiments, the display process design module, the content layout module, and the display use module can be different units in a system, or one unit can implement the functions of two or more of the above modules. For another example, each module can share a storage device, or each unit can have its own storage device. Such variations are all within the scope of protection of the present application.

[0094] The human-computer interaction method for mixed reality provided by the present invention is as follows: Figures 1 to 9 As shown, including:

[0095] The display process design includes: designing at least one display step, wherein the display step includes at least one virtual content 22 to be displayed; and designing the triggering conditions of each display step. When designing the display process, the user can create, modify, delete one or more display steps, and specify the virtual content 22 to be displayed for each display step.

[0096] Content arrangement includes: setting the properties of the virtual content 22, wherein the properties at least include the placement position (position and / or posture) of the virtual content 22 in the 3D space. In the content arrangement stage, the content arrangement module and the head mounted display device 30 in the system can help the user arrange the placement layout and other properties of the virtual content 22 in the 3D space.

[0097] And display use, including: in each of the display steps, displaying the corresponding virtual content 22 based on the set attributes. In this stage, the head mounted display device 30 displays the virtual content 22 according to the display steps and trigger conditions in the display process design stage, and the posture and other attributes of the virtual content 22 generated in the content layout stage.

[0098] Figure 3 A realization process of display process design and content layout is described. In S101, the user first designs the display process, defines each display step in the display process, the virtual content 22 to be displayed under the display step, and the triggering conditions of the start or step-to-step flow of each display step. Then the user enters the content layout stage and arranges the virtual content 22. In S102, the head-mounted display device 30 can be used to establish a reference coordinate system 11 according to the user's design, and then in S103, the head-mounted display device 30 is used to identify and locate the positioning pattern 21 displayed by the handheld mobile device 20 in the environment, and the virtual content 22 currently being arranged by the user is superimposed and displayed near the handheld mobile device 20 according to the positioning result, so that it moves with the handheld device 20. In S104, the user adjusts the posture of the virtual content 22 by adjusting the posture of the handheld mobile device 20, and confirms the layout and placement of the virtual content 22 according to the preview effect of the head-mounted display device 30. In S105, when the user needs to arrange more virtual content 22, the user can select the virtual content 22 to be arranged and repeat S103 and S104. If the user chooses to complete the content arrangement, the data of the display process design and the content arrangement will be stored in S106.

[0099] In some embodiments, S106 may be executed synchronously with other processes. For example, when the user makes any changes to the display process and content layout, or when the user decides to store the current design, relevant data may be stored.

[0100] In some embodiments, display process design and content layout can be carried out simultaneously. For example, when arranging content, the user can enter the display process design stage to add or delete display steps, change the trigger conditions of display steps, increase or decrease the corresponding virtual content 22, or move the virtual content 22 to different display steps, etc.

[0101] Figure 4Describes another implementation approach for the display process design and content layout phases. Figure 3 Compared with the implementation described above, in this implementation, the user can define different reference coordinate systems 11 for different virtual contents 22. When the user starts to arrange a virtual content 22, S111 will decide whether to establish a different reference coordinate system 11 according to the design in S101 or the user's current selection. For example, when the reference coordinate system corresponding to the virtual content 22 is different from the current reference coordinate system, the reference coordinate system is not currently established, or the user specifies a new reference coordinate system, S111 will choose to enter S102 to establish a reference coordinate system. If there is no need to establish a different reference coordinate system, the established reference coordinate system will be continued to be used, and S103 will be directly entered to start the arrangement of the virtual content 22.

[0102] Figure 5 An implementation method for the display use phase is described. In S201, the trigger condition designed by the user is met, and the head mounted display device 30 is ready to display the corresponding virtual content 22. S202 determines the reference coordinate system 11 corresponding to the virtual content 22. If the reference coordinate system 11 has not been established, S203 identifies and locates the reference object 10 in the environment and establishes the reference coordinate system 11. S204 displays the layout data of the virtual content 22 under the reference coordinate system 11.

[0103] In some embodiments, the virtual content 22 includes at least one or more of the following combinations: text, pictures, videos, audios, 3D models, and animations. For example, the virtual content 22 may be an arrow indicating the position of a component; a text introduction or an audio or video introduction; or a demonstration animation of a method of use.

[0104] In some embodiments, the multiple display steps can be displayed in a fixed order. For example, when the start key is pressed, display step 1 starts to be displayed, and after the playback is completed (or after the playback is completed and delayed for a certain period of time), display step 2 starts to be displayed, and so on, until all display steps are displayed.

[0105] In some embodiments, multiple display steps may be displayed according to trigger conditions. That is, the display order of multiple display steps may be linear or non-linear, such as multiple display steps are displayed simultaneously, or the start or end of a display step is determined by a user-defined condition.

[0106] Figure 6 This describes a linear step design in the display process design phase. When trigger condition 1 is met, the display of display step 1 is triggered, and then when trigger condition 2 is met, the display of display step 2 is triggered.

[0107] Figure 7 This paper describes the design of a nonlinear step in the display process design stage, in which when trigger condition 1 is met, the display of display step 1 is triggered; when trigger condition 2 is met, the display of display step 2 is triggered; when trigger condition 3 is met, the display of display step 3 is triggered. The display processes of the three display steps are independent of each other and do not affect each other.

[0108] Figure 8 This paper describes the design of a nonlinear step in the display process design phase. When trigger condition 1 is met, display step 1 is triggered, and then when trigger condition 2 is met, display step 2 is triggered when condition 1 is met, and display step 3 is triggered when condition 2 is met.

[0109] Fig. 9 The present invention describes the design of another nonlinear step in the display process design stage, in which after display step 1 is executed, there may be multiple subsequent conditions. When trigger condition 3 is met, display step 4 will be triggered. When trigger condition 2 is met, display step 2 will be triggered. When trigger condition 2 is not met, display step 3 will be triggered.

[0110] In some embodiments, the trigger condition may be in various forms, such as a time condition, a location condition, a direction condition, a user input event, a pre-stored program script, or a custom program script.

[0111] In some embodiments, the trigger condition may be a timer, such as triggering the display of a display step after a predefined period of time after the display program starts running; or triggering the display of another display step after a predetermined period of time after the display of one display step.

[0112] In some embodiments, the trigger condition may be the distance between the user of the head mounted display device 30 and a specific location in 3D space. For example, when the distance between the user of the head mounted display device 30 and a specific location is less than or greater than a threshold, the display step specified by the trigger condition is started or stopped.

[0113] In some embodiments, the trigger condition may be a spatial range, for example, when the user of the head mounted display device 30 enters or leaves a preset spatial area, the display steps specified by the trigger condition are started or stopped.

[0114] In some embodiments, the trigger condition may be the orientation of the head of the user of the head mounted display device 30 in 3D space. For example, when the head of the user of the head mounted display device 30 faces or turns away from a direction interval, the display step specified by the trigger condition is started or stopped.

[0115] In some embodiments, the trigger condition may also be a user input event, such as the user of the head mounted display device 30 pressing a physical button or inputting an indication signal in the user interface, etc. In some embodiments, the user input event may also be generated by the user of the head mounted display device 30 through an external hardware that has a data connection with the head mounted display device 30, such as a Bluetooth headset, a smart phone, a tablet computer, etc.

[0116] In some embodiments, the trigger condition may also be a pre-stored program script, such as a trigger condition set based on a visual algorithm or program algorithm, such as a certain scene, object, etc. falling into the visual range of the head mounted display device 30, triggering a certain display step; or a trigger condition set based on a network data event, such as a certain device connecting to a specified network, triggering a specific display step, etc. In some embodiments, a user may define a trigger condition through a custom program script according to their own needs.

[0117] In some embodiments, the trigger condition can also be a combination of multiple trigger conditions after logical operation. For example, trigger condition C can be defined as condition A and condition B are met at the same time. For example, condition C can be defined as at least one of condition A and condition B is met, and so on.

[0118] In some embodiments, the attributes of the virtual content 22 may also include: the size, color, animation behavior, play speed or volume of the virtual content 22 and other attribute information related to the display, such as the color of the indicator arrow, the play speed and volume of the audio, etc.

[0119] In some embodiments, the placement of the virtual content 22 in the 3D space may be configured as follows:

[0120] Obtaining a reference position based on the 3D space, that is, establishing a reference coordinate system 11;

[0121] The virtual content 22 is moved to the position to be placed by an anchor bound to the virtual content 22, the relationship between the anchor and the reference position (i.e. the coordinates of the anchor in the reference coordinate system 11) is calculated and recorded, and the placed position is defined as the anchor position;

[0122] The placement position of the virtual content 22 is determined based on the anchor position.

[0123] In some embodiments, the reference positioning can be obtained by identifying and locating the reference object 10 in the 3D space. In some embodiments, the pose information of the virtual content 22 finally displayed is defined in at least one reference coordinate system 11, and the reference coordinate system 11 can be defined on a reference object 10, and the reference object 10 can be a fixed environment, such as a room or a venue; it can also be defined on an object, such as an industrial equipment, furniture, home appliance, electronic product, vehicle, etc.; it can also be a logo, such as a specific pattern logo, etc.

[0124] In some embodiments, the reference coordinate system 11 may also be defined to move and / or rotate synchronously with the head mounted display device 30 itself, and the position and posture of the reference coordinate system 11 and the position and posture of the head mounted display device 30 change in the same manner or trend. In some embodiments, if the reference coordinate system 11 is not defined on the head mounted display device 30 itself, the head mounted display device 30 may establish a reference coordinate system 11 and record its identification features, such as the encoded content of the QR code, by identifying, locating and / or tracking at least one specific pattern marker pre-arranged in the environment, such as a two-dimensional code, a barcode, a picture, etc.

[0125] In other embodiments, the head mounted display device 30 can identify and locate the reference object 10 through the visual features of the reference object 10 itself, such as visual feature points, lines, graphics, etc. In some embodiments, the user can record a video or image of the reference object 10, and use the recorded material for training to obtain an artificial intelligence model for identifying and locating the reference object 10. When the user confirms the posture of a virtual content 22 to be displayed, the head mounted display device 30 calculates and records its posture relative to the reference coordinate system 11. In some embodiments, after the user confirms the layout operation of a virtual content 22, the content layout module will associate its corresponding reference coordinate system 11 and its posture information with the display step information of the virtual content 22, so as to reproduce the user's layout of the virtual content 22 in the corresponding display step and the corresponding reference coordinate system 11 during the display and use phase.

[0126] In some embodiments, the user can select the virtual content 22 that he wants to arrange currently through the user interaction interface provided by the content arrangement module, and send the information to the head mounted display device 30 through the data communication connection to help the head mounted display device 30 select the correct virtual content 22 for visual preview and arrangement data entry. In some embodiments, the user can also make additional modifications to the properties of the virtual content 22 through the content arrangement module or the user interface in the head mounted display device 30, such as further adjusting the display size, color, playback speed, volume of the virtual content 22, etc.

[0127] In some embodiments, the anchor can be an entity that is easy to move and locate and has a display function, preferably a handheld mobile device 20, such as a mobile phone, a tablet, a handheld display screen, etc., which has a relatively regular physical structure, is easy to identify and locate, and can provide an operational user interface.

[0128] In some embodiments, the anchor has a positioning pattern 21 , and the anchor position is acquired by identifying and locating the positioning pattern 21 .

[0129] In some embodiments, the reference object 10 and the positioning pattern 21 may be identified and located using an image acquisition device mounted on the head mounted display device 30. For example, a camera, an infrared camera, a depth camera, etc. may be used to acquire at least one picture containing the positioning pattern 21, and the position and posture of the handheld mobile device 20 in the 3D space may be calculated using visual features in the positioning pattern 21, such as points, lines, and outer contours. In some embodiments, the positioning pattern 21 may be a picture, a QR code, a barcode, a specific graphic, etc.

[0130] In some embodiments, the positioning pattern 21 may be pre-stored in the content placement module; in other embodiments, the content placement module may download the positioning pattern 21 from other devices or software modules, such as the built-in software of the head mounted display device 30, the display process design module, or a server program, etc. In some embodiments, the head mounted display device 30 may use the positioning pattern 21 to identify and locate the handheld mobile device 20 at least once, and then subsequently locate the handheld mobile device 20 by identifying and locating the visual features of the handheld mobile device 20.

[0131] In some embodiments, the anchor position may be determined as the placement posture of the virtual content 22 , that is, the head mounted display device 30 directly uses the posture in the 3D space obtained by positioning as the posture to anchor a virtual content 22 .

[0132] In some embodiments, the anchor position may be subjected to mathematical operations to determine the placement posture of the virtual content 22. For example, a posture offset may be added to the anchor position.

[0133] In some embodiments, the head mounted display device 30 may select a portion of the posture information obtained by positioning as the posture of the anchored virtual content 22 , such as using the position coordinates of the positioning result and its orientation on the horizontal plane as the position and orientation of the virtual content 22 .

[0134] In some embodiments, the head-mounted display device 30 can overlay the virtual content 22 onto the real world, and update the display posture of the virtual content 22 in real time according to the posture information obtained by positioning, to help users visually preview the effects of content layout. Users can also confirm or cancel the results of content layout through the user interaction interface provided by the content layout module.

[0135] In some embodiments, the user may also make additional adjustments to the posture of the virtual content 22 through the content arrangement module, such as adding additional offsets to the posture.

[0136] It should be noted that the above description of the relevant process is only for example and explanation, and does not limit the scope of application of the present application. For those skilled in the art, various modifications and changes can be made to the process under the guidance of the present application. However, these modifications and changes are still within the scope of the present application.

[0137] In some other embodiments of the present invention, a human-computer interaction device for mixed reality is provided, comprising at least one processor, at least one handheld mobile device 20 and at least one head-mounted display device 30, wherein the processor is configured to execute a display process design step; the handheld mobile device 20 is configured to cooperate with the head-mounted display device 30 to execute a content layout step; and the head-mounted display device 30 is configured to execute a display usage step.

[0138] In some embodiments, the display and use module can run on multiple head-mounted display devices 30. When a head-mounted display device 30 establishes a reference coordinate system 11, the head-mounted display device 30 can share the reference coordinate system 11 with other head-mounted display devices 30. Other head-mounted display devices 30 can indirectly calculate the reference coordinate system 11 through the relative position and posture relationship between them and the head-mounted display device 30.

[0139] In some embodiments, the processor can be independent, for example, it can be an independent terminal; the processor can also be integrated in the head-mounted display device 30, for example, the display process design module and the display usage module are integrated in a head-mounted display device 30; the processor can also be integrated in the handheld mobile device 20, for example, the display process design module and the content layout module are integrated in the same handheld mobile device 20.

[0140] The beneficial effects that may be brought about by the embodiments of the present application include but are not limited to: (1) using code-free means to edit, generate and use various virtual contents without the need for customers to have relevant software development experience; (2) in the real world, corresponding virtual resources can be displayed according to the customer's personalized needs.

[0141] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other beneficial effects that may be obtained.

[0142] The above content describes the present application and / or some other examples. According to the above content, the present application can also make different variations. The subject matter disclosed in the present application can be implemented in different forms and examples, and the present application can be applied to a large number of applications. All applications, modifications and changes required for protection in the claims belong to the scope of the present application.

[0143] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment", "an embodiment", or "another embodiment" or "another embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0144] Those skilled in the art will appreciate that the contents disclosed in this application may be subject to various variations and improvements. For example, the different system modules described above are all implemented through hardware devices, but may also be implemented only through software solutions. For example: installing the system on an existing server.

[0145] All or part of the Software may from time to time be communicated over a network, such as the Internet or other communications network. Such communications enable the Software to be loaded from one computer device or processor to another.

[0146] In addition, unless explicitly stated in the claims, the use of numbers, letters, or other names in this application is not intended to limit the order of the processes and methods of this application. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of the present application. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0147] Similarly, it should be noted that in order to simplify the description of the disclosure of this application and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this application, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the object of this application are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0148] Finally, it should be understood that the embodiments described in this application are only used to illustrate the principles of the embodiments of the present application. Other variations may also fall within the scope of the present application. Therefore, as an example and not a limitation, the alternative configurations of the embodiments of the present application may be considered to be consistent with the teachings of the present application. Accordingly, the embodiments of the present application are not limited to the embodiments explicitly introduced and described in the present application.

Claims

1. A human-computer interaction method for mixed reality, It is characterized in that include: The display process design includes: designing at least one display step, wherein the display step includes at least one virtual content to be displayed; and designing triggering conditions for each display step; Content arrangement, including: setting properties of the virtual content, the properties at least including the placement position of the virtual content in the 3D space; and The display and use comprises: in each of the display steps, displaying corresponding virtual content based on the set attributes.

2. The human-computer interaction method for mixed reality according to claim 1, It is characterized in that The virtual content includes at least one or more of the following combinations: text, picture, video, audio, 3D model and animation.

3. The human-computer interaction method for mixed reality according to claim 1, It is characterized in that The multiple display steps are displayed in sequence according to a fixed order.

4. The human-computer interaction method for mixed reality according to claim 1, It is characterized in that The multiple display steps are displayed according to the triggering conditions.

5. The human-computer interaction method for mixed reality according to claim 4, It is characterized in that The trigger conditions include time conditions, location conditions, direction conditions, user input events, pre-stored program scripts or custom program scripts.

6. The human-computer interaction method for mixed reality according to claim 5, It is characterized in that The user input event includes pressing a physical key or inputting an indication signal.

7. The human-computer interaction method for mixed reality according to claim 5, It is characterized in that The trigger condition is a combination of multiple trigger conditions after logical operation.

8. The human-computer interaction method for mixed reality according to claim 1, It is characterized in that The attributes of the virtual content also include: size, color, animation behavior, play speed or volume of the virtual content.

9. The human-computer interaction method for mixed reality according to claim 1, It is characterized in that The method for setting the placement position of the virtual content in the 3D space includes: Obtain reference positioning based on 3D space; The virtual content is moved to a position to be placed by an anchor bound to the virtual content, a relationship between the anchor and the reference position is calculated and recorded, and the placed position is defined as the anchor position; The placement position of the virtual content is determined based on the anchor position.

10. The human-computer interaction method for mixed reality according to claim 9, It is characterized in that The reference positioning is obtained by identifying and positioning a reference object in the 3D space.

11. The human-computer interaction method for mixed reality according to claim 10, It is characterized in that The reference object at least includes an environment, an object or a logo.

12. The human-computer interaction method for mixed reality according to claim 9, It is characterized in that The anchor is a handheld mobile device.

13. The human-computer interaction method for mixed reality according to claim 10, It is characterized in that The anchor is provided with a positioning pattern, and the anchor position is acquired by identifying and locating the positioning pattern.

14. The human-computer interaction method for mixed reality according to claim 13, It is characterized in that The positioning pattern includes a picture, a QR code, a barcode or a specific graphic.

15. The human-computer interaction method for mixed reality according to claim 13, It is characterized in that The reference object and the positioning pattern are identified and positioned using an image acquisition device mounted on a head mounted display device.

16. The human-computer interaction method for mixed reality according to claim 9, It is characterized in that The anchor position is determined as the placement posture of the virtual content.

17. The human-computer interaction method for mixed reality according to claim 9, It is characterized in that The anchor position is mathematically calculated to determine the placement position of the virtual content.

18. A human-computer interaction system for mixed reality, It is characterized in that include: A display process design module, used to design at least one display step, wherein the display step includes at least one virtual content to be displayed; Design the triggering conditions for each display step; A content arrangement module, used to set the attributes of the virtual content, wherein the attributes at least include the placement position of the virtual content in the 3D space; as well as The display and use module is used to display the corresponding virtual content based on the set attributes in each of the display steps.

19. The human-computer interaction system for mixed reality according to claim 18, It is characterized in that The virtual content includes at least one or more of the following combinations: text, picture, video, audio, 3D model and animation.

20. The human-computer interaction system for mixed reality according to claim 18, It is characterized in that The multiple display steps are displayed in sequence according to a fixed order.

21. The human-computer interaction system for mixed reality according to claim 18, It is characterized in that The multiple display steps are displayed according to the triggering conditions.

22. The human-computer interaction system for mixed reality according to claim 21, It is characterized in that The trigger conditions include time conditions, location conditions, direction conditions, user input events, pre-stored program scripts or custom program scripts.

23. The human-computer interaction system for mixed reality according to claim 22, It is characterized in that The user input event includes pressing a physical key or inputting an indication signal.

24. The human-computer interaction system for mixed reality according to claim 22, It is characterized in that The trigger condition is a combination of multiple trigger conditions after logical operation.

25. The human-computer interaction system for mixed reality according to claim 18, It is characterized in that The attributes of the virtual content also include: size, color, animation behavior, play speed or volume of the virtual content.

26. The human-computer interaction system for mixed reality according to claim 18, It is characterized in that The method for setting the placement position of the virtual content in the 3D space includes: Obtain reference positioning based on 3D space; The virtual content is moved to a position to be placed by an anchor bound to the virtual content, a relationship between the anchor and the reference position is calculated and recorded, and the placed position is defined as the anchor position; The placement position of the virtual content is determined based on the anchor position.

27. The human-computer interaction system for mixed reality according to claim 26, It is characterized in that The reference positioning is obtained by identifying and positioning a reference object in the 3D space.

28. The human-computer interaction system for mixed reality according to claim 27, It is characterized in that The reference object at least includes an environment, an object or a logo.

29. The human-computer interaction system for mixed reality according to claim 26, It is characterized in that The anchor is a handheld mobile device.

30. The human-computer interaction system for mixed reality according to claim 27, It is characterized in that The anchor is provided with a positioning pattern, and the anchor position is acquired by identifying and locating the positioning pattern.

31. The human-computer interaction system for mixed reality according to claim 30, It is characterized in that The positioning pattern includes a picture, a QR code, a barcode or a specific graphic.

32. The human-computer interaction system for mixed reality according to claim 30, It is characterized in that The reference object and the positioning pattern are identified and positioned using an image acquisition device mounted on a head mounted display device.

33. The human-computer interaction system for mixed reality according to claim 26, It is characterized in that The anchor position is determined as the placement posture of the virtual content.

34. The human-computer interaction system for mixed reality according to claim 26, It is characterized in that The anchor position is mathematically calculated to determine the placement position of the virtual content.

35. A human-computer interaction device for mixed reality, applied in the method according to any one of claims 1 to 17, It is characterized in that The apparatus comprises at least one processor, at least one handheld mobile device and at least one head mounted display device, The processor is configured to perform steps of displaying a flow design; The handheld mobile device is configured to cooperate with the head mounted display device to perform a content arrangement step; The head mounted display device is configured to perform the steps of demonstrating use.

36. The human-computer interaction device for mixed reality according to claim 35, It is characterized in that The processor is integrated in the head mounted display device.

37. The human-computer interaction device for mixed reality according to claim 35, It is characterized in that The processor is integrated into the handheld mobile device.