Augmented reality interaction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an augmented reality interaction method and device, electronic equipment and a storage medium, and relates to the technical field of augmented reality, and the method comprises the steps: when the augmented reality equipment detects wearing, generating a three-dimensional display space, recognizing a current interaction scene, and determining at least one interaction module corresponding to the interaction scene, and dividing at least one interaction area in the three-dimensional display space according to the interaction module, and responding to an interaction instruction through the interaction area to obtain interaction result information. By determining the interaction module corresponding to the interaction scene and displaying the interaction module according to the interaction area in the three-dimensional display space of the augmented reality device, the augmented reality-based interaction module partition display is realized, and the problems of content display disorder and inconvenient user operation are relieved.

Description

Technical Field

[0001] This application relates to the field of extended reality technology, specifically to an extended reality interaction method, device, electronic device, and storage medium. Background Technology

[0002] Extended Reality (XR) is a comprehensive term encompassing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). Extended Reality technology expands human perception and interaction, allowing interaction with the real world through digital technology to create virtual, augmented, and mixed reality experiences. It is rapidly developing in fields such as entertainment, gaming, education, healthcare, enterprise production, and social communication, demonstrating enormous potential and impact. Wearable smart glasses, as a representative of Extended Reality devices, are considered the best carriers of "XR+AI" technology and have already been widely applied in areas such as industrial inspection, education and training, and marketing demonstrations.

[0003] As extended reality technology or devices develop, more and more content is displayed in the three-dimensional display space of extended reality devices, leading to problems such as cluttered content display and inconvenient user operation. Summary of the Invention

[0004] This application provides an extended reality interactive method, device, electronic device, and storage medium that can display content in different areas, alleviating the problems of cluttered content display and inconvenient user operation.

[0005] In a first aspect, embodiments of this application provide an extended reality interaction method applied to an extended reality device, the method comprising: A three-dimensional display space is generated upon detection of wearing; Identify the current interaction scenario and determine at least one interaction module corresponding to the interaction scenario; The interactive module divides at least one interactive area in the three-dimensional display space; The interactive area responds to interactive commands and provides interactive result information.

[0006] Secondly, embodiments of this application also provide an extended reality interactive device, applied to an extended reality device, the device comprising: The generation module is used to generate a three-dimensional display space when wearing is detected; The identification module is used to identify the current interaction scenario and determine at least one interaction module corresponding to the interaction scenario; A partitioning module is used to partition at least one interactive area in the three-dimensional display space according to the interactive module; The response module is used to respond to interactive commands through the interactive area and obtain interactive result information.

[0007] Optionally, in some embodiments of this application, before responding to the interaction command through the interaction area to obtain the interaction result information, the method further includes: Determine the module feature information of the interaction module corresponding to the interaction area; Configure first interactive attribute information for the interactive area based on the module feature information; The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: Based on the first interactive attribute information, the interactive area is controlled to respond to the interactive command, and the interactive result information is obtained.

[0008] Optionally, in some embodiments of this application, the interaction instruction includes a first user action and a second user action, and the step of responding to the interaction instruction through the interaction area to obtain interaction result information includes: The target interaction area is determined from the interaction area based on the first user behavior; The interaction result information is obtained by responding to the second user's behavior through the target interaction area; The first user behavior or the second user behavior includes at least one of facial orientation behavior, gaze behavior, voice behavior, or gesture behavior; The interaction result information includes at least one of the following: second interaction attribute information obtained from the interaction attribute configuration, voice response result information for voice control commands, or gesture response result information for gesture control commands.

[0009] Optionally, in some embodiments of this application, after determining the target interaction area from the interaction area through the first user behavior, the method further includes: Display a visual identifier for the target interaction area, wherein the visual identifier is used to identify the currently selected target interaction area or to identify the third interaction attribute information corresponding to the target interaction area; Wherein, the second interactive attribute information or the third interactive attribute information includes at least one of the following: interactive method information, role interactive permission information, or behavior response logic information; The interaction method information includes at least one of voice interaction, eye-tracking interaction, and gesture interaction; the role interaction permission information includes at least one of read-only, hidden, or editable; and the behavior response logic information includes at least one of responding or not responding.

[0010] Optionally, in some embodiments of this application, the interaction result information includes regional response result information and associated response result information; The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: The interactive area responds to the interactive command, and the response result information of the area is obtained. Determine the associated region corresponding to the interaction region, and adjust the associated region according to the region response result information to obtain the associated response result information.

[0011] Optionally, in some embodiments of this application, the interactive area includes an editing area, the associated area includes a display area, the interactive instruction includes an editing instruction, the area response result information includes editing result information for the editing area, and the associated response result information includes display result information for the display area.

[0012] Optionally, in some embodiments of this application, the method further includes: Display task phase indication information, which is used to indicate the current phase to be executed for the collaborative task; Display operation controls, which, when triggered, are used to adjust the task stage indication information. The operation controls include at least one of branch jump controls or process rollback controls. Each of the aforementioned interactive areas corresponds to a stage to be executed in a collaborative task, and the task stage indication information indicates the currently pending interactive area. The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: Based on the interactive area corresponding to the task stage indication information, the interactive instruction is responded to, and the interactive result information is obtained.

[0013] Optionally, in some embodiments of this application, before responding to the interaction command through the interaction area to obtain the interaction result information, the method further includes: Determine user preference information or behavioral hotspot recording information; Based on the user preference information or the behavior hotspot recording information, the distribution of the interactive area in the three-dimensional display space is adjusted, and the distribution adjustment includes at least one of display position adjustment, area outline shrinkage, or interactive area hiding. The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: The interaction result information is obtained by responding to the interaction command through the adjusted interaction area.

[0014] Optionally, in some embodiments of this application, determining at least one interactive module corresponding to the interactive scenario includes: Determine at least one interactive module corresponding to the interactive scene based on the scene module mapping relationship; The scene module mapping relationship includes the mapping relationship between preset interactive scenes and at least one preset interactive module; The three-dimensional display space is generated based on the actual spatial structure; The interaction module includes functional modules operable in the interaction scenario.

[0015] Thirdly, embodiments of this application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the extended reality interaction method described above.

[0016] Fourthly, embodiments of this application also provide a storage medium, which includes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps in the extended reality interaction method described above.

[0017] Fifthly, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in embodiments of this application.

[0018] In summary, when the extended reality device of this application detects wearing, it generates a three-dimensional display space, identifies the current interaction scene, determines at least one interaction module corresponding to the interaction scene, divides at least one interaction area in the three-dimensional display space according to the interaction module, responds to interaction commands through the interaction area, and obtains interaction result information.

[0019] In this embodiment, the interactive modules corresponding to the interactive scene are determined and displayed in the three-dimensional display space of the extended reality device according to the interactive area, thereby realizing the partitioned display of interactive modules based on extended reality and alleviating the problems of chaotic content display and inconvenient user operation. Attached Figure Description

[0020] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a schematic diagram of a scenario in which the extended reality device provided in this application executes the extended reality interaction method; Figure 2 This is a flowchart illustrating the extended reality interaction method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the three-dimensional display space provided in the embodiments of this application; Figure 4 This is another schematic diagram of the three-dimensional display space provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the extended reality interactive device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0022] Explanation of icon numbers: 101-Extended Reality Device; 501-Generation Module; 502-Recognition Module; 503-Division Module; 504-Response Module; 601-Processor; 602-Memory; 603-Power Supply; 604-Input Unit. Detailed Implementation

[0023] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] In the description of the embodiments of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," "third," and "fourth" may explicitly or implicitly include one or more features. In the description of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0025] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0026] This application provides an extended reality interaction method, apparatus, electronic device, and computer-readable storage medium. Specifically, this application provides an extended reality interaction apparatus suitable for electronic devices (e.g., extended reality devices). The electronic device includes an extended reality device, which includes, but is not limited to, head-mounted displays, wearable glasses, and can be an integrated extended reality device built into a computing processing unit, or a separate extended reality device external to the computing processing unit. The extended reality device includes, but is not limited to, airborne optical display systems (i.e., head-up display systems) used in vehicles such as aircraft, automobiles, and ships, such as AR-HUD (Augmented Reality Head-Up Display) mounted on intelligent connected vehicles; extended reality applications (such as extended reality games, virtual tourism, telemedicine, or virtual experiments) used in handheld mobile devices such as mobile phones, laptops, and tablets; and near-eye display systems used in wearable devices such as head-mounted displays and smart glasses.

[0027] For example, please see Figure 1 , Figure 1 This is a schematic diagram of a scenario in which an extended reality device, according to an embodiment of this application, executes the extended reality interaction method. Specifically, the execution process of the extended reality device executing the extended reality interaction method is as follows: When the extended reality device 101 detects that the device is being worn, it generates a three-dimensional display space, identifies the current interaction scene, determines at least one interaction module corresponding to the interaction scene, divides at least one interaction area in the three-dimensional display space according to the interaction module, responds to interaction commands through the interaction area, and obtains interaction result information.

[0028] For example, in a speech scenario, the prompting module, display module, and control module required for the speech can be displayed in separate sections, so that each module is displayed in an orderly manner, making it easy for users to browse and operate.

[0029] In summary, the embodiments of this application determine the interactive modules corresponding to the interactive scene and display the interactive modules according to the interactive area in the three-dimensional display space of the extended reality device, thereby realizing the partitioned display of interactive modules based on extended reality and alleviating the problems of chaotic content display and inconvenient user operation.

[0030] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0031] See also Figure 2 , Figure 2This is a flowchart illustrating an extended reality interaction method provided in an embodiment of this application. Although the flowchart shows a logical order, in some cases, the steps shown or described can be performed in a different order than that shown in the flowchart. Specifically, this extended reality interaction method is applied to an extended reality device, and the specific flow of the extended reality interaction method is as follows: S201. When wearing is detected, a three-dimensional display space is generated.

[0032] In this embodiment of the application, the distance to the user's head can be detected by a distance sensor, thereby determining whether the augmented reality device is being worn.

[0033] The three-dimensional display space is the space in which extended reality devices can project and output information. For example, please refer to [link to relevant documentation]. Figure 3 As shown, an extended reality device forms a display space centered on it, extending 180, 360, or 720 degrees around it. For example, after a user wears the extended reality device, information can be displayed in the space in front of the user, or on the user's sides and back, or above the user.

[0034] S202. Identify the current interaction scenario and determine at least one interaction module corresponding to the interaction scenario.

[0035] An interaction scenario refers to the specific context in which a user and a system (such as extended reality devices, smart terminals, etc.) exchange information and perform tasks through input / output methods within a specific environment or application context. It describes how the user interacts with the device, how the system responds, and the contextual conditions under which the interaction occurs. For example, such interaction scenarios include presentations, writing sessions, or meetings using extended reality devices.

[0036] Interactive modules are user-operable functional modules in interactive scenarios. For example, interactive modules for speech scenarios include editing modules, prompting modules, display modules, and control modules; interactive modules for writing scenarios include editing modules, display modules, search modules, and control modules; and interactive modules for meeting scenarios include dialogue text display modules, avatar display modules, and virtual meeting screen display modules.

[0037] Understandably, identifying multiple interactive modules corresponding to an interactive scenario helps in processing the display of each interactive module to meet display requirements.

[0038] S203. Divide at least one interactive area in the three-dimensional display space according to the interactive module.

[0039] In this embodiment, the interactive area is a portion of the three-dimensional display space used to carry and display content. For example, the interactive area includes a prompting area, an editing area, a display area, and a control area.

[0040] For example, in this embodiment, a corresponding interactive area is configured for each interactive module, enabling each interactive module to be displayed separately according to its designated area. For example, please refer to... Figure 4 As shown, the three-dimensional display space is divided into a prompting area corresponding to the prompting module, an information display area corresponding to the display module, and a control area corresponding to the control module.

[0041] S204. Respond to the interaction command through the interaction area to obtain the interaction result information.

[0042] Interactive commands refer to instructions that are intended to elicit specific actions from the augmented reality device. These commands may include navigation commands, dialogue commands, editing commands, or permission configuration commands. It is understood that these interactive commands can be obtained by receiving the user's voice, receiving the user's text input, detecting the user's distance, capturing the user's image, or recognizing the user's gestures.

[0043] The interaction result information is the result obtained after responding to the interaction command and executing the response action. For example, responding to the edit command yields the edited text, and responding to the permission configuration command yields the permissions for the user role.

[0044] It is understandable that by responding to interactive commands through interactive areas, user operations can be applied to the corresponding interactive modules, avoiding the problem of inaccurate application to specific interactive modules due to the disordered display of content in various modules, thus preventing inconvenience in operation.

[0045] In summary, the embodiments of this application determine the interactive modules corresponding to the interactive scene and display the interactive modules according to the interactive area in the three-dimensional display space of the extended reality device, thereby realizing the partitioned display of interactive modules based on extended reality and alleviating the problems of chaotic content display and inconvenient user operation.

[0046] Optionally, after dividing the interactive area, interactive attributes for each interactive area can be configured to enrich the interactive diversity of the interactive areas. That is, optionally, in some embodiments of this application, before the step of "responding to interactive instructions through the interactive area to obtain interactive result information", the method further includes: Determine the module feature information of the interaction module corresponding to the interaction area; Configure first interactive attribute information for the interactive area based on the module feature information; The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: Based on the first interactive attribute information, the interactive area is controlled to respond to the interactive command, and the interactive result information is obtained.

[0047] It should be noted that the module feature information reflects the characteristics or type of the interactive module. For example, the module feature information includes, but is not limited to, editing type, display type, or control type.

[0048] The first interactive attribute information refers to the interactive attributes corresponding to the interactive area, which is used to control the response of the interactive area to interactive commands. For example, the first interactive attribute information includes, but is not limited to, interaction method information, role interaction permission information, or behavior response logic information. The interaction method information includes voice interaction, eye-tracking interaction, gesture interaction, etc.; the role interaction permission information includes read-only, hidden, or editable, etc.; and the behavior response logic information includes responding or not responding, etc.

[0049] In this embodiment, the first interactive attribute information of the corresponding interactive area is automatically configured based on the module feature information of the interactive module. For example, if the interactive module is a display module, the corresponding module feature information is display type, and the corresponding first interactive attribute information is that it supports voice interaction and is read-only; if the interactive module is an editing type, the corresponding module feature information is editing type, and the corresponding first interactive attribute information is that it supports voice interaction, eye-tracking interaction, and gesture interaction and is editable, etc.

[0050] Correspondingly, the system responds to interaction commands based on the configured first interaction attribute information to obtain interaction result information. For example, if the first interaction attribute information of the interaction area is read-only, then the system does not respond to editing interaction commands; if the first interaction attribute information of the interaction area is editable, then the system responds to editing interaction commands.

[0051] Optionally, in some embodiments of this application, a target interactive area to be interacted with can be determined from multiple interactive areas in the three-dimensional display space based on user behavior, so as to respond to the user's interaction command through the target interactive area. That is, optionally, in some embodiments of this application, the interaction command includes a first user behavior and a second user behavior, and the step "responding to the interaction command through the interactive area to obtain interaction result information" includes: The target interaction area is determined from the interaction area based on the first user behavior; The interaction result information is obtained by responding to the second user's behavior through the target interaction area; The first user behavior or the second user behavior includes at least one of facial orientation behavior, gaze behavior, voice behavior, or gesture behavior; The interaction result information includes at least one of the following: second interaction attribute information obtained from the interaction attribute configuration, voice response result information for voice control commands, or gesture response result information for gesture control commands.

[0052] For example, a user can select a target interactive area from multiple interactive areas in a 3D display space by facial orientation, voice commands, or eye gaze. The augmented reality device then analyzes the user's facial orientation, voice commands, or eye gaze direction to determine the target interactive area with a matching position and angle from multiple interactive areas at different locations in the 3D display space. Gestures, such as clicking, pointing, clenching a fist, or extending a palm, correspond to different interactive commands.

[0053] Correspondingly, after determining the target interaction area, the system can also respond to user behavior to obtain corresponding interaction result information. For example, if a user selects a preview area by facial orientation and gazes at the preview area for 1 second, the interaction attributes of the preview area can be configured as read-only or voice commands can be disabled. If a user selects an editing area by gesture (e.g., pointing with a finger), and a voice instruction is given to enable voice and gesture input, then voice and gesture commands can be enabled.

[0054] It is understandable that when responding to user behavior, the interactive attributes of the interactive area can also be configured to obtain the corresponding second interactive attribute information. It is also understood that both the first and second interactive attribute information refer to the interactive attributes of the interactive area; the distinction between their names is merely based on the needs of different configuration methods, different implementation methods, and step descriptions, and does not indicate that they are different types of information.

[0055] It is understandable that, in addition to adjusting the interactive attributes of the interactive area through interactive commands (such as user behavior), the interactive area can also respond to user interactive commands and execute corresponding business logic operations. For example, it can respond to user voice control commands to obtain edited text through speech-to-text conversion (i.e., corresponding to voice response result information), or respond to user gesture control commands to convert text to speech and broadcast it (i.e., corresponding to gesture response result information), etc.

[0056] Optionally, in embodiments of this application, after determining the target interaction area, a selection prompt can also be generated for the user through visual identifiers. That is, optionally, in some embodiments of this application, after the step "determining the target interaction area from the interaction area through the first user behavior", the method further includes: Display a visual identifier for the target interaction area, wherein the visual identifier is used to identify the currently selected target interaction area or to identify the third interaction attribute information corresponding to the target interaction area; Wherein, the second interactive attribute information or the third interactive attribute information includes at least one of the following: interactive method information, role interactive permission information, or behavior response logic information; The interaction method information includes at least one of voice interaction, eye-tracking interaction, and gesture interaction; the role interaction permission information includes at least one of read-only, hidden, or editable; and the behavior response logic information includes at least one of responding or not responding.

[0057] For example, after selecting an editing area, a blue box is used to identify the currently selected target interactive area as the editing area, and the blue box can also indicate that the interactive attribute of the editing area supports gesture input; similarly, after selecting a control area, a red box is used to identify the currently selected target interactive area as the control area, and the red box can also indicate that the interactive attribute of the control area is enabled only by the primary user, and cannot be enabled by secondary users or other user roles. It can be understood that the blue or red box is a virtual box generated in the three-dimensional display space.

[0058] Similarly, the third interaction attribute information, the first interaction attribute information, and the second interaction attribute information are all interaction attributes for the interaction area. The distinction between their names is only based on the needs of different configuration methods, different embodiments, and step descriptions, and does not indicate that they are different types of information. Accordingly, the second interaction attribute information or the third interaction attribute information also includes, but is not limited to, interaction method information, role interaction permission information, or behavior response logic information; wherein, the interaction method information includes voice interaction, eye-tracking interaction, gesture interaction, etc., the role interaction permission information includes read-only, hidden, or editable, etc., and the behavior response logic information includes respond or not respond, etc.

[0059] Optionally, in this embodiment of the application, when responding to interactive commands based on the interactive area, the associated interactive areas can also be linked for control based on business needs. That is, optionally, in some embodiments of this application, the interactive result information includes area response result information and associated response result information. The step "obtaining interactive result information by responding to interactive commands through the interactive area" includes: The interactive area responds to the interactive command, and the response result information of the area is obtained. Determine the associated region corresponding to the interaction region, and adjust the associated region according to the region response result information to obtain the associated response result information.

[0060] Among them, the regional response result information is the result of the interactive region responding to the interactive command, and the associated response result information is the result of the linkage adjustment based on the response of the interactive region.

[0061] It is understandable that interactive areas respond to interactive commands and synchronize information and status with associated interactive areas to ensure the timeliness and timely updates of information.

[0062] For example, taking the interactive area as the editing area, the associated areas include the display area. Correspondingly, interactive commands include editing commands, area response result information includes editing result information for the editing area, and associated response result information includes display result information for the display area. For instance, after paragraph content is entered in the editing area, the display area automatically refreshes and synchronizes the version after a 0.5-second delay. Or, taking the control area as an example, after a lock command is issued from the control area, other interactive areas cannot respond to interactive input.

[0063] Understandably, the design of interconnected control across various interactive areas aims to ensure timely content updates, maintain information consistency, and enhance the user experience when displaying content in partitioned areas based on extended reality.

[0064] Understandably, the configuration of role-based access permissions for interactive areas allows multiple users to use the extended reality device and operate each interactive area according to their respective permissions. For example, the editing area can be operated by the primary user or an administrator user, while the display area can be operated by a regular user or a secondary user.

[0065] In this application embodiment, the configuration of role permission information for the interactive area enables multiple users to collaborate on tasks using the extended reality device. For example, each user operates the corresponding interactive area according to their own permissions to execute their respective branch tasks. Optionally, in some embodiments of this application, the method further includes: Display task phase indication information, which is used to indicate the current phase to be executed for the collaborative task; Display operation controls, which, when triggered, are used to adjust the task stage indication information. The operation controls include at least one of branch jump controls or process rollback controls. Each of the aforementioned interactive areas corresponds to a stage to be executed in a collaborative task, and the task stage indication information indicates the currently pending interactive area. The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: Based on the interactive area corresponding to the task stage indication information, the interactive instruction is responded to, and the interactive result information is obtained.

[0066] The task stage indicator information is used to indicate the execution progress of the current collaborative task. This can be represented by highlighting the task path. For example, for the task of publishing a manuscript, the task path can be represented as "Draft → Proofreading → Practice → Publish". It can be understood that four users can respectively execute the four tasks of "Draft", "Proofreading", "Practice", and "Publish". Accordingly, if the first user completes the "Draft" task in the editing area based on editable interactive attributes, the "Proofreading" task in the corresponding task path for the manuscript publishing task will be highlighted to indicate that the "Proofreading" task is currently required. The second user can then wear the extended reality device and operate the proofreading of the draft in the display area based on read-only interactive attributes.

[0067] The operation controls are used to process task stage indication information and corresponding business logic. These operation controls include branch jump controls or process rollback controls. For example, after operating the branch jump control, the user can directly jump from the "Proofreading" task to the "Publish" task to speed up the publication of the manuscript and improve the publication efficiency. Or, after operating the process rollback control, the user can roll back from the "Proofreading" task to the "Draft" task so that the first user can modify and improve the draft of the manuscript.

[0068] For example, in this embodiment of the application, taking each interactive area as corresponding to each branch task in the collaborative task, the task stage indication information can be represented by sorting the various interactive areas. For example, the control area, editing area, and display area can be arranged in order from front to back (differences in visual distance from the user) or from left to right, and a virtual box can be added to the currently pending interactive area to identify the currently pending interactive area.

[0069] Correspondingly, the extended reality device controls the response to the user's interaction commands based on the interaction area corresponding to the task stage indication information to obtain the interaction result information, that is, to obtain the execution result of the current subtask in the collaborative task.

[0070] Optionally, in some embodiments of this application, when there are multiple interactive areas in the three-dimensional display space, the display of each interactive area in the three-dimensional display space can be dynamically adjusted based on user preferences or the frequency of user operations. That is, optionally, in some embodiments of this application, before the step "responding to interactive instructions through the interactive areas to obtain interactive result information", the method further includes: Determine user preference information or behavioral hotspot recording information; Based on the user preference information or the behavior hotspot recording information, the distribution of the interactive area in the three-dimensional display space is adjusted, and the distribution adjustment includes at least one of display position adjustment, area outline shrinkage, or interactive area hiding. The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: The interaction result information is obtained by responding to the interaction command through the adjusted interaction area.

[0071] User preference information refers to user preferences, such as user-inputted preferences and habits. Behavioral hotspot recording information refers to records of frequently used interaction areas. By analyzing user historical operation records, frequently used interaction areas are determined. For example, if a user consistently uses the display area to showcase movies or TV shows, then that display area is considered a behavioral hotspot, and is recorded in the behavioral task record.

[0072] Distribution adjustment refers to adjusting the display and position of interactive areas in the 3D display space. This adjustment includes, but is not limited to, adjusting display position, shrinking area outlines, or hiding interactive areas. For example, frequently used interactive areas that are preferred by users can be aligned with the user's main visual axis to reduce head shifting and body rotation when using extended reality devices. The display area of ​​interactive areas that have not been used for extended periods can be reduced or hidden to minimize visual interference from these less frequently used areas.

[0073] Accordingly, in this embodiment, the interaction commands are responded to based on the adjusted interaction area to improve the user experience.

[0074] Optionally, in this embodiment, a mapping relationship between each interaction scenario and interaction module can be pre-configured. Based on this mapping relationship, the interaction module corresponding to the current interaction scenario can be quickly determined. That is, optionally, in some embodiments of this application, the step "determining at least one interaction module corresponding to the interaction scenario" includes: The scene module mapping relationship determines at least one interactive module corresponding to the interactive scene, and the scene module mapping relationship includes the mapping relationship between the preset interactive scene and at least one preset interactive module.

[0075] For example, establish mapping relationships between speech scenarios and modules such as editing, prompting, display, and control; establish mapping relationships between writing scenarios and modules such as editing, display, search, and control; and establish mapping relationships between meeting scenarios and modules such as dialogue text display, avatar display, and virtual meeting screen display.

[0076] In this embodiment, the three-dimensional display space can be generated based on the actual spatial structure of the user's location. For example, the three-dimensional display space can be generated based on the spatial structure of a conference scene for a speech, or based on the spatial structure of an office meeting scene. Correspondingly, the distribution of the interactive area can be set based on the characteristics of the actual spatial structure. For example, in the three-dimensional display space corresponding to a speech, the prompting area can be set in the direction of the audience; in a conference scene, the speech obtained by the conference member through speech-to-text conversion can be displayed on the side of the conference member's face (i.e., the display area is displayed on the face of the conference member), etc.

[0077] In summary, the embodiments of this application determine the interactive modules corresponding to the interactive scene and display the interactive modules according to the interactive area in the three-dimensional display space of the extended reality device, thereby realizing the partitioned display of interactive modules based on extended reality and alleviating the problems of chaotic content display and inconvenient user operation.

[0078] It is understood that the extended reality interaction method of this application embodiment is applicable to scenarios with multiple interactive modules, such as scenarios such as speeches / conferences (display modules, control modules, editing modules, etc.) and education (teaching, practice, feedback, etc.).

[0079] This application embodiment sets up interactive modules according to regional partitions, which not only facilitates users to make reasonable use of virtual space and makes the viewing or use path more orderly, avoiding spatial layout chaos, but also allows for the setting of corresponding interactive attributes for each interactive module, enabling differentiated and personalized designs for each interactive area and enhancing the extended reality interactive experience. Furthermore, the configuration of interactive attributes can reduce misoperations; for example, students can only interact in the practical practice area, while other areas are only for viewing, preventing different students from operating AR glasses independently and causing system chaos, reducing the complexity of use, and ensuring user privacy, such as the hiding attribute of privacy areas and read-only, non-editable, and tamper-proof core data.

[0080] To facilitate better implementation of the extended reality interaction method of this application, this application also provides an application processing device based on the above-described extended reality interaction method. The meanings of the terms used are the same as in the extended reality interaction method described above, and specific implementation details can be found in the descriptions of the method embodiments.

[0081] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of the extended reality interaction device provided in the embodiments of this application, wherein the extended reality interaction device is applied to an extended reality device, and the extended reality interaction device can be specifically as follows: The generation module 501 is used to generate a three-dimensional display space when wearing is detected; The identification module 502 is used to identify the current interaction scenario and determine at least one interaction module corresponding to the interaction scenario; The partitioning module 503 is used to partition at least one interactive area in the three-dimensional display space according to the interactive module; The response module 504 is used to respond to the interaction command through the interaction area and obtain the interaction result information.

[0082] Optionally, in some embodiments of this application, before responding to the interaction command through the interaction area to obtain the interaction result information, the method further includes: Determine the module feature information of the interaction module corresponding to the interaction area; Configure first interactive attribute information for the interactive area based on the module feature information; The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: Based on the first interactive attribute information, the interactive area is controlled to respond to the interactive command, and the interactive result information is obtained.

[0083] Optionally, in some embodiments of this application, the interaction instruction includes a first user action and a second user action, and the step of responding to the interaction instruction through the interaction area to obtain interaction result information includes: The target interaction area is determined from the interaction area based on the first user behavior; The interaction result information is obtained by responding to the second user's behavior through the target interaction area; The first user behavior or the second user behavior includes at least one of facial orientation behavior, gaze behavior, voice behavior, or gesture behavior; The interaction result information includes at least one of the following: second interaction attribute information obtained from the interaction attribute configuration, voice response result information for voice control commands, or gesture response result information for gesture control commands.

[0084] Optionally, in some embodiments of this application, after determining the target interaction area from the interaction area through the first user behavior, the method further includes: Display a visual identifier for the target interaction area, wherein the visual identifier is used to identify the currently selected target interaction area or to identify the third interaction attribute information corresponding to the target interaction area; Wherein, the second interactive attribute information or the third interactive attribute information includes at least one of the following: interactive method information, role interactive permission information, or behavior response logic information; The interaction method information includes at least one of voice interaction, eye-tracking interaction, and gesture interaction; the role interaction permission information includes at least one of read-only, hidden, or editable; and the behavior response logic information includes at least one of responding or not responding.

[0085] Optionally, in some embodiments of this application, the interaction result information includes regional response result information and associated response result information; The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: The interactive area responds to the interactive command, and the response result information of the area is obtained. Determine the associated region corresponding to the interaction region, and adjust the associated region according to the region response result information to obtain the associated response result information.

[0086] Optionally, in some embodiments of this application, the interactive area includes an editing area, the associated area includes a display area, the interactive instruction includes an editing instruction, the area response result information includes editing result information for the editing area, and the associated response result information includes display result information for the display area.

[0087] Optionally, in some embodiments of this application, the method further includes: Display task phase indication information, which is used to indicate the current phase to be executed for the collaborative task; Display operation controls, which, when triggered, are used to adjust the task stage indication information. The operation controls include at least one of branch jump controls or process rollback controls. Each of the aforementioned interactive areas corresponds to a stage to be executed in a collaborative task, and the task stage indication information indicates the currently pending interactive area. The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: Based on the interactive area corresponding to the task stage indication information, the interactive instruction is responded to, and the interactive result information is obtained.

[0088] Optionally, in some embodiments of this application, before responding to the interaction command through the interaction area to obtain the interaction result information, the method further includes: Determine user preference information or behavioral hotspot recording information; Based on the user preference information or the behavior hotspot recording information, the distribution of the interactive area in the three-dimensional display space is adjusted, and the distribution adjustment includes at least one of display position adjustment, area outline shrinkage, or interactive area hiding. The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: The interaction result information is obtained by responding to the interaction command through the adjusted interaction area.

[0089] Optionally, in some embodiments of this application, determining at least one interactive module corresponding to the interactive scenario includes: Determine at least one interactive module corresponding to the interactive scene based on the scene module mapping relationship; The scene module mapping relationship includes the mapping relationship between preset interactive scenes and at least one preset interactive module; The three-dimensional display space is generated based on the actual spatial structure; The interaction module includes functional modules operable in the interaction scenario.

[0090] In this embodiment, the generation module 501 first generates a three-dimensional display space when the device is detected to be worn. The recognition module 502 identifies the current interaction scene and determines at least one interaction module corresponding to the interaction scene. The division module 503 divides at least one interaction area in the three-dimensional display space according to the interaction module. The response module 504 responds to the interaction command through the interaction area and obtains the interaction result information.

[0091] In summary, the embodiments of this application determine the interactive modules corresponding to the interactive scene and display the interactive modules according to the interactive area in the three-dimensional display space of the extended reality device, thereby realizing the partitioned display of interactive modules based on extended reality and alleviating the problems of chaotic content display and inconvenient user operation.

[0092] In addition, this application also provides an electronic device, such as Figure 6 As shown, it illustrates a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically: The electronic device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will understand that... Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 601 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 602, and by calling data stored in the memory 602, thereby providing overall monitoring of the electronic device. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 601.

[0093] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0094] The electronic device also includes a power supply 603 that supplies power to the various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 603 may also include one or more DC or AC power supplies, recharging systems, power equipment debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0095] The electronic device may also include an input unit 604, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0096] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 601 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 602 according to the following instructions, and the processor 601 runs the applications stored in the memory 602, thereby implementing the steps in any of the extended reality interaction methods provided in the embodiments of this application.

[0097] When the extended reality device of this application detects wearing, it generates a three-dimensional display space, identifies the current interaction scene, determines at least one interaction module corresponding to the interaction scene, divides at least one interaction area in the three-dimensional display space according to the interaction module, responds to interaction commands through the interaction area, and obtains interaction result information.

[0098] In this embodiment, the interactive modules corresponding to the interactive scene are determined and displayed in the three-dimensional display space of the extended reality device according to the interactive area, thereby realizing the partitioned display of interactive modules based on extended reality and alleviating the problems of chaotic content display and inconvenient user operation.

[0099] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0100] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0101] To this end, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor to perform the steps in any of the extended reality interaction methods provided in this application.

[0102] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0103] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0104] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the extended reality interaction methods provided in this application, the beneficial effects that any of the extended reality interaction methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0105] The above provides a detailed description of an extended reality interaction method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An extended reality interaction method, characterized in that, Applied to extended reality devices, the method includes: A three-dimensional display space is generated upon detection of wearing. Identify the current interaction scenario and determine at least one interaction module corresponding to the interaction scenario; The interactive module divides at least one interactive area in the three-dimensional display space; The interactive area responds to interactive commands and provides interactive result information.

2. The extended reality interaction method according to claim 1, characterized in that, Before responding to the interaction command through the interaction area and obtaining the interaction result information, the method further includes: Determine the module feature information of the interaction module corresponding to the interaction area; Configure first interactive attribute information for the interactive area based on the module feature information; The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: Based on the first interactive attribute information, the interactive area is controlled to respond to the interactive command, and the interactive result information is obtained.

3. The extended reality interaction method according to claim 1, characterized in that, The interaction instructions include a first user action and a second user action. Responding to the interaction instructions through the interaction area to obtain interaction result information includes: The target interaction area is determined from the interaction area based on the first user behavior; The interaction result information is obtained by responding to the second user's behavior through the target interaction area; The first user behavior or the second user behavior includes at least one of facial orientation behavior, gaze behavior, voice behavior, or gesture behavior; The interaction result information includes at least one of the following: second interaction attribute information obtained from the interaction attribute configuration, voice response result information for voice control commands, or gesture response result information for gesture control commands.

4. The extended reality interaction method according to claim 3, characterized in that, After determining the target interaction area from the interaction area based on the first user behavior, the method further includes: Display a visual identifier for the target interaction area, wherein the visual identifier is used to identify the currently selected target interaction area or to identify the third interaction attribute information corresponding to the target interaction area; Wherein, the second interactive attribute information or the third interactive attribute information includes at least one of the following: interactive method information, role interactive permission information, or behavior response logic information; The interaction method information includes at least one of voice interaction, eye-tracking interaction, and gesture interaction; the role interaction permission information includes at least one of read-only, hidden, or editable; and the behavior response logic information includes at least one of responding or not responding.

5. The extended reality interaction method according to claim 1, characterized in that, The interaction result information includes regional response result information and associated response result information; The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: The interactive area responds to the interactive command, and the response result information of the area is obtained. Determine the associated region corresponding to the interaction region, and adjust the associated region according to the region response result information to obtain the associated response result information.

6. The extended reality interaction method according to claim 5, characterized in that, The interactive area includes an editing area, the associated area includes a display area, the interactive instructions include editing instructions, the area response result information includes editing result information for the editing area, and the associated response result information includes display result information for the display area.

7. The extended reality interaction method according to claim 1, characterized in that, The method further includes: Display task phase indication information, which is used to indicate the current phase to be executed for the collaborative task; Display operation controls, which, when triggered, are used to adjust the task stage indication information. The operation controls include at least one of branch jump controls or process rollback controls. Each of the aforementioned interactive areas corresponds to a stage to be executed in a collaborative task, and the task stage indication information indicates the currently pending interactive area. The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: Based on the interactive area corresponding to the task stage indication information, the interactive instruction is responded to, and the interactive result information is obtained.

8. The extended reality interaction method according to claim 1, characterized in that, Before responding to the interaction command through the interaction area and obtaining the interaction result information, the method further includes: Determine user preference information or behavioral hotspot recording information; Based on the user preference information or the behavior hotspot recording information, the distribution of the interactive area in the three-dimensional display space is adjusted, and the distribution adjustment includes at least one of display position adjustment, area outline shrinkage, or interactive area hiding. The step of responding to interactive commands through the interactive area and obtaining interactive result information includes: The interaction result information is obtained by responding to the interaction command through the adjusted interaction area.

9. The extended reality interaction method according to claim 1, characterized in that, The step of determining at least one interactive module corresponding to the interactive scenario includes: The scene module mapping relationship determines at least one interactive module corresponding to the interactive scene, and the scene module mapping relationship includes the mapping relationship between a preset interactive scene and at least one preset interactive module; The three-dimensional display space is generated based on the actual spatial structure; The interaction module includes functional modules operable in the interaction scenario.

10. An extended reality interactive device, characterized in that, Applied to an extended reality device, the device includes: The generation module is used to generate a three-dimensional display space when wearing is detected; The identification module is used to identify the current interaction scenario and determine at least one interaction module corresponding to the interaction scenario; A partitioning module is used to partition at least one interactive area in the three-dimensional display space according to the interactive module; The response module is used to respond to interactive commands through the interactive area and obtain interactive result information.

11. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the extended reality interaction method as described in any one of claims 1-9.

12. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the extended reality interaction method as described in any one of claims 1-9.

Images