Application form control method, device and equipment in extended reality space
By detecting application opening events and responding to user operations, synchronously moving virtual subjects and independent entities, the flexibility problem when multiple applications are opened simultaneously in the extended real space is solved, the interaction and separation effect is achieved, and the user experience is improved.
Patent Information
- Application Number
- CN202310659536.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-06-05
AI Technical Summary
In the extended real-life space, it is difficult for the prior art to achieve the flexibility of multiple applications when they are turned on at the same time, and the interaction and separation effects between movable objects cannot be achieved, which affects the application's user experience.
By detecting application opening events, determine the location of the virtual subject and the attribute information of the independent entity, respond to user operations, synchronize the virtual subject, operation menu and environmental factors, realize the interaction and separation effect between independent entities, and gather content at startup exhibit.
It improves the flexibility of expanding applications in real-life space, and users can easily identify and interact with content in different applications, enhancing the user experience.
Smart Images

Figure CN117093070B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of extended reality technology, and in particular to a method, device and equipment for controlling application forms in an extended reality space. Background Art
[0002] On Extended Reality (XR) devices, traditional applications (i.e., applications) are in an immersive state (i.e., immersive mode), that is, an application can completely occupy the entire extended reality space. In this case, only one application can be run at the same time, that is, a single-task experience. This experience is usually used in application scenarios such as gaming or watching movies. For more general scenarios, users often need a multi-tasking experience similar to that of PCs, that is, users need to open multiple applications at the same time in the extended reality space.
[0003] Currently, one way to open multiple applications at the same time in an extended reality space is to treat each application as a window. In this window, the overlapping and separation effects of the different movable objects of the application when interacting can be achieved (for example, a puppy is biting a ball, and the puppy lets go of the ball). However, the movable objects are restricted to this window and cannot move outside the window in the entire extended reality space (for example, it is impossible to achieve a puppy chasing a moving ball in the entire extended reality space), which affects the flexibility of the application.
[0004] Another way to open multiple applications simultaneously in the extended reality space is to split an application into several windows, for example, each movable object in the application uses a separate window, so that each movable object can achieve the effect of moving in the entire extended reality space. However, since each movable object is independent of the other in the window, it is difficult to achieve the separation and overlap effect when different movable objects interact, which also affects the flexibility of the application. Summary of the invention
[0005] The purpose of the embodiments of this specification is to provide a method, device and equipment for controlling application forms in an extended reality space, so as to improve the flexibility of applications in the extended reality space.
[0006] To achieve the above objectives, on the one hand, an embodiment of the present specification provides a method for controlling an application form in an extended reality space, including:
[0007] Detect application launch events in the extended reality space;
[0008] Determine the detected occurrence direction of the application start event as the direction information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space;
[0009] Determine attribute information of each independent entity in the application; each of the independent entities represents a visual object in the application that can run independently outside the virtual subject;
[0010] The application is run and displayed according to the location information and the attribute information.
[0011] The application form control method in the extended reality space of the embodiment of this specification, after determining the attribute information of each independent entity in the application, further includes:
[0012] Determine the impact scope of environmental factors of other applications opened in the extended reality space;
[0013] Determine the virtual subject and / or independent entity in the present application that is affected by the environmental factors of the other opened applications according to the position information of the virtual subject of the present application, the attribute information of each independent entity and the influence range of the environmental factors;
[0014] Demonstrate the effects of virtual subjects and / or independent entities in this application being affected by environmental factors of the other applications that have been opened.
[0015] The method for controlling the application form in the extended reality space of the embodiment of the present specification, after determining the occurrence orientation of the detected application start event as the orientation information of the virtual subject of the application, further includes:
[0016] Determine the default position of the operation menu of the application according to the virtual subject of the application;
[0017] The operation menu is displayed at the default position.
[0018] The method for controlling the application form in the extended real space of the embodiment of this specification, after running and displaying the application according to the orientation information and the attribute information, further includes:
[0019] Respond to the user's mobile operation instructions for this application or according to the program settings of this application, move the virtual subject and / or independent entity of this application.
[0020] The application form control method in the extended reality space of the embodiment of this specification responds to the user's movement operation instruction of the application or moves the virtual subject and / or independent entity of the application according to the program setting of the application, including:
[0021] In response to the user's first movement operation instruction for the application, the virtual subject, operation menu, environmental factors and all independent entities of the application are synchronously moved;
[0022] In response to the user's second movement operation instruction for the application, the virtual subject, the operation menu, the environmental factors and the designated independent entity in the application are synchronously moved;
[0023] In response to the user's third movement operation instruction for the application, the virtual subject, operation menu and environmental factors of the application are synchronously moved; or,
[0024] In response to the user's fourth move operation instruction to this application or according to the program settings of this application, move the designated independent entity in this application.
[0025] The method for controlling the application form in the extended real space of the embodiment of this specification, after running and displaying the application according to the orientation information and the attribute information, further includes:
[0026] When the virtual subject and / or the independent entity is moved, the influence effects of the virtual subject and / or the independent entity in the present application affected by the environmental factors of other applications that have been opened are synchronously updated.
[0027] In the method for controlling application forms in an extended reality space in the embodiment of this specification, the application start event includes:
[0028] The user drags the application icon of the application from the desktop launcher and completes the placement in the extended reality space.
[0029] The method for controlling the application form in the extended reality space of the embodiment of the present specification determines the default position of the operation menu of the application according to the virtual subject of the application, including:
[0030] Based on the orientation of the virtual subject of this application, a target position at a specified distance from the orientation is used as the default orientation of the operation menu of this application.
[0031] The method for controlling the application form in the extended real space of the embodiment of this specification, after running and displaying the application according to the orientation information and the attribute information, further includes:
[0032] In response to the user's first mode switching operation instruction for the application, the application is switched to the immersive mode.
[0033] The method for controlling the application form in the extended reality space of the embodiment of this specification switches the application to the immersive mode, including:
[0034] The scene contents of other applications that have been opened in the extended reality space are gradually hidden, and the scene contents of the current application are gradually presented.
[0035] The method for controlling the application form in the extended reality space of the embodiment of this specification, after switching the application to the immersive mode, further includes:
[0036] In response to the user's second mode switching operation instruction for the application, the application is caused to exit the immersive mode.
[0037] The method for controlling the application form in the extended reality space of the embodiment of this specification enables the application to exit the immersive mode, including:
[0038] The scene content of the application is gradually hidden, and the scene content of other applications that have been opened in the extended reality space is gradually presented.
[0039] In the application form control method in the extended reality space of the embodiment of this specification, the operation menu includes a close key, a hide key and an adjust key;
[0040] After running and displaying the application according to the location information and the attribute information, the method further includes:
[0041] In response to the user's operation on the close button, first display a pre-close effect animation, and then close the application; or,
[0042] In response to the user's operation on the hide key, a pre-hide effect animation is first displayed, and then the application is hidden.
[0043] In the application form control method in the extended reality space of the embodiment of this specification, the present application and the other applications are extended reality space device applications, desktop applications, mobile applications or web applications.
[0044] The method for controlling the application form in the extended reality space in the embodiment of the present specification is that the virtual subject is not perceived by the user.
[0045] In the application form control method in the extended reality space of the embodiment of the present specification, the environmental factors include: environmental factors perceivable by the user in a virtual environment, a real environment, or a virtual-real mixed environment.
[0046] On the other hand, the embodiment of this specification also provides an application form control device in an extended reality space, including:
[0047] An event detection module, used to detect application start events in the extended reality space;
[0048] A subject determination module, configured to determine the occurrence orientation of the detected application start event as the orientation information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space;
[0049] An entity determination module, used to determine the attribute information of each independent entity in the application; each of the independent entities represents a visual object in the application that can run independently outside the virtual subject;
[0050] An application display module is used to run and display the application according to the position information and the attribute information.
[0051] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of the above method are executed.
[0052] On the other hand, an embodiment of the present specification further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer device, the instructions of the above method are executed.
[0053] On the other hand, an embodiment of the present specification further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor of a computer device, the computer program executes instructions of the above method.
[0054] It can be seen from the technical solutions provided in the above embodiments of this specification that, in the embodiments of this specification, since each independent entity represents a visual object in the application that can run independently outside the virtual body, an independent entity in an application can be separated from the corresponding virtual body and run independently in the entire extended reality space, and since the relationship between different independent entities is the relationship between visual objects in the application rather than the relationship between windows, overlapping and separation effects can be achieved when different independent entities interact, thereby improving the flexibility of applications in the extended reality space, and when the application is started, the content in the virtual body is displayed in a relatively continuous space in a clustered manner with each independent entity, so that the user can easily recognize which application the content distributed in different positions in the extended reality space as the application runs and / or user interacts belongs to. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:
[0056] Figure 1 A schematic diagram showing an application form control scenario in an extended reality space in some embodiments of this specification;
[0057] Figure 2 A flowchart of a method for controlling an application form in an extended reality space in some embodiments of this specification is shown;
[0058] Figure 3 A flowchart showing a method for controlling an application form in an extended reality space in some other embodiments of this specification is shown;
[0059] Figure 4 A flowchart showing a method for controlling an application form in an extended reality space in some other embodiments of this specification is shown;
[0060] Figure 5 A flowchart showing a method for controlling an application form in an extended reality space in some other embodiments of this specification is shown;
[0061] Figure 6 A flowchart showing a method for controlling an application form in an extended reality space in some other embodiments of this specification is shown;
[0062] Figure 7 A schematic diagram showing a virtual subject, an independent entity, and an operation menu applied in an exemplary embodiment of the present specification;
[0063] Figure 8 A schematic diagram showing opening two applications in an extended reality space in an exemplary embodiment of this specification is shown;
[0064] Fig. 9 Shows Figure 8 A schematic diagram showing an independent entity (butterfly) of the left application moving to the right application in the illustrated embodiment;
[0065] Fig.10 A structural block diagram of an application form control device in an extended reality space in some embodiments of this specification is shown;
[0066] Fig.11 The structure block diagram of the computer device in some embodiments of the present specification is shown.
[0067] [Description of Reference Numerals]
[0068] 10. Electronic equipment;
[0069] 20. Extended reality devices;
[0070] 101. Event detection module;
[0071] 102. Subject determination module;
[0072] 103. Entity determination module;
[0073] 104. Application display module;
[0074] 1102. Computer equipment;
[0075] 1104. Processor;
[0076] 1106. Memory;
[0077] 1108. Driving mechanism;
[0078] 1110, input / output interface;
[0079] 1112. Input device;
[0080] 1114. Output device;
[0081] 1116. Presentation equipment;
[0082] 1118. Graphical user interface;
[0083] 1120, network interface;
[0084] 1122. Communication link;
[0085] 1124. Communication bus. DETAILED DESCRIPTION
[0086] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.
[0087] Figure 1 The application scenario of application form control in the extended reality space in some embodiments of this specification is shown in the figure. In this application scenario, the extended reality device 20 can detect the application start event in the extended reality space; determine the occurrence direction of the detected application start event as the direction information of the virtual subject of this application; the virtual subject represents the belonging space of at least a part of this application in the extended reality space; determine the attribute information of each independent entity in this application; each of the independent entities represents a visual object in this application that can run independently outside the virtual subject; run and display the application according to the direction information and the attribute information. The flexibility of applications in the extended reality space can be improved through the embodiments of this specification. Among them, the application corresponding to the application start event can be an application installed or configured in the extended reality device 20, or it can be an application installed or configured in other electronic devices 10 and mapped to the application opened on the extended reality device 20 (such as desktop applications, mobile applications, and web applications, etc.).
[0088] It should be noted that, in the embodiments of the present specification, the application form control in the extended reality space is particularly suitable for a scenario where multiple applications are opened simultaneously in the extended reality space.
[0089] In the embodiments of this specification, the extended reality space is a three-dimensional scene. The scene can be a simulated environment of the real world (such as a scene in augmented reality), a semi-simulated and semi-fictional virtual environment (such as a scene in mixed reality), or a purely fictional virtual environment (such as a virtual environment in virtual reality). Any static or movable objects (such as user interfaces, etc.) can be displayed in the scene, and these objects are visible to the user.
[0090] In some embodiments, the electronic device 10 may be a mobile terminal (i.e., a smart phone), a desktop computer, a tablet computer, a laptop computer, or a smart TV, etc. The extended reality device 20 may be VR glasses, AR glasses, MR glasses, or other smart wearable devices based on XR technology.
[0091] The embodiment of this specification provides an application form control method in an extended reality space, which can be applied to the above-mentioned extended reality device side, referring to Figure 2 As shown, in some embodiments, the application form control method in the extended reality space may include the following steps:
[0092] Step 201: Detect an application start event in an extended reality space.
[0093] Step 202: Determine the detected occurrence direction of the application start event as the direction information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space.
[0094] Step 203: Determine the attribute information of each independent entity in the application; each of the independent entities represents a visual object in the application that can run independently outside the virtual subject.
[0095] Step 204: Run and display the application according to the location information and the attribute information.
[0096] In the embodiments of the present specification, since each independent entity represents a visual object in the application that can run independently outside the virtual body, an independent entity in an application can be separated from the corresponding virtual body and run independently in the entire extended reality space. Moreover, since the relationship between different independent entities is the relationship between visual objects in the application rather than the relationship between windows, overlapping and separation effects can be achieved when different independent entities interact, thereby improving the flexibility of applications in the extended reality space. When the application is started, the content in the virtual body is displayed in a relatively continuous space in a clustered manner with each independent entity, so that the user can easily recognize which application the content distributed in different locations in the extended reality space as the application runs and / or the user interacts belongs to.
[0097] In the embodiments of this specification, each application may include a virtual main body (Mainstay), one or more independent entities (Entity), an operation menu (optional) and one or more environmental factors (optional).
[0098] The virtual subject can be understood as an abstract concept similar to a container, which can represent the belonging space of at least part of the application in the extended reality space. It has no fixed shape or fixed range, and can be imperceptible to the user to reduce visual interference to the user. In the same application, independent entities, operation menus and environmental factors can be bound to a virtual subject. If the virtual subject is regarded as a root node, the independent entity, operation menu and environmental factors can be regarded as child nodes under the root node. Independent entities, operation menus and environmental factors may not be bound to virtual entities, in which case they can be understood as a parallel relationship. The virtual subject has position attributes and rotation attributes; the position attribute can represent the position of the virtual subject in the extended reality space, that is, the position of the application as a whole in the extended reality space. The rotation attribute can represent the viewing direction of the virtual subject in the extended reality space. The position attribute and the rotation attribute are collectively referred to as orientation information.
[0099] Each independent entity represents a visual object in the application that can run independently outside the virtual subject, that is, each independent entity can be separated from the virtual subject and run independently in the entire extended reality space, and can also return to the virtual subject. Which objects in an application can be independent entities can be defined by the application developer. Among them, being able to run independently outside the virtual subject means: being able to run independently outside the virtual subject based on user operations (for example, a dart thrown by a user can fly in a parabolic trajectory in the entire extended reality space), or being able to run independently outside the virtual subject based on the application's program settings (for example, a butterfly flies in the entire extended reality space based on the flight trajectory set by the application's program).
[0100] The operation menu is the operation menu of the application. In some embodiments, the operation menu may include but is not limited to a close key, a hide key, and an adjustment key (which may be appropriately increased or decreased according to needs during actual implementation, and the embodiments of this specification do not limit this); the close key can be used to close the corresponding application in the extended reality space; the hide key can be used to hide the corresponding application in the extended reality space; the adjustment key can be used to adjust the corresponding application in the extended reality space. Among them, the adjustment may include operations such as scaling, translating, and rotating the entire application or a specific object. Whether an application provides an operation menu or what kind of operation menu it provides can be defined by the application developer.
[0101] Environmental factors may be environmental factors perceivable by users in a virtual environment, a real environment, or a virtual-real mixed environment. For example, in an exemplary embodiment, environmental factors may include weather environmental factors (such as wind, rain, snow, light, etc.), gravity environmental factors, etc. Whether an application requires environmental factors and which environmental factors are required may be defined by the application developer.
[0102] For example, in Figure 7 In the exemplary embodiment shown, an application includes: a tree and three butterflies, a total of four independent entities, an operation menu, an environmental factor (wind), and a virtual subject (i.e., the three-dimensional space portion to which the independent entities, the operation menu, and the environmental factor belong). The three butterflies can land on the tree, thereby achieving an overlapping effect between the butterflies and the tree; the three butterflies can also fly away from the tree, thereby achieving a separation effect between the butterflies and the tree.
[0103] In the embodiments of this specification, detecting an application launch event in the extended reality space refers to: detecting whether an application is currently launched in the extended reality space. Among them, the application launch event can be pre-defined by the developer. For example, in one embodiment, the application launch event can be defined as: the user drags out the application icon of the application from the desktop launcher (Launcher) and completes the placement in the extended reality space (the specific placement position can be selected by the user). When an application launch event is detected, the occurrence orientation of the detected application launch event can be determined as the orientation information of the virtual subject of the application. Among them, the occurrence orientation is determined according to the application launch event; for example, taking the application launch event as the user dragging out the application icon of the application from the desktop launcher (Launcher) and completing the placement in the extended reality space, the occurrence orientation of the application launch event can be understood as the position and orientation of the user placing the application icon in the extended reality space. Therefore, the orientation information in the embodiments of this specification includes position information and direction information (i.e., rotation information).
[0104] In some embodiments, when an application is opened in the extended reality space, the application can actively send the attribute information of all visual objects (i.e., all independent entities) to the extended reality device for subsequent rendering and display, so that the extended reality device can determine each independent entity in the application. The attribute information may include, for example, the name, style, shape, color, position, rotation, scale, etc. of the independent entity. Figure 7 As an example of the exemplary embodiment shown, when Figure 7 When the application shown is opened in the extended reality space, the application can actively send the attribute information of the independent entities such as the tree and the three butterflies to the extended reality device.
[0105] Displaying visual objects including independent entities in the application in the virtual body is the display after the application is initialized, waiting for the client to interact with the application. Since the application is displayed in a relatively continuous space by gathering the content in the virtual body and each independent entity when the application is started, the user can easily recognize which application the content distributed in different locations in the extended reality space as the application runs and / or the user interacts belongs to.
[0106] In an embodiment of the present specification, running and displaying the application according to the orientation information and the attribute information may include: running and displaying an independent entity in the application according to the attribute information, and making the viewing direction of the independent entity consistent with the viewing direction in the orientation information.
[0107] The present specification provides another method for controlling the application form in an extended reality space, which can be applied to the above-mentioned extended reality device side, referring to Figure 3 As shown, in some embodiments, the application form control method in the extended reality space may include the following steps:
[0108] Step 301: Detect an application start event in an extended reality space.
[0109] Step 302: Determine the detected occurrence direction of the application start event as the direction information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space.
[0110] Step 303a: Determine the attribute information of each independent entity in the application; each of the independent entities represents a visual object in the application that can run independently outside the virtual subject.
[0111] Step 303b: Determine the impact range of environmental factors of other applications that have been opened in the extended reality space.
[0112] In some embodiments, when an application is opened in the extended reality space, the application can actively send all its environmental factors and their impact ranges to the extended reality device, so that the extended reality device can judge the impact of environmental factors between different applications based on this.
[0113] Step 304: Determine the virtual subjects and / or independent entities in the present application that are affected by the environmental factors of the other opened applications based on the position information of the virtual subject of the present application, the attribute information of each independent entity and the influence range of the environmental factors.
[0114] In the extended reality space, the influence range of an applied environmental factor can be a sphere with a fixed point as the center and a fixed length as the radius. When the environmental factor is bound to a virtual subject, the fixed point refers to the position of the virtual subject; when the environmental factor is bound to a specific independent entity, the fixed point is the position of the specific independent entity. The fixed length is the influence radius (or action radius) of the environmental factor.
[0115] For example, in an exemplary embodiment, taking the binding of environmental factors and virtual subjects as an example, application A and application B have been opened in the extended reality space, wherein the coordinates of the orientation information P1 of the virtual subject of application A are (0,0,0), and the coordinates of the orientation information P2 of the virtual subject of application B are (10,0,10), and the influence range of the lighting environment of application A is a sphere with position P1 (0,0,0) as the center and a radius of 15; the distance between position P1 (0,0,0) and position P2 (10,0,10) is 200, and obviously 200<15, so it can be confirmed that the independent entity in application B is affected by the lighting environment of application A. Among them, the coordinates, radius and distance in this exemplary embodiment are all in the same length unit.
[0116] Step 305: Run and display the application according to the position information and the attribute information; the display includes the effect of the virtual subject and / or independent entity in the application affected by the environmental factors of the other opened applications. In this way, the environmental factors of an application opened in the extended reality space affect the independent entity in another application opened in the extended reality space, thereby further improving the flexibility of opening applications in the extended reality space.
[0117] The present specification provides another method for controlling the application form in an extended reality space, which can be applied to the above-mentioned extended reality device side, referring to Figure 4 As shown, in some embodiments, the application form control method in the extended reality space may include the following steps:
[0118] Step 401: Detect an application start event in an extended reality space.
[0119] Step 402: Determine the detected occurrence direction of the application start event as the direction information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space.
[0120] Step 403a: Determine the attribute information of each independent entity in the application; each of the independent entities represents a visual object in the application that can run independently outside the virtual subject.
[0121] Step 403b: determine the default position of the operation menu of the application according to the virtual subject of the application.
[0122] In some embodiments, determining the default position of the operation menu of the application according to the virtual subject of the application may include: taking the position of the virtual subject of the application as a reference, and taking the target position at a specified distance from the position as the default position of the operation menu of the application. The target position at a specified distance from the position specifically refers to: a target position at a specified distance from a position in the position.
[0123] Step 404: Run and display the application according to the location information, the attribute information and the default location.
[0124] In an embodiment of the present specification, running and displaying the application according to the position information, the attribute information and the default position may include: running and displaying an independent entity in the application according to the position information and the attribute information, and displaying the operation menu according to the default position.
[0125] In this way, the operation menu of the application can be placed near the periphery of the virtual body of the application (for example, the operation menu can be Figure 7 As shown, it is placed directly below the virtual body of the application) to avoid affecting the user's line of sight and to help the user easily distinguish which application the operation menu is for, thereby improving the user's experience.
[0126] The present specification provides another method for controlling the application form in an extended reality space, which can be applied to the above-mentioned extended reality device side, referring to Figure 5 As shown, in some embodiments, the application form control method in the extended reality space may include the following steps:
[0127] Step 501: Detect an application start event in an extended reality space.
[0128] Step 502: Determine the detected occurrence direction of the application start event as the direction information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space.
[0129] Step 503a: Determine the attribute information of each independent entity in the application; each independent entity represents a visual object in the application that can run independently outside the virtual subject.
[0130] Step 503b: Determine the impact range of environmental factors of other applications that have been opened in the extended reality space.
[0131] Step 503c: determine the default position of the operation menu of the application according to the virtual subject of the application.
[0132] Step 504: Determine the virtual subjects and / or independent entities in the present application that are affected by the environmental factors of the other opened applications based on the position information of the virtual subject of the present application, the attribute information of each independent entity and the influence range of the environmental factors.
[0133] Step 505: Run and display the application according to the orientation information, the attribute information and the default orientation, wherein the display includes the influence effects of the virtual subject and / or independent entity in the application affected by the environmental factors of the other opened applications.
[0134] This not only further improves the flexibility of launching applications in the extended reality space, but also improves the user experience.
[0135] The present specification provides another method for controlling the application form in an extended reality space, which can be applied to the above-mentioned extended reality device side, referring to Figure 6 As shown, in some embodiments, the application form control method in the extended reality space may include the following steps:
[0136] Step 601: Detect an application start event in an extended reality space.
[0137] Step 602: Determine the detected occurrence direction of the application start event as the direction information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space.
[0138] Step 603a: Determine the attribute information of each independent entity in the application; each of the independent entities represents a visual object in the application that can run independently outside the virtual subject.
[0139] Step 603b: Determine the impact range of environmental factors of other applications opened in the extended reality space.
[0140] Step 603c: determine the default position of the operation menu of the application according to the virtual subject of the application.
[0141] Step 604: Determine the virtual subjects and / or independent entities in the present application that are affected by the environmental factors of the other opened applications based on the position information of the virtual subject of the present application, the attribute information of each independent entity and the influence range of the environmental factors.
[0142] Step 605: Run and display the application according to the orientation information, the attribute information and the default orientation, wherein the display includes the influence effects of the virtual subject and / or independent entity in the application affected by the environmental factors of the other opened applications.
[0143] Step 606: respond to the user's move operation instruction for the application or according to the program setting of the application, move the virtual subject and / or independent entity of the application.
[0144] The movement here can be translation, rotation, or scaling, etc. In this way, not only the flexibility of opening applications in the extended reality space is further improved, the user experience is improved, but also the user can control the virtual subject and / or independent entity.
[0145] In some embodiments, in response to a user's moving operation instruction for the application or according to the program setting of the application, moving the virtual subject and / or independent entity of the application may include:
[0146] In response to the user's first move operation instruction for the application, the virtual subject, operation menu, environmental factors and all independent entities in the application are synchronously moved. In this way, not only the integrity of the application is maintained, but also when there is a fixed relative distance constraint between the virtual subject and the independent entity, the functionality of the application can be prevented from being affected by operations outside the application logic.
[0147] In other embodiments, in response to a user's moving operation instruction for the application or according to the program setting of the application, moving the virtual subject and / or independent entity of the application may include:
[0148] In response to the user's second mobile operation instruction for the application, the virtual subject, operation menu, environmental factors and designated independent entities of the application are synchronously moved. The designated independent entity may be one or more independent entities. In this way, the personalized mobile needs of the user can be better met.
[0149] In other embodiments, in response to a user's moving operation instruction for the application or according to the program setting of the application, moving the virtual subject and / or independent entity of the application may include:
[0150] In response to the user's third mobile operation instruction for the application, the virtual subject, operation menu and environmental factors of the application are synchronously moved. In this way, the user's personalized mobile needs can be better met.
[0151] In other embodiments, in response to a user's moving operation instruction for the application or according to the program setting of the application, moving the virtual subject and / or independent entity of the application may include:
[0152] In response to the user's fourth move operation instruction for this application or according to the program settings of this application, the specified independent entity in this application is moved. In this way, the user's personalized movement needs can be better met. For example, in Figure 8 In the exemplary embodiment shown, there are two applications opened in the extended reality space (see Figure 8 ). Combined with Fig. 9 As shown, based on the program setting of the left application, the butterfly as an independent entity in the left application can break away from the virtual body of the left application and enter the right application.
[0153] In some embodiments, after running and displaying the application according to the position information and the attribute information, it also includes: when moving the virtual subject and / or the independent entity, synchronously updating the impact effects of the virtual subject and / or independent entity in the application that are affected by the environmental factors of other applications that have been opened.
[0154] It should be pointed out that when synchronously updating the influence effects of virtual subjects and / or independent entities in this application that are affected by the environmental factors of other applications that have been opened, if an independent entity is separated from the virtual subject, the influence effect of the independent entity can be determined based on whether the position of the independent entity is within the influence range of the environmental factors of the other applications that have been opened.
[0155] In some embodiments, after running and displaying the application according to the location information and the attribute information, the method further includes:
[0156] In response to the user's first mode switching operation instruction for the application, the application is switched to the immersive mode (i.e., from the current multitasking mode to the immersive mode). The application can display more content in the immersive mode, such as Figure 7 The application of the tree and butterfly shown in the figure only displays the tree and the butterfly, as well as a small part of the ground under the tree, by default. When entering the immersive mode, the entire background and larger spatial content (such as the sky, other ground and other content) can be displayed. In this way, the user experience when using a single application can be improved.
[0157] In addition, when an application is in immersive mode, it can support other applications to be displayed in it to meet some special scene requirements. For example, in video applications, the application defaults to the effect of hanging on the wall. You can switch to immersive mode to experience the environment of a private theater. At the same time, if you want to send messages to others during the movie, you can turn on the corresponding application to display in the video application for multi-tasking interaction; or open the system menu in the immersive mode application, and pop up message prompt boxes and other effects. Among them, multi-tasking mode means: multiple applications are opened in the extended reality space, and each application is in the foreground running state.
[0158] In other embodiments, after switching the application to the immersive mode, it may also include: responding to the user's second mode switching operation instruction to the application, so that the application exits the immersive mode; in this way, mutual switching between different modes can be achieved to meet the user's usage needs and further improve the user experience.
[0159] In some embodiments, the above-mentioned mode switching may be direct switching to improve switching efficiency.
[0160] In other embodiments, the above-mentioned mode switching may also be a progressive (or transitional) switching to present a better visual experience to the user. For example, taking switching the application to the immersive mode as an example, the scene content of other applications that have been opened in the extended reality space may be gradually hidden, and the scene content of the application may be gradually presented. For another example, taking the example of making the application exit the immersive mode (i.e., switching from the immersive mode to the multitasking mode), the scene content of the application may be gradually hidden, and the scene content of other applications that have been opened in the extended reality space may be gradually presented.
[0161] In some embodiments, after running and displaying the application according to the position information and the attribute information, it may also include: responding to the user's operation on the operation menu and executing the corresponding control logic. For example, in response to the user's operation on the close key in the operation menu, the control logic of closing the application is executed; in response to the user's operation on the hide key in the operation menu, the control logic of hiding the application is executed; in response to the user's operation on the adjust key in the operation menu, the control logic of adjusting the application is executed. When the application is closed or hidden, the corresponding environmental factor effects disappear. In addition, when the application is hidden, a location prompt can also be displayed at the position information of the virtual body of the application to prompt the user that the application is located and is in a hidden state. Among them, the location prompt can be customized as needed, for example, it can be the name of the application or a three-dimensional icon.
[0162] In some embodiments, after running and displaying the application according to the position information and the attribute information, it may also include: in response to the user's operation on the operation menu, first displaying the pre-operation effect animation, and then executing the corresponding control logic; in this way, the user experience can be enhanced. For example, in response to the user's operation on the close key, first displaying the pre-close effect animation, and then closing the application; for another example, in response to the user's operation on the hide key, first displaying the pre-hide effect animation, and then hiding the application.
[0163] For example, in Figure 7 In the exemplary embodiment shown, in response to the user's closing operation, an animation effect such as a butterfly flying back to the tree and then the entire tree being closed may be displayed.
[0164] In some embodiments, the user can operate the application (including virtual subjects, independent entities, and operation menus, etc.) through any suitable input device, and the embodiments of this specification are not limited to this. In some embodiments, the user can perform contact operations through a mouse, keyboard, joystick (such as a thumb rocker, etc.), touch screen, etc. In other embodiments, the user can also perform non-contact operations through a voice input device (such as a microphone, etc.) and a camera.
[0165] For example, in an exemplary embodiment, the application adjustment operation is performed by dragging the bar + joystick. In this scenario, the distance of the application relative to the user can be adjusted by pushing the joystick farther or closer; the virtual subject, independent entity, etc. in the application can be translated by holding the drag bar and moving it; the application can be rotated by pressing the drag bar and pushing the joystick left or right at the same time, etc.
[0166] For example, in another exemplary embodiment, the application adjustment operation is performed by using the joystick + the adjustment key of the operation menu. In this scenario, the adjustment key of the operation menu can be pressed first, and then the distance of the application relative to the user can be adjusted by pushing the joystick farther or closer; the application can be rotated by pushing the joystick left or right.
[0167] Although the process flows described above include multiple operations that occur in a particular order, it should be clearly understood that the processes may include more or fewer operations, which may be performed sequentially or in parallel (eg, using parallel processors or a multi-threaded environment).
[0168] Corresponding to the above-mentioned application form control method in the extended reality space, the embodiment of this specification also provides an application form control device in the extended reality space, which can be configured on the above-mentioned extended reality device, with reference to Fig.10 As shown, in some embodiments, the application form control device in the extended reality space may include:
[0169] An event detection module 101 is used to detect an application start event in an extended reality space;
[0170] The subject determination module 102 is used to determine the occurrence direction of the detected application start event as the direction information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space;
[0171] The entity determination module 103 is used to determine the attribute information of each independent entity in the application; each independent entity represents a visual object in the application that can run independently outside the virtual subject;
[0172] The application display module 104 is used to run and display the application according to the location information and the attribute information.
[0173] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0174] It should be noted that in the embodiments of this specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user and fully authorized by all parties.
[0175] The embodiments of this specification also provide a computer device. Fig.11As shown, in some embodiments of this specification, the computer device 1102 may include one or more processors 1104, such as one or more central processing units (CPUs) or graphics processing units (GPUs), and each processing unit may implement one or more hardware threads. The computer device 1102 may also include any memory 1106, which is used to store any kind of information such as code, settings, data, etc. In a specific embodiment, the computer program on the memory 1106 and can be run on the processor 1104, when the computer program is run by the processor 1104, the instructions of the application form control method in the extended reality space described in any of the above embodiments may be executed. Non-limiting, for example, the memory 1106 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 1102. In one embodiment, when the processor 1104 executes the associated instructions stored in any memory or combination of memories, the computer device 1102 can perform any operation of the associated instructions. The computer device 1102 also includes one or more drive mechanisms 1108 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.
[0176] The computer device 1102 may also include an input / output interface 1110 (I / O) for receiving various inputs (via input devices 1112) and for providing various outputs (via output devices 1114). A specific output mechanism may include a presentation device 1116 and an associated graphical user interface 1118 (GUI). In other embodiments, the input / output interface 1110 (I / O), input device 1112, and output device 1114 may not be included, and the computer device 1102 may be used as a computer device in a network. The computer device 1102 may also include one or more network interfaces 1120 for exchanging data with other devices via one or more communication links 1122. One or more communication buses 1124 couple the components described above together.
[0177] The communication link 1122 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1122 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.
[0178] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), computer-readable storage media, and computer program products of some embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processor to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processor generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0179] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processor to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0180] These computer program instructions can also be loaded onto a computer or other programmable data processor so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0181] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0182] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0183] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computer device. As defined in this specification, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0184] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0185] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processors connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0186] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0187] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0188] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0189] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A method for controlling application form in an extended reality space, characterized in that: include: Detect application launch events in the extended reality space; Determine the location of the detected application start event as the location information of the virtual subject of the application; The virtual subject represents a home space of at least a part of the application in the extended reality space; Determine the attribute information of each independent entity in the application; each independent entity represents a visual object in the application that can independently operate in the entire extended reality space outside the virtual subject; the independent operation means that it independently operates outside the virtual subject according to user operation or application program settings to achieve overlapping and separation effects when different independent entities interact; The application is run and displayed according to the location information and the attribute information.
2. The method for controlling application forms in an extended reality space according to claim 1, characterized in that: After determining the attribute information of each independent entity in this application, it also includes: Determine the impact scope of environmental factors of other applications opened in the extended reality space; Determine the virtual subject and / or independent entity in the present application that is affected by the environmental factors of the other opened applications according to the position information of the virtual subject of the present application, the attribute information of each independent entity and the influence range of the environmental factors; Demonstrate the effects of virtual subjects and / or independent entities in this application being affected by environmental factors of the other applications that have been opened.
3. The method for controlling application forms in an extended reality space according to claim 1, characterized in that: After determining the detected occurrence position of the application start event as the position information of the virtual subject of the application, the method further includes: Determine the default position of the operation menu of the application according to the virtual subject of the application; The operation menu is displayed at the default position.
4. The method for controlling application forms in an extended reality space according to claim 1, wherein: After running and displaying the application according to the location information and the attribute information, the method further includes: Respond to the user's mobile operation instructions for this application or according to the program settings of this application, move the virtual subject and / or independent entity of this application.
5. The method for controlling application forms in an extended reality space according to claim 4, characterized in that: Respond to the user's mobile operation instructions for this application or according to the program settings of this application, move the virtual subject and / or independent entity of this application, including: In response to the user's first movement operation instruction for the application, the virtual subject, operation menu, environmental factors and all independent entities of the application are synchronously moved; In response to the user's second movement operation instruction for the application, the virtual subject, the operation menu, the environmental factors and the designated independent entity in the application are synchronously moved; In response to the user's third movement operation instruction for the application, the virtual subject, operation menu and environmental factors of the application are synchronously moved; or, In response to the user's fourth move operation instruction to this application or according to the program settings of this application, move the designated independent entity in this application.
6. The method for controlling application forms in an extended reality space according to claim 4, characterized in that: After running and displaying the application according to the location information and the attribute information, the method further includes: When the virtual subject and / or the independent entity is moved, the influence effects of the virtual subject and / or the independent entity in the present application affected by the environmental factors of other applications that have been opened are synchronously updated.
7. The method for controlling application forms in an extended reality space according to claim 1, characterized in that: The application start event includes: The user drags the application icon of the application from the desktop launcher and completes the placement in the extended reality space.
8. The method for controlling application forms in an extended reality space according to claim 3, characterized in that: Determining the default position of the operation menu of the application according to the virtual subject of the application, including: Based on the position of the virtual subject of this application, a target position at a specified distance from the position of the virtual subject is used as a default position of the operation menu of this application.
9. The method for controlling application forms in an extended reality space according to claim 1, wherein: After running and displaying the application according to the location information and the attribute information, the method further includes: In response to the user's first mode switching operation instruction for the application, the application is switched to the immersive mode.
10. The method for controlling application forms in an extended reality space according to claim 9, characterized in that: Switch the app to immersive mode, including: The scene contents of other applications that have been opened in the extended reality space are gradually hidden, and the scene contents of the current application are gradually presented.
11. The method for controlling application forms in an extended reality space according to claim 9, characterized in that: After switching the app to immersive mode, it also includes: In response to the user's second mode switching operation instruction for the application, the application is caused to exit the immersive mode.
12. The method for controlling application forms in an extended reality space according to claim 11, characterized in that: Make this application exit the immersive mode, including: The scene content of the application is gradually hidden, and the scene content of other applications that have been opened in the extended reality space is gradually presented.
13. The method for controlling application forms in an extended reality space according to claim 3, characterized in that: The operation menu includes a close key, a hide key and an adjust key; After running and displaying the application according to the location information and the attribute information, the method further includes: In response to the user's operation on the close button, first display a pre-close effect animation, and then close the application; or, In response to the user's operation on the hide key, a pre-hide effect animation is first displayed, and then the application is hidden.
14. The method for controlling application forms in an extended reality space according to claim 2, wherein: The present application and other applications are extended reality space device applications, desktop applications, mobile applications or web applications.
15. The method for controlling application forms in an extended reality space according to claim 1, characterized in that: The virtual subject is not perceived by the user.
16. The method for controlling application forms in an extended reality space according to claim 2, wherein: The environmental factors include: environmental factors perceivable by the user in a virtual environment, a real environment or a virtual-real mixed environment.
17. An application form control device in an extended reality space, characterized in that: include: An event detection module, used to detect application start events in the extended reality space; A subject determination module, configured to determine the occurrence orientation of the detected application start event as the orientation information of the virtual subject of the application; the virtual subject represents the belonging space of at least a part of the application in the extended reality space; An entity determination module, used to determine the attribute information of each independent entity in the application; each independent entity represents a visual object in the application that can independently operate in the entire extended reality space outside the virtual subject; the independent operation means that it independently operates outside the virtual subject according to user operations or application program settings to achieve overlapping and separation effects when different independent entities interact; An application display module is used to run and display the application according to the position information and the attribute information.
18. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, the computer program executes the instructions of the method according to any one of claims 1 to 16.
19. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes instructions of the method according to any one of claims 1 to 16.
20. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor of a computer device, the computer program executes instructions of the method according to any one of claims 1 to 16.
