Display processing method, device and equipment of two-dimensional component and medium

Abstract

Embodiments of the present disclosure relate to a display processing method and device of a two-dimensional component, an apparatus and a medium, wherein the method comprises: obtaining a display area reference value required for displaying at least one two-dimensional component; determining a target display space in an extended reality space according to the display area reference value, wherein the target display space contains at least one display plane area with a display area greater than the display area reference value; and performing display processing on the at least one two-dimensional component in the target display space. In the embodiments of the present disclosure, the display space of the two-dimensional component is determined according to the display area reference value required by the two-dimensional component, which realizes intelligent determination of the display space, improves the flexibility of the display of the two-dimensional component, and fully utilizes the depth characteristics in the extended reality space.

Description

Technical Field

[0001] This disclosure relates to the field of extended reality technology, and in particular to a display processing method, apparatus, device and medium for two-dimensional components. Background Technology

[0002] Extended reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. It's a collective term for various technologies such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). By integrating the visual interaction technologies of these three technologies, it provides users with a seamless "immersive" experience between the virtual and real worlds. Enhancing the sense of intelligence in extended reality scenarios has become a mainstream approach.

[0003] In related technologies, when displaying 2D interface components in an extended reality scene to quickly obtain information such as text and images, relevant technicians plan the display space for the 2D interface components in the extended reality space and then display the 2D interface components in the planned display space. This method of manually setting the display space is not very intelligent and does not make full use of the depth characteristics in the extended display space. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a display processing method, apparatus, device, and medium for two-dimensional components. The display space of the two-dimensional component is determined based on a reference value of the required display area of ​​the two-dimensional component, thereby realizing the intelligent determination of the display space, improving the flexibility of the display of the two-dimensional component, and making full use of the depth characteristics in the extended reality space.

[0005] This disclosure provides a display processing method for a two-dimensional component, the method comprising the following steps: obtaining a display area reference value required to display at least one two-dimensional component; determining a target display space in an extended reality space based on the display area reference value, wherein the target display space includes at least one display plane region with a display area greater than the display area reference value; and performing display processing on the at least one two-dimensional component in the target display space.

[0006] This disclosure also provides a display processing apparatus for a two-dimensional component, the apparatus comprising: an acquisition module for acquiring a display area reference value required to display at least one two-dimensional component; a determination module for determining a target display space in an extended reality space based on the display area reference value, wherein the target display space includes at least one display plane region with a display area greater than the display area reference value; and a display module for performing display processing on the at least one two-dimensional component in the target display space.

[0007] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the display processing method for two-dimensional components as provided in this disclosure.

[0008] This disclosure also provides a computer-readable storage medium storing a computer program for executing a display processing method for a two-dimensional component as provided in this disclosure.

[0009] The technical solution provided in this disclosure has the following advantages compared with the prior art: The display processing scheme for two-dimensional components provided in this disclosure obtains a reference value for the display area required to display at least one two-dimensional component, determines a target display space in extended reality space based on the reference value, wherein the target display space includes at least one display plane region with a display area larger than the reference value, and then performs display processing on at least one two-dimensional component in the target display space. Therefore, by determining the display space of the two-dimensional component based on the required display area reference value, intelligent determination of the display space is achieved, improving the flexibility of two-dimensional component display and fully utilizing the depth characteristics of extended reality space. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a schematic diagram of a viewing scene in a virtual reality space provided by an embodiment of the present disclosure; Figure 2 A schematic flowchart illustrating a display processing method for a two-dimensional component provided in an embodiment of this disclosure; Figure 3 A schematic diagram illustrating a display processing scenario for a two-dimensional component provided in an embodiment of this disclosure; Figure 4 A schematic diagram illustrating a display processing scenario for another two-dimensional component provided in an embodiment of this disclosure; Figure 5 A schematic diagram illustrating a display processing scenario for another two-dimensional component provided in an embodiment of this disclosure; Figure 6 A schematic diagram illustrating a display processing scenario for another two-dimensional component provided in an embodiment of this disclosure; Figure 7 A schematic diagram illustrating a display processing scenario for another two-dimensional component provided in an embodiment of this disclosure; Figure 8A A schematic diagram illustrating a display processing scenario for another two-dimensional component provided in an embodiment of this disclosure; Figure 8B A schematic diagram illustrating a display processing scenario for another two-dimensional component provided in an embodiment of this disclosure; Figure 8C A schematic diagram illustrating a display processing scenario for another two-dimensional component provided in an embodiment of this disclosure; Figure 9 A schematic diagram of the structure of a display processing device for a two-dimensional component provided in an embodiment of this disclosure; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0015] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0016] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0018] The following explanations are provided for some of the technical concepts or terms used in this article: AR: An AR scene refers to a simulated scene in which at least one virtual object is superimposed on a physical scene or its representation. For example, an electronic system may have an opaque display and at least one imaging sensor for capturing images or videos of the physical scene, which are representations of the physical scene. The system combines the images or videos with virtual objects and displays this combination on the opaque display. Individuals use the system to indirectly view the physical scene via images or videos of the physical scene and observe the virtual objects superimposed on the physical scene. When the system uses one or more image sensors to capture images of the physical scene and uses those images to present the AR scene on an opaque display, the displayed images are referred to as video pass-through. Alternatively, the electronic system for displaying the AR scene may have a transparent or semi-transparent display through which an individual can directly view the physical scene. The system may display virtual objects on the transparent or semi-transparent display, allowing an individual to observe the virtual objects superimposed on the physical scene using the system. As another example, the system may include a projection system that projects virtual objects onto the physical scene. Virtual objects can be projected, for example, onto a physical surface or as holograms, allowing individuals to observe virtual objects superimposed on a physical setting using the system. Specifically, a technique involves calculating the camera's pose parameters in the real world (or 3D world, the real world) in real time during image acquisition, and then adding virtual objects to the captured images based on these parameters. Virtual objects include, but are not limited to, 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive experiences.

[0019] MR (Mixed Reality): MR enhances the realism of the user experience by presenting extended information about the real-world scene, establishing an interactive feedback loop between the real world, the virtual world, and the user. For example, computer-created sensory input (e.g., virtual objects) is integrated with sensory input or its representation from a physical setting within a simulated scene. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical setting, enabling virtual objects to interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings. VR: VR is a technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, which is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, smell perception, etc.) to achieve a fusion of virtual environment, interactive three-dimensional dynamic visual scenes and simulation of physical behavior, so that users can immerse themselves in the simulated virtual reality environment, and realize applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, manufacturing assistance, maintenance and repair.

[0020] Virtual reality devices, the terminals that enable virtual reality effects in VR, can typically be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these, and they can be further miniaturized or enlarged as needed.

[0021] The virtual reality devices described in this disclosure can include, but are not limited to, the following types: PC-based virtual reality (PCVR) devices utilize a PC for calculations and data output related to virtual reality functions. External PC-based virtual reality devices then use the data output from the PC to achieve the virtual reality effect.

[0022] Mobile virtual reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connections with the mobile terminal, the mobile terminal performs calculations related to virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through a mobile terminal's app.

[0023] All-in-one virtual reality devices have a processor for performing virtual functions, thus possessing independent virtual reality input and output capabilities. They do not require connection to a PC or mobile terminal, offering a high degree of freedom of use.

[0024] Taking VR scenarios as an example, such as Figure 1 As shown, HMDs are relatively lightweight, ergonomically comfortable, and provide high-resolution content with low latency. Virtual reality devices incorporate posture detection sensors (such as a nine-axis sensor) to detect changes in the device's posture in real time. If a user is wearing a virtual reality device, the processor transmits the user's real-time head posture to the processor when the user's head posture changes. This allows the processor to calculate the user's gaze point in the virtual environment. Based on the gaze point, the processor calculates the image within the user's field of view (i.e., the virtual field of view) in the 3D model of the virtual environment and displays it on the screen, creating an immersive experience as if the user were in a real-world environment.

[0025] In this embodiment, when a user wears an HMD device and opens a pre-defined application, such as a live video streaming application, the HMD device runs a corresponding virtual scene. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual scene, or a purely fictional virtual scene. The virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this embodiment does not limit the dimension of the virtual scene. For example, the virtual scene may include people, sky, land, ocean, etc. The land may include environmental elements such as deserts and cities. The user can control the movement of objects within the virtual scene and can also interact with controls, models, displayed content, people, and other objects in the virtual scene using a gamepad, bare-hand gestures, etc.

[0026] To address the aforementioned issues, this disclosure provides a method for displaying two-dimensional components, which will be described below with reference to specific embodiments.

[0027] Figure 2 This is a flowchart illustrating a display processing method for a two-dimensional component according to an embodiment of this disclosure. The method can be executed by a display processing device for the two-dimensional component, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 2 As shown, the method includes: Step 201: Obtain a reference value for the display area required to display at least one two-dimensional component.

[0028] In this embodiment, the two-dimensional component can be any 2D UI element displayed in extended reality space. For example, the two-dimensional component can include a bullet screen component, an image component, etc.

[0029] In this embodiment, in order to ensure that the subsequently determined display space provides sufficient real space and to prevent the two-dimensional components from exceeding the corresponding display space and affecting the display of other elements in the extended space, a reference value for the display area required by at least one two-dimensional component is obtained.

[0030] The required display area reference value for at least one two-dimensional component can be understood as the maximum display area required when at least one two-dimensional component is arranged and displayed in its possible display modes. In some possible embodiments, the maximum tiled display area for displaying at least one two-dimensional component is obtained, and the maximum tiled display area is determined as the display area reference value.

[0031] Alternatively, the required display area reference value for at least one two-dimensional component can be understood as the display area required for at least one two-dimensional component to be displayed in a preset manner, ensuring that at least one two-dimensional component is displayed normally in a preset manner.

[0032] In different application scenarios, the methods for determining the maximum tiled display area of ​​at least one two-dimensional component vary. For example, in some application scenarios, each two-dimensional component has the same size, which is 'a'. A square of size 'a' can display a maximum of 6 two-dimensional components simultaneously, and when the two-dimensional components are tiled, a maximum of 3 components can be displayed per row. Therefore, if... Figure 3 As shown, the reference value for the display area required for at least one two-dimensional component is 3a. 2a.

[0033] It should be emphasized that the two-dimensional components corresponding to the reference display area can be connected or not connected to each other.

[0034] Step 202: Determine the target display space in the extended reality space based on the display area reference value. The target display space includes at least one display plane area with a display area greater than the display area reference value. In the actual display process, the two-dimensional component can randomly determine the display position in the target display space. Therefore, when there is a display situation corresponding to the display area reference value, the corresponding two-dimensional component can be displayed on the display plane area.

[0035] In one embodiment of this disclosure, a target display space is determined in the extended reality space based on a display area reference value. To ensure that the target display space can meet display requirements, the target display space includes at least one display plane region with a display area greater than the display area reference value. The display plane region can be any plane region located inside the target display space.

[0036] For example, continue to refer to Figure 3If the target display space A is determined in the extended reality space based on the display area reference value, and includes the display plane area s, and the area of ​​s is greater than 3a... 2a (In this embodiment, to take into account the subsequent "safety zone", in the "safety zone" embodiment, the tiling area is related to the size corresponding to the "safety zone").

[0037] In this embodiment, a candidate display space for displaying two-dimensional components can be determined in the extended reality space. The candidate display space can be any free space in the extended reality space, or it can be a space pre-defined in the extended reality space for displaying two-dimensional components.

[0038] Then, a target display space is determined from the candidate display space based on the display area reference value. The target display space can be of any shape and can be located at any position within the candidate display space. In some possible embodiments, preview images of the shapes and positions corresponding to multiple target display spaces can be displayed, and the final shape and position of the target display space can be selected based on the user's confirmation operation.

[0039] Step 203: Display at least one two-dimensional component in the target display space.

[0040] In one embodiment of this disclosure, after determining the target display space, at least one two-dimensional component is displayed in the target display space. Thus, on the one hand, displaying the two-dimensional component in the target display space avoids the two-dimensional component from affecting other elements in the extended reality space during the display process; on the other hand, the two-dimensional component can be randomly displayed at any position in the target display space, utilizing the depth information of the extended reality space to achieve display flexibility and interest; furthermore, the target display space is delineated based on the reference value of the display area required by at least one two-dimensional component, improving the intelligence level of the target display space determination.

[0041] In summary, the display processing method for two-dimensional components according to the embodiments of this disclosure obtains a reference value for the display area required to display at least one two-dimensional component, determines a target display space in extended reality space based on the reference value for the display area, wherein the target display space includes at least one display plane region with a display area larger than the reference value for the display area, and then performs display processing on at least one two-dimensional component in the target display space. Therefore, by determining the display space of the two-dimensional component based on the reference value for the required display area, intelligent determination of the display space is achieved, improving the flexibility of two-dimensional component display and fully utilizing the depth characteristics of extended reality space.

[0042] It should be noted that the interaction methods for displaying at least one two-dimensional component in the target display space differ in different application scenarios, and different display methods enhance the fun of displaying at least one two-dimensional component.

[0043] The following describes, with reference to specific embodiments, how to display at least one two-dimensional component in the target display space.

[0044] In one embodiment of this disclosure, displaying at least one two-dimensional component in a target display space includes: determining a reference display position for each two-dimensional component in the target display space according to a preset random algorithm, and displaying the corresponding two-dimensional component at the corresponding reference display position. That is, in this embodiment, as long as the two-dimensional component is confined to the target display space for display, it can be displayed at any position within the target display space, thereby improving the display flexibility of the two-dimensional component.

[0045] In actual display, when multiple 2D components are included, to avoid "clipping" (i.e., overlapping display positions of different 2D components), a component bounding box can be generated to surround each 2D component. The component bounding box encloses the corresponding 2D component, such as... Figure 4 The bounding box of the two-dimensional component P1 shown surrounds the two-dimensional component P1. The bounding box of each two-dimensional component can be regarded as the "safe area" of the corresponding two-dimensional component. If the "safe areas" between two-dimensional components do not touch, the "clipping" phenomenon mentioned above can be avoided.

[0046] The component bounding box should not be too large. Generally, the size of the component bounding box can be set to be slightly larger than the size of the two-dimensional component, or the height of the component bounding box can be set. The width of the component bounding box can be determined by multiplying the width of the corresponding two-dimensional component by the preset magnification ratio, and the height of the component bounding box can be determined by multiplying the height of the corresponding two-dimensional component by the preset magnification ratio, etc. For example, when the size of the two-dimensional component is 0.11m × 0.11m, then a component bounding box of 0.12 × 0.12 × 0.02 m³ should be set for the corresponding component.

[0047] In this embodiment, during the initial display phase, it is determined whether each two-dimensional component meets a preset collision condition with other two-dimensional components based on the bounding box position of the component bounding box. In some optional embodiments, it is possible to calculate whether the minimum distance between different component bounding boxes is less than a small preset distance value (e.g., it can be 0). If it is less than the preset distance value, it is determined that the corresponding component bounding box meets the preset collision condition. In some possible embodiments, it is determined whether different component bounding boxes have overlapping parts based on the bounding box position of the component bounding box. If there are overlapping parts, it is determined that the preset collision condition is met.

[0048] Furthermore, if there are reference display positions for two-dimensional components that satisfy the preset collision conditions, then the reference display positions of these components are updated so that each two-dimensional component does not satisfy the preset collision conditions with other two-dimensional components. When updating the reference display positions of components satisfying the preset collision conditions, a random algorithm can be used to continue determining reference display positions for components satisfying the preset collision conditions until they are satisfied; alternatively, the reference display positions of the two-dimensional components can be determined at any free location in the target display space.

[0049] Furthermore, when it is determined that multiple two-dimensional components do not meet the preset collision conditions, the corresponding two-dimensional component is displayed at the corresponding reference display position. When displaying the two-dimensional component, if there are no other dragging instructions, the two-dimensional component can be fixedly displayed at the corresponding reference display position.

[0050] In one embodiment of this disclosure, after displaying a two-dimensional component, in order to further enhance the flexibility and fun of displaying the two-dimensional component, the two-dimensional component can be dragged, and during the dragging process, the relevant two-dimensional component can be displayed as a dragging collision effect.

[0051] In this embodiment, a drag operation on the first two-dimensional component among multiple two-dimensional components can be obtained. In some possible embodiments, the drag operation can be to capture an image of the user's hand, recognize the user's hand posture based on the hand image, and when the hand posture is a preset drag gesture, detect the displacement of the user's hand and take the displacement process of the hand as the drag operation. In some possible embodiments, when a user uses a virtual control device (such as a gamepad), an "operation direction indicator model" can be displayed in the extended reality space. The "operation direction indicator model" is used to indicate the operation direction of the virtual control device. When the user triggers the "drag confirmation" control corresponding to the virtual control device, the movement operation of the "operation direction indicator model" is taken as a drag operation. When the "drag confirmation" control is triggered, the first two-dimensional component under the indicated direction of the "operation direction indicator model" moves along with the movement of the operation direction indicator model.

[0052] Among them, such as Figure 5 As shown, the "operation direction indicator model" can be a ray trajectory model. When the endpoint of the ray trajectory model is located on the two-dimensional component P1, if the "drag confirmation" control is triggered, P1 is considered to be the first two-dimensional component, and P1 is controlled to move following the ray trajectory model. That is, in this embodiment, the first two-dimensional component is moved according to the drag operation.

[0053] Furthermore, during the movement, based on the corresponding bounding box position, it is detected whether there is a second two-dimensional component within the bounding box of other two-dimensional components that meets the preset collision conditions with the first two-dimensional component; that is, it is detected whether there are other two-dimensional components that collide with the first two-dimensional component. If a second two-dimensional component exists, collision motion display processing is performed on the second two-dimensional component. (Continue referring to...) Figure 5 If the "safe zone" of P1 and P2 collides, then P2 becomes the second two-dimensional component, controlling P2 to perform collision movement, in order to further enhance the realism of P1's dragging movement, making full use of the depth information characteristics of the extended real space for display effects.

[0054] It should be noted that the methods for handling collision motion display of second-dimensional components differ in different application scenarios. Examples are illustrated below: In some optional implementations, the movement trajectory of the collision motion can be predefined, and the second two-dimensional component can be controlled to move from the current position as the starting point of the movement along the movement trajectory to achieve the "collision" motion effect.

[0055] In some optional implementations, to further enhance the realism of the "collision," a first movement parameter of the first two-dimensional component is used to satisfy preset collision conditions. This first movement parameter includes, but is not limited to, movement speed and direction. A second movement parameter of the second two-dimensional component is then determined based on the first movement parameter. This second movement parameter includes, but is not limited to, movement speed, movement acceleration, and movement direction. The second movement parameter can be calculated using a pre-learned deep learning algorithm based on the input first movement parameter. The second movement parameter is then used to control the collision movement of the second two-dimensional component in the target display space. Since the second movement parameter is calculated based on the first movement parameter, it more realistically simulates the "collision" motion in the real world. The achieved motion effect can be further referenced. Figure 5 After colliding with P1, P2 begins to move in the corresponding direction to simulate the "collision" motion in the real world.

[0056] Of course, during the collision movement of the second two-dimensional component in the target display space, it may also collide with other two-dimensional components, and other two-dimensional components may also collide with other two-dimensional components. Therefore, in the case of collision, in one embodiment of this disclosure, other two-dimensional components are also controlled to make chain collision movements until there are no chain collisions. The collision movement mode can refer to the collision movement mode of the second two-dimensional component, which will not be listed one by one here.

[0057] In one embodiment of this disclosure, considering that, as mentioned above, in order not to affect the display of other elements in the extended reality space, the two-dimensional components are restricted to display in the target display space, therefore, during the drag collision process mentioned in the above embodiment, some two-dimensional components may collide with the spatial boundary of the target display space. In order to further enhance the fun of the display, the two-dimensional components that collide with the spatial boundary of the target display space can also be subjected to "bounce" motion processing.

[0058] In this embodiment, based on the bounding box position of at least one two-dimensional component and the spatial boundary position of the target display space, it is detected whether there is a third two-dimensional component that satisfies the preset edge collision condition with the spatial boundary of the target display space. In some possible embodiments, if the minimum distance between the bounding box position and the spatial boundary position is less than or equal to 0, it is considered that the preset edge collision condition is satisfied.

[0059] In this embodiment, when a third two-dimensional component exists, a third movement parameter of the third two-dimensional component is determined when the preset edge collision condition is met. The third movement parameter includes, but is not limited to, movement speed and movement direction. A fourth movement parameter is determined based on the third movement parameter. The fourth movement parameter includes, but is not limited to, movement speed, movement acceleration and movement direction. The movement direction of the fourth movement parameter may be opposite to the movement direction of the third movement parameter. In some optional implementations, the corresponding fourth movement parameter may be determined based on the input third movement parameter using a pre-set deep learning algorithm.

[0060] After determining the fourth movement parameter, the third two-dimensional component is controlled to bounce and move in the target display space according to the fourth movement parameter to achieve the corresponding "bounce" motion effect. During the "bounce" motion, it can also collide with other two-dimensional components, and other two-dimensional components can also perform collision effects, etc.

[0061] Of course, in some optional embodiments, default motion parameters corresponding to the rebound motion of the spatial boundary can also be set, and the third two-dimensional component can be controlled to bounce and move in the target display space based on the default motion parameters.

[0062] In some possible embodiments, in order to further utilize the depth information in the extended reality control, the display direction of the two-dimensional component can also be moved according to the user's line of sight, that is, the display vector of the two-dimensional component is controlled to follow the user's line of sight to achieve a "line of sight following" display effect.

[0063] In this embodiment, in response to a change in the user's gaze direction, gaze change information is determined, including gaze movement angle and movement direction. Based on the gaze change information, the display direction of at least one component is rotated so that the display direction of at least one two-dimensional component after selection is consistent with the changed user gaze direction.

[0064] In some alternative embodiments, the spatial center point of the target display space is determined. In this embodiment, the center point location of the target display space is determined. The center point location is located at the center of the target display space, and the center point location is related to the shape of the target display space.

[0065] The gaze change information indicates the change in the angle of the user's current gaze direction relative to the previously acquired gaze direction. For example, the gaze change information may include the direction and angle of change of the user's current gaze direction relative to the previously acquired gaze direction. Based on the gaze change information, at least one two-dimensional component is controlled to rotate around the spatial center point, thereby rotating the display direction of at least one two-dimensional component.

[0066] For example, such as Figure 6 As shown, the target display space is a "box-shaped" cube space. The display direction corresponding to the two-dimensional component that was last displayed in the target display space is W1. The change information of the user's current gaze direction S2 relative to the previous gaze direction S1 is obtained as shown in the figure. Then, based on the change information of the gaze, the display direction is controlled to rotate around the center point of the target display space to determine the display direction W2 after rotation.

[0067] In some optional embodiments, if there are no other display elements in the surrounding area of ​​the target display space of at least one two-dimensional component, the entire target display space can be rotated according to the line-of-sight change information, wherein at least one two-dimensional component inside the target space rotates along with the rotation of the target display space.

[0068] For example, such as Figure 7 As shown, the target display space is a "box-shaped" cube space. The display direction corresponding to the last time the target display space was displayed is T1. The change information of the user's current gaze direction S4 relative to the last gaze direction S3 is obtained as shown in the figure. Then, based on the change information of the gaze direction, the display direction is controlled to rotate around the spatial center point O of the target display space to determine the rotated display direction T2.

[0069] In one embodiment of this disclosure, a preset number of two-dimensional components are displayed in the target display space, and the user can delete one or more of the two-dimensional components as needed.

[0070] In this embodiment, in response to detecting a deletion determination operation of a target two-dimensional component in at least one two-dimensional component, the target two-dimensional component is deleted. It is then detected whether the number of remaining two-dimensional components in the target display space is less than a preset minimum number. If it is less than the preset minimum number, supplementary two-dimensional components are acquired and displayed in the target display space. For example, the number of supplementary two-dimensional components may be the same as the number of deleted two-dimensional components, or the number of supplementary two-dimensional components may be slightly less than the number of deleted two-dimensional components. The supplementary two-dimensional components may be randomly selected from the current two-dimensional component library, and the preset minimum number is set according to the needs of the scenario.

[0071] It should be noted that the methods for detecting and confirming the deletion of a target two-dimensional component within at least one two-dimensional component differ in different application scenarios: In some possible embodiments, in response to a deletion request instruction for a target two-dimensional component, a deletion control model of the target two-dimensional component is displayed in the target display space, and a trigger confirmation operation on the deletion control model is detected (for example, by performing a trigger confirmation operation through a virtual control device, displaying an operation direction indicator model in the extended reality space, which is used to display the indicator direction in the extended reality space; when the indicator direction is located on the deletion control model, if a trigger confirmation operation is detected, then a deletion confirmation operation is confirmed). In the case of detecting a trigger confirmation operation (the confirmation trigger operation can be a gesture operation or a triggering of a preset control on the virtual control device, etc.), the deletion confirmation operation is confirmed.

[0072] For example, such as Figure 8A As shown, when the operation direction indicator model is a ray trajectory model, in response to the deletion request of the target two-dimensional component P1, the deletion control model C of the target two-dimensional component is displayed in the target display space. When the ray trajectory model points to C, if a trigger confirmation operation for C is detected, it is considered that a deletion confirmation operation has been detected.

[0073] In some possible embodiments, when the target two-dimensional component is a preset function execution component, it is detected whether the target two-dimensional component executes a preset function. The preset function execution component is a type of component with an execution function. After the preset function component completes the execution of the corresponding function, it is considered that the corresponding preset function execution component has failed. For example, if the preset function component can be a "fireworks" component, then after the "fireworks" component finishes playing the "fireworks" animation, it is considered that the corresponding preset function execution component has failed.

[0074] In this embodiment, when the target two-dimensional component is a preset function execution component, it is detected whether the target two-dimensional component executes the preset function. After detecting that the target two-dimensional component executes the preset function, it is confirmed that a deletion confirmation operation has been detected.

[0075] For example, such as Figure 8B As shown, when the target functional component is P1, if it is detected that P1 has executed the corresponding preset function on the active panel, the deletion confirmation operation on P1 is confirmed.

[0076] In some possible embodiments, a drag operation on the target two-dimensional component is detected, and it is determined whether the drag position corresponding to the drag operation exceeds the target display space. If it is detected that the target display space is exceeded, it is confirmed that a deletion determination operation has been detected.

[0077] Specifically, the dragging position corresponding to the dragging operation can be considered to be outside the target display space if the farthest position of the target 2D component from the corresponding spatial boundary of the target display space is greater than a distance threshold (i.e., the target 2D component partially moves out of the target display space); or if the minimum position of the target 2D component from the corresponding spatial boundary of the target display space is greater than a distance threshold (i.e., the target 2D component completely moves out of the target display space). The dragging operation can be executed through a ray trajectory model, whereby when the indicated position of the ray trajectory model is on the target 2D component, if a preset dragging command is detected, the target 2D component is controlled to move following the ray trajectory model.

[0078] For example, such as Figure 8C As shown, when the target two-dimensional component is P3, when it is detected that P3 has moved out of the target display space, a deletion confirmation operation on P3 is confirmed.

[0079] In summary, the display processing method for two-dimensional components in this embodiment of the present disclosure fully utilizes the depth information in the extended reality space when displaying at least one two-dimensional component in the target display space. This allows for various activities of the two-dimensional component in the target space, flexibly realizing interactive experiences under various activities, and enhancing the fun and flexibility of the display processing of two-dimensional components.

[0080] To achieve the above embodiments, this disclosure also proposes a display processing device for two-dimensional components.

[0081] Figure 9 This is a schematic diagram of a display processing device for two-dimensional components provided in an embodiment of this disclosure. This device can be implemented by software and / or hardware and is generally integrated into an electronic device for display processing of two-dimensional components. Figure 9As shown, the device includes: an acquisition module 910, a determination module 920, and a display module 930, wherein, The acquisition module 910 is used to acquire a reference value of the display area required to display at least one two-dimensional component; Module 920 is used to determine the target display space in the extended reality space based on the display area reference value. The target display space includes at least one display plane area with a display area larger than the display area reference value; Display module 930 is used to perform display processing on at least one two-dimensional component in a target display space.

[0082] The display processing device for two-dimensional components provided in this disclosure can execute the display processing method for two-dimensional components provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method. The implementation principle is similar and will not be repeated here.

[0083] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the display processing method for the two-dimensional components in the above embodiments.

[0084] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0085] The following is a detailed reference. Figure 10 The diagram illustrates a structural schematic suitable for implementing the electronic device 1000 in the embodiments of this disclosure. The electronic device 1000 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0086] like Figure 10As shown, the electronic device 1000 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a memory 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processor 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0087] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; memory devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0088] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from memory 1008, or installed from ROM 1002. When the computer program is executed by processor 1001, it performs the functions defined in the display processing method for two-dimensional components of embodiments of this disclosure.

[0089] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0090] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0091] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0092] The aforementioned computer-readable medium carries one or more programs. When these programs are executed by the electronic device, the electronic device causes the following: it acquires a reference value for the display area required to display at least one two-dimensional component; determines a target display space in extended reality space based on the display area reference value, wherein the target display space includes at least one display plane region with a display area larger than the display area reference value; and then performs display processing on the at least one two-dimensional component in the target display space. Thus, by determining the display space of the two-dimensional component based on the required display area reference value, intelligent determination of the display space is achieved, improving the flexibility of two-dimensional component display and fully utilizing the depth characteristics of extended reality space.

[0093] Electronic devices can be programmed with computer program code in one or more programming languages ​​or combinations thereof to perform the operations of this disclosure. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0096] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0097] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0099] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0100] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A display processing method for a two-dimensional component, characterized in that, Includes the following steps: Obtain a reference value for the display area required to display at least one two-dimensional component; The target display space is determined in the extended reality space based on the aforementioned display area reference value. The target display space includes at least one display plane area with a display area larger than the display area reference value; The at least one two-dimensional component is displayed in the target display space.

2. The method as described in claim 1, characterized in that, The step of obtaining the display area reference value required to display at least one two-dimensional component includes: Obtain the maximum tiled display area of ​​the at least one two-dimensional component; The maximum tiled display area is determined as the display area reference value.

3. The method as described in claim 1, characterized in that, Determining the target display space in the extended reality space based on the display area reference value includes: In the extended reality space, determine candidate display spaces for displaying two-dimensional components; The target display space is determined from the candidate display spaces based on the display area reference value.

4. The method as described in claim 1, characterized in that, The display processing of the at least one two-dimensional component in the target display space includes: The reference display position of each of the two-dimensional components is determined in the target display space according to a preset random algorithm; The corresponding two-dimensional component is displayed at the corresponding reference display position.

5. The method as described in claim 4, characterized in that, When there are multiple two-dimensional components, before displaying the corresponding two-dimensional component at the corresponding reference display position, the method further includes: Generate a component bounding box that surrounds each of the two-dimensional components, and determine whether each of the two-dimensional components satisfies a preset collision condition with other two-dimensional components based on the bounding box position of the component bounding box; Update the reference display position of the two-dimensional components that meet the preset collision conditions, so that each of the two-dimensional components does not meet the preset collision conditions with other two-dimensional components.

6. The method as described in claim 5, characterized in that, Also includes: In response to a drag operation on a first two-dimensional component among the plurality of two-dimensional components, the first two-dimensional component is moved according to the drag operation; During the movement, based on the corresponding bounding box position, it is detected whether there is a second two-dimensional component in the bounding box of other two-dimensional components that satisfies the preset collision condition with the first two-dimensional component; In the presence of the second two-dimensional component, collision motion display processing is performed on the second two-dimensional component.

7. The method as described in claim 6, characterized in that, The collision motion display processing of the second two-dimensional component includes: Determine the first movement parameter of the first two-dimensional component when the preset collision condition is met; The second movement parameter of the second two-dimensional component is determined based on the first movement parameter, and the second movement parameter is used to control the collision movement of the second two-dimensional component in the target display space.

8. The method as described in claim 5, characterized in that, Also includes: Based on the bounding box position of the at least one two-dimensional component and the spatial boundary position of the target display space, detect whether there is a third two-dimensional component that satisfies a preset edge collision condition with the spatial boundary of the target display space; In the presence of the third two-dimensional component, determine the third movement parameter of the third two-dimensional component when the preset edge collision condition is met; A fourth movement parameter is determined based on the third movement parameter, and the third two-dimensional component is controlled to bounce and move within the target display space based on the fourth movement parameter.

9. The method as described in claim 1, characterized in that, Also includes: In response to changes in the user's gaze direction, determine the gaze change information of the user's gaze direction; The display direction of the at least one two-dimensional component is rotated according to the change in the user's gaze, so that the display direction of the at least one two-dimensional component after selection is consistent with the changed user gaze direction.

10. The method as described in claim 9, characterized in that, The step of rotating the display orientation of the at least one two-dimensional component according to the gaze change information includes: Determine the spatial center point of the target display space; Based on the line-of-sight change information, control at least one two-dimensional component to rotate around the center point of the space, so as to rotate the display direction of the at least one two-dimensional component.

11. The method as described in claim 1, characterized in that, Also includes: In response to detecting a deletion determination operation for a target two-dimensional component in the at least one two-dimensional component, the target two-dimensional component is deleted; Detect whether the number of remaining two-dimensional components in the target display space is less than a preset lower limit value; If the number is less than the preset minimum value, acquire and display the supplementary two-dimensional component in the target display space.

12. The method as described in claim 11, characterized in that, Prior to the detection of a deletion determination operation for a target two-dimensional component in the at least one two-dimensional component, the method includes: In response to a deletion request instruction for the target two-dimensional component, a deletion control model of the target two-dimensional component is displayed in the target display space. Detect a confirmation operation triggered on the delete control model, wherein, if the confirmation operation is detected, it is confirmed that the delete confirmation operation has been detected; or, When the target two-dimensional component is a component that executes a preset function, it is detected whether the target two-dimensional component executes the preset function. Specifically, after detecting that the target two-dimensional component executes the preset function, it is confirmed that the deletion confirmation operation has been detected; or... Detect a drag operation on the target 2D component, and determine whether the drag position corresponding to the drag operation exceeds the target display space. If it is detected that the drag position exceeds the target display space, confirm that the deletion confirmation operation has been detected.

13. A display processing device for a two-dimensional component, characterized in that, include: The acquisition module is used to acquire a reference value for the display area required to display at least one two-dimensional component; The determination module is used to determine the target display space in the extended reality space based on the display area reference value. The target display space includes at least one display plane area with a display area larger than the display area reference value; The display module is used to perform display processing on the at least one two-dimensional component in the target display space.

14. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the display processing method of the two-dimensional component as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for executing the display processing method of the two-dimensional component according to any one of claims 1-12.

Images