Virtual display interface creation method and device, equipment, storage medium and product

Abstract

The invention provides a virtual display interface creating method and device, equipment, a storage medium and a product. The method comprises the following steps: in response to a creation operation of a user, detecting whether a target object exists in a user operation space or not; in response to the target object existing in the user operation space, acquiring surface geometric data of the target object; determining a target surface area based on the surface geometric data; according to the method, the virtual display interface matched with the contour of the target object is created in the target surface area, the limitation of a fixed form of a traditional virtual display interface can be broken through, the virtual display interface can adapt to flexible generation of target objects with different contours, and the adaptability of the virtual display interface in space use is improved.

Description

Technical Field

[0001] This disclosure relates to the field of metaverse technology, and in particular to a method, apparatus, device, storage medium, and product for creating a virtual display interface. Background Technology

[0002] With the rapid development of Extended Reality (XR), users can obtain information anytime and anywhere using various virtual display interfaces, i.e., virtual screens. However, the existing virtual screen forms are usually determined in the development stage, resulting in fixed virtual display interface forms and insufficient adaptability. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, storage medium, and product for creating a virtual display interface, in order to solve the problems of fixed form and insufficient adaptability of virtual display interfaces in related technologies.

[0004] A first aspect of this disclosure provides a method for creating a virtual display interface, the method comprising: In response to the user's creation action, detect whether the target object exists within the user's operating space; In response to the presence of a target object in the user's operating space, acquire the surface geometry data of the target object; The target surface region is determined based on surface geometry data; Create a virtual display interface on the target surface area that matches the outline of the target object.

[0005] In one embodiment, in response to the presence of a target object within the user's operating space, acquiring the surface geometry data of the target object includes: In response to the existence of a target object in the user's operating space, determine whether the target object is a virtual object; In response to the fact that the target object is a virtual object, the surface geometry data of the target object is obtained from the 3D model data of the target object; In response to the fact that the target object is not a virtual object, acquire the point cloud data of the target object's surface; The point cloud data is processed for 3D reconstruction to generate the surface geometry data of the target object.

[0006] In one embodiment, determining the target surface region based on surface geometry data includes: Based on surface geometry data, determine the normal direction of the target object's surface; Based on the normal direction and the user's viewpoint direction, the outer surface of the target object is determined as the target surface region.

[0007] In one embodiment, creating a virtual display interface matching the contour of the target object on the target surface area includes: In response to the user's occlusion exclusion selection operation, a visibility analysis is performed on each pixel in the target surface region to exclude invisible areas that are occluded from the user's current viewpoint; Determine the visible area based on the target surface area and the invisible area; Create a virtual display interface in the visible area that matches the outline of the target object.

[0008] In one embodiment, after detecting whether a target object exists within the user's operating space in response to a user's creation operation, the method provided in this disclosure further includes: In response to the absence of a target object within the user's operating space, a virtual display interface is created within the user's operating space based on the user's control commands.

[0009] In one embodiment, after creating a virtual display interface matching the contour of the target object on the target surface area, the method provided in this disclosure further includes: In response to the failure to detect the target resource in the virtual display interface, the virtual display interface is configured as the creation interface to respond to the user's creation instructions; In response to the detection of a target resource in the virtual display interface, the virtual display interface is controlled to display the target resource in a rendering mode corresponding to the type of the target resource, based on the type of the target resource.

[0010] A second aspect of this disclosure provides a virtual display interface creation apparatus, the apparatus comprising: The detection unit is used to detect whether a target object exists in the user's operation space in response to the user's creation operation. The acquisition unit is used to acquire the surface geometry data of the target object in response to the existence of the target object in the user's operating space. The determining unit is used to determine the target surface region based on surface geometry data; Create a unit to create a virtual display interface on the target surface area that matches the outline of the target object.

[0011] A third aspect of this disclosure provides an electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0012] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0013] A fifth aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the methods described in the first aspect of this disclosure.

[0014] In summary, this disclosure proposes a method for creating a virtual display interface, which includes: in response to a user's creation operation, detecting whether a target object exists in the user's operation space; in response to the existence of a target object in the user's operation space, acquiring the surface geometric data of the target object; determining the target surface region based on the surface geometric data; and creating a virtual display interface in the target surface region that matches the outline of the target object.

[0015] According to the solution provided in this disclosure, in response to a user's creation operation, the system detects whether a target object exists in the user's operating space; in response to the existence of a target object in the user's operating space, it acquires the surface geometric data of the target object; based on the surface geometric data, it determines the target surface area; and it creates a virtual display interface in the target surface area that matches the outline of the target object. This breaks the limitation of the fixed form of traditional virtual display interfaces, allowing the virtual display interface to be flexibly generated to adapt to target objects with different outlines, thereby improving the adaptability of the virtual display interface in spatial use.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0018] Figure 1 A flowchart illustrating a method for creating a virtual display interface provided in an embodiment of this disclosure; Figure 2 A schematic diagram of a virtual display interface provided in an embodiment of this disclosure; Figure 3 A schematic diagram illustrating yet another virtual display interface provided in an embodiment of this disclosure; Figure 4 A schematic diagram of a creation interface provided in an embodiment of this disclosure; Figure 5 A schematic diagram of the structure of a virtual display interface creation device provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0019] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0020] To facilitate a better understanding of the technical solutions described in the embodiments of this disclosure by those skilled in the art, the technical terms in the embodiments of this disclosure are explained as follows before introducing the embodiments of this disclosure.

[0021] One of the conveniences that XR brings is the availability of various virtual screens, allowing users to access information and engage in entertainment anytime, anywhere. However, the existing virtual screens, in terms of both form and function, were already determined during the program development stage. They don't offer significant differences in personalization, ease of use, and convenience compared to real life, such as the mobile phones we carry with us.

[0022] To address the shortcomings of related technologies, this disclosure detects the existence of a target object within the user's operating space in response to a user's creation operation; acquires the surface geometric data of the target object in response to the existence of the target object within the user's operating space; determines the target surface region based on the surface geometric data; and creates a virtual display interface matching the outline of the target object within the target surface region. This breaks the limitations of the fixed form of traditional virtual display interfaces, allowing the virtual display interface to be flexibly generated to adapt to target objects with different outlines, thereby improving the adaptability of the virtual display interface in spatial use.

[0023] The present disclosure will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0024] The virtual display interface creation method provided in this disclosure can be applied to XR, metaverse, terminal interaction, and game scenarios. For example, it can be applied to XR entertainment and interactive scenarios such as immersive media playback, interactive games, and virtual social interaction, as well as metaverse office collaboration scenarios such as temporary online meetings, multi-person document collaboration, and co-creation of design solutions. The executing entity of the method can be a system / device with XR spatial perception, data processing, graphics rendering, and interaction capabilities, such as an XR terminal device.

[0025] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for creating a virtual display interface according to an embodiment of this disclosure. The method for creating a virtual display interface according to an embodiment of this disclosure includes the following steps: Step 101: In response to the user's creation operation, detect whether there is a target object in the user's operation space; In one embodiment, the creation operation refers to the active operation triggered by the user in the XR space to generate a virtual display interface, including but not limited to gesture operation, resource call operation, terminal controller button operation, etc., wherein the gesture operation can be air drag or triggered by a specific posture, and the resource call operation is to drag and release resources from the associated resource library.

[0026] In one embodiment, the user operating space refers to the three-dimensional spatial range that the XR terminal device can perceive and interact with, centered on the user, and its boundaries are determined by the detection capabilities of the device's sensing modules, such as cameras and sensors.

[0027] In one embodiment, the target object may include a real object or a virtual object, wherein a virtual object refers to a preset three-dimensional (3D) modeling element in XR space, such as a virtual table and chair or a virtual building; a real object refers to a real entity that exists in the physical world, such as a desktop, a wall, or furniture.

[0028] In one embodiment, the user's gestures, such as dragging and releasing with both hands in the air, or pointing to a point in space and holding for 3 seconds, can be captured by the built-in camera of the XR terminal device. The operation is then confirmed by an image recognition algorithm, and a depth sensor can be used to scan the user's operating space to detect whether there is a target object.

[0029] In one embodiment, users can also open the associated resource library through the interactive interface of the XR terminal and drag and drop resources such as audio, video, and documents into the space. When the XR terminal device detects the resource dragging and releasing action, it starts the LiDAR to scan the user's operating space to identify whether there is a target object.

[0030] In one embodiment, a creation command can also be triggered by a custom button, and the terminal simultaneously starts the camera and infrared sensor fusion detection to perform a full-range scan of the operating space to determine whether a target object exists.

[0031] Step 102: In response to the existence of a target object in the user's operating space, acquire the surface geometry data of the target object; In one embodiment, the surface geometry data is mesh data, including but not limited to vertex data and normal data.

[0032] In one embodiment, if the target object is a virtual object, i.e. a 3D modeled object, the surface geometry data of the target object can be obtained from the object's 3D model file.

[0033] In one embodiment, if the target object is a real object, depth information of the object's surface can be collected by a depth camera to generate point cloud data containing three-dimensional coordinate points. Then, a three-dimensional reconstruction algorithm is used to process the point cloud data to construct the surface mesh data of the target object. At the same time, the normal direction of each surface element is calculated to obtain the surface geometric data of the target object.

[0034] Step 103: Determine the target surface region based on surface geometry data; In one embodiment, the target surface area is a specific area selected from the surface of the target object that meets the requirements for creating a virtual display interface. It is typically a surface area that is obstructed by the user, or that is adapted to interactive needs.

[0035] In one embodiment, the orientation of each mesh can be determined based on the normal data in the surface geometry data, and a set of meshes whose normal direction is toward the user's current viewpoint can be selected to form the target surface region.

[0036] In one embodiment, the flatness of each region can be calculated by analyzing the height difference between adjacent vertices in the surface geometry data, and regions whose flatness meets the preset standard can be selected as target surface regions. This is suitable for resource display scenarios with high requirements for interface flatness, such as document display. The preset standard can be that the height difference is no more than 5mm.

[0037] Step 104: Create a virtual display interface on the target surface area that matches the outline of the target object.

[0038] In one embodiment, the virtual display interface refers to an interactive display carrier generated in XR space based on digital technology, which has the functions of displaying resources and receiving user operations. Its outline is adapted to the surface of the target object and can be presented as an irregularly shaped interface or an independent planar interface that fits the object.

[0039] In one embodiment, by calling a graphics engine such as Unity, the mesh material of the target surface area can be replaced with a virtual screen-specific material. Through material rendering, a virtual display interface with display and interactive functions can be directly formed on the target surface area, with the outline completely consistent with the target surface area.

[0040] In one embodiment, a virtual interface mesh that is perfectly adapted to the mesh outline can also be generated based on the mesh data of the target surface area, and a display layer and an interaction layer can be loaded. The interface can then be rendered onto the target surface area through a graphics application programming interface (API), such as OpenGL.

[0041] In one embodiment, occlusion detection can be performed on the target surface area first, traversing all pixels in the area, excluding occluded pixels by ray detection, and generating a virtual display interface only in the unoccluded visible area.

[0042] In one embodiment, the virtual display interface in the space will be retained until the user manually destroys it, and the user can choose to synchronize it so that other users in the space can see the virtual display interface.

[0043] By responding to the user's creation operation, the system first detects whether there is a target object in the user's operating space to flexibly adapt to different spatial usage scenarios. When a target object is detected, the system acquires the surface geometry data of the target object. The surface geometry data can reflect the surface morphology characteristics of the target object. Based on this surface geometry data, the system selects the target surface area that meets the interface creation requirements. Finally, a virtual display interface that matches the outline of the target object is created in the target surface area, achieving a precise fit between the virtual display interface and the target object.

[0044] This application breaks through the limitations of the fixed form of traditional virtual display interfaces, enabling virtual display interfaces to be flexibly generated to adapt to target objects with different outlines, thereby improving the adaptability and flexibility of virtual display interfaces in spatial use.

[0045] In one embodiment, in response to the presence of a target object within the user's operating space, acquiring the surface geometry data of the target object includes: In response to the existence of a target object in the user's operating space, determine whether the target object is a virtual object; In one embodiment, the presence or absence of a preset 3D model data file can be used to determine whether the target object is a virtual object. Specifically, if a file such as a locally stored .obj or .fbx format file can be read, it is determined to be a virtual object; if a preset model file cannot be read, and the spatial information of the object is collected in real time by a sensor, it is determined to be a real object.

[0046] In one embodiment, it is also possible to determine whether a target object is a virtual object by analyzing the spatial existence characteristics of the object. Specifically, if the object only exists in the XR virtual space and does not collide or interact with the real physical environment, such as being able to penetrate a real wall, it is determined to be a virtual object; if the object has collision feedback with the real physical environment, such as being unable to penetrate a user's gesture, it is determined to be a real object.

[0047] In one embodiment, it is also possible to determine whether the target object is a virtual object by matching identification information. Specifically, virtual objects are assigned unique identification information, such as virtual object tags, when they are created. If an object with identification information is detected, it is determined to be a virtual object; if no identification information is detected, it is determined to be a real object.

[0048] In response to the fact that the target object is a virtual object, the surface geometry data of the target object is obtained from the 3D model data of the target object; In one embodiment, the three-dimensional model data is a digital data file used to describe the virtual object, including the object's geometry (vertices, surfaces, normals), material properties, texture information, etc.

[0049] In one embodiment, surface geometry data can be obtained by directly reading the pre-stored surface mesh data and normal data from the 3D model file of the virtual object.

[0050] In one embodiment, if the 3D model data only stores vertex coordinates and surface element connection relationships, and does not preset normal data, then the edge vectors can be determined based on the vertex coordinates, and the normal data can be generated in real time through vector cross product and normalization processing, thereby obtaining the surface geometry data.

[0051] In one embodiment, surface geometric data of corresponding precision can be selected from the 3D model data according to the creation precision requirements of the virtual display interface. For example, high-density grid data can be selected for high-precision scenes, and low-density grid data can be selected for ordinary scenes, so as to balance performance and display effect.

[0052] In response to the fact that the target object is not a virtual object, acquire the point cloud data of the target object's surface; In one embodiment, point cloud data refers to a set of discrete three-dimensional coordinate points on the surface of a target object collected by a sensor. Each point contains (x, y, z) spatial coordinate information, which is used to reflect the surface contour and spatial position of the object.

[0053] In one embodiment, an infrared or laser light can be emitted by a depth camera to receive the reflected signal from the surface of an object, calculate the spatial depth corresponding to each pixel, and generate point cloud data containing three-dimensional coordinates.

[0054] In one embodiment, a lidar sensor can also be used to emit a laser beam toward the target object by rotating and scanning, thereby acquiring depth information from multiple angles on the object's surface and generating high-density, high-precision point cloud data.

[0055] The point cloud data is processed for 3D reconstruction to generate the surface geometry data of the target object.

[0056] In one embodiment, 3D reconstruction processing refers to the process of converting discrete point cloud data into continuous and complete object surface geometric data, connecting adjacent points to construct a surface mesh, and restoring the object surface morphology.

[0057] In one embodiment, the Poisson reconstruction algorithm can be used to perform three-dimensional reconstruction processing on point cloud data. First, an octree is constructed to store the point cloud data. Based on the point cloud normal vector, the interior and exterior of the object are distinguished. By fitting an implicit surface function, the discrete point cloud is transformed into a continuous surface mesh. At the same time, the normal vector of each surface element is calculated to generate surface geometric data.

[0058] Specifically, construct an octree g to store point cloud data, record the depth b of each octree node, and record the depth b of each node. Set the function f's space to: , where n represents the number of convolutions, and each node o has a corresponding space function f.

[0059] In one embodiment, the Marching Cubes (MC) algorithm can also be used to process the point cloud data for three-dimensional reconstruction. The space where the point cloud data is located is divided into a three-dimensional voxel mesh, the density value of each voxel vertex is calculated, the surface contour of the voxel is extracted by judging the distribution of the point cloud within the voxel, a triangular mesh is generated, and the normal data is calculated based on the mesh vertex.

[0060] In one embodiment, the Iterative Closest Point (ICP) algorithm can also be used to perform three-dimensional reconstruction processing on the point cloud data. First, the original point cloud data is denoised and downsampled using a coarse registration algorithm. Then, the ICP algorithm is used to perform fine registration on the processed point cloud to optimize the point cloud distribution accuracy. Subsequently, meshing processing is performed to generate high-precision surface geometric data.

[0061] When a target object is detected within the user's operating space, the attributes of the target object are first determined: if the recognition result is a virtual object, the preset surface mesh data is directly extracted from the 3D model data of the virtual object. This data includes the vertex coordinates, surface cell connection relationships, and preset normal information of the model; if the recognition result is a real object, the 3D spatial coordinates of each pixel in the terminal display screen of the object are determined by collecting the depth data of the scene where the target object is located, and then the point cloud data is combined to form point cloud data. Subsequently, the Poisson reconstruction algorithm is used to mesh the point cloud data, and the point cloud information is stored by constructing an octree. Based on the point cloud normal vector, the surface function of the object is fitted, and the discrete point cloud data is transformed into continuous surface mesh data, finally generating the surface geometry data of the target object.

[0062] By distinguishing the acquisition paths of surface geometry data for virtual and real objects, we can ensure that different types of target objects can provide reliable data support for the creation of virtual display interfaces.

[0063] In one embodiment, determining the target surface region based on surface geometry data includes: Based on surface geometry data, determine the normal direction of the target object's surface; In one embodiment, the normal direction is a vector direction perpendicular to the surface of the target object or a single facet in the surface mesh, used to accurately identify the orientation of the surface.

[0064] For example, for each triangular facet in the surface mesh, extract the three-dimensional coordinates of its three vertices A(x1,y1,z1), B(x2,y2,z2), and C(x3,y3,z3). Calculate the edge vectors AB = (x2-x1, y2-y1, z2-z1) and AC = (x3-x1, y3-y1, z3-z1). Perform a cross product operation on the two vectors to obtain the original normal vector of the facet. Then, the unit normal vector is obtained through normalization. This determines the normal direction of each face element; Based on the normal direction and the user's viewpoint direction, the outer surface of the target object is determined as the target surface region.

[0065] In one embodiment, the user's viewpoint direction is the ray direction pointing from the user's eye to the surface of the target object, reflecting the user's viewing angle of the target object, which can be obtained by the XR terminal device tracking the user's head posture or line of sight in real time.

[0066] In one embodiment, the outer surface of the target object refers to the area on the surface of the target object where the normal direction is within a preset angle range with the user's viewpoint direction, such as 0° to 90°, that is, the unobstructed surface area of ​​the object that can be directly observed from the user's current perspective.

[0067] In one embodiment, the angle between the normal direction of each surface element and the user's viewpoint direction can be calculated, and an angle threshold, such as 90°, can be set. The set of surface elements with an angle less than or equal to the threshold can be determined as the outer surface and combined to form the target surface area, ensuring that the entire area faces the user.

[0068] In one embodiment, the dot product operation can also be performed on the normal direction vector and the user viewpoint direction vector. If the dot product result is positive, it means that the angle between the two vectors is less than 90° and the directions are roughly the same. Then the corresponding surface element belongs to the outer surface. All surface elements that meet the conditions are summarized to form the target surface region.

[0069] In one embodiment, by combining the user's field of view, in addition to determining the angle between the normal and the viewpoint direction, additionally filter out the surface elements located within the user's field of view, and determine the outer surface portion composed of them as the target surface area, so as to avoid generating invalid interfaces that exceed the user's field of view.

[0070] By combining the user's viewpoint in the three-dimensional coordinates of the XR space, the angle between the normal direction of each surface element and the user's viewpoint direction is determined, and the outer surface of the object formed by the set of surface elements whose normal direction is towards the user's viewpoint is determined as the target surface region.

[0071] By calculating the direction of the surface element normal and combining it with the user's viewpoint position to determine the target surface area, the virtual display interface is ensured to be generated only on the outer surface of the object that the user can observe. This avoids the invalid operation of creating the interface on the invisible inner surface or the back surface, which can improve the rationality of the virtual display interface and the user's observation experience.

[0072] In one embodiment, creating a virtual display interface matching the contour of the target object on the target surface area includes: In response to the user's occlusion exclusion selection operation, a visibility analysis is performed on each pixel in the target surface region to exclude invisible areas that are occluded from the user's current viewpoint; In one embodiment, the occlusion removal selection operation refers to an active operation triggered by the user when creating a virtual display interface, based on their own needs, to select whether to remove the occluded part in the target surface area. The occlusion removal selection operation includes, but is not limited to, gesture commands, controller button operations, or voice commands.

[0073] In one embodiment, visibility analysis refers to the process of determining whether each pixel in the target surface area can be observed by the user's current viewpoint, i.e. whether the ray can reach the user's viewpoint without obstruction, in order to filter out unobstructed pixels.

[0074] In one embodiment, the user's current viewpoint refers to the origin of the user's current viewing angle of the target object, which is determined by the XR terminal device in real time by tracking the user's head posture and eye position.

[0075] In one embodiment, the invisible region refers to the area on the target surface where the rays emitted by the pixels are blocked by other objects or the target object itself and cannot reach the user's current viewpoint.

[0076] In one embodiment, a virtual ray can be emitted from each pixel of the target surface area in a direction pointing towards the user's current viewpoint. A collision detection algorithm is used to determine whether the ray intersects with other objects in its propagation path. If the ray is blocked, the pixel is marked as an invisible pixel and included in the invisible area.

[0077] In one embodiment, the mesh data of the target surface region can also be passed to the fragment shader, and visibility judgment logic can be embedded in the shader. By calculating whether the connection between the pixel and the user's viewpoint is occluded by other geometry, invisible pixels can be directly filtered out, and the output can contain only preliminary data containing potentially visible pixels.

[0078] In one embodiment, the target surface area can be divided into multiple sub-regions according to spatial coordinates. Coarse detection is performed on each sub-region. If the entire sub-region is occluded, it is directly marked as an invisible area without traversing the internal pixels. Fine ray detection is performed only on the pixels of the sub-regions that are not completely occluded, which can improve the analysis efficiency.

[0079] Determine the visible area based on the target surface area and the invisible area; In one embodiment, the visible area refers to the portion of the target surface area remaining after removing the invisible areas, that is, the continuous area composed of all pixels from which rays can reach the user's current viewpoint.

[0080] In one embodiment, the boundary contour of the invisible area can be identified first, and the target surface area can be cropped based on the contour to remove the invisible part within the boundary. The edge pixels of the remaining area can be smoothed to determine the visible area.

[0081] Create a virtual display interface in the visible area that matches the outline of the target object.

[0082] In one embodiment, the material of the visible mesh area can be replaced with a material specifically for the virtual screen by calling the graphics engine. When the material is rendered, it only covers the pixels in the visible area, forming a virtual display interface that matches the outline of the target object.

[0083] In one embodiment, the mesh data of the visible area can be extracted, and a perfectly matching virtual interface mesh can be generated based on the data. The interface display layer and interaction layer can then be rendered to the visible area through a graphics API to form a virtual display interface that adapts to the outline of the target object.

[0084] In one embodiment, after the interface is created, the user's viewpoint position or changes in the state of surrounding objects can be monitored in real time. If the invisible area changes, such as the movement of occluding objects, the visibility of each pixel in the target surface area is re-analyzed, the visible area range is dynamically updated, and the outline of the virtual display interface is adjusted synchronously to ensure that the virtual display interface is always fully displayed.

[0085] In response to a user-triggered occlusion exclusion selection operation, all pixels in the target surface area are traversed. A virtual ray is emitted from each pixel in the direction pointing to the user's current viewpoint. It is determined whether the ray will intersect with other objects in its propagation path. If the ray is blocked and cannot reach the user's viewpoint, the pixel is marked as invisible and excluded. After all pixels have been traversed, the continuous area formed by the unmarked visible pixels is determined as the final visible area. Based on the mesh data and normal information of the visible area, a virtual display interface that perfectly fits the outline of the target object is created.

[0086] By using pixel-level visibility analysis to eliminate occlusion areas, the virtual display interface is generated only in the user's unobstructed visible area, avoiding the problem of incomplete interface display caused by occlusion. This is especially suitable for scenarios with high requirements for display integrity, such as video playback.

[0087] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a virtual display interface provided in an embodiment of the present disclosure, wherein, Figure 2 The left side of the middle section does not remove the obstruction. Figure 2 The right side of the screen removes the lower occlusion. If the user chooses not to remove the occlusion, a virtual screen will be generated on the surface composed of all pixels.

[0088] In one embodiment, in response to a user's creation operation, after detecting whether a target object exists within the user's operating space, the virtual display interface creation method further includes: In response to the absence of a target object within the user's operating space, a virtual display interface is created within the user's operating space based on the user's control commands.

[0089] In one embodiment, control commands refer to instructions used by the user in a scenario without a target object to define virtual display interface parameters, trigger interface creation, and make subsequent adjustments. These commands include, but are not limited to, gesture commands, terminal controller operation commands, and voice commands, and are used to specify the interface's position, size, orientation, and other attributes.

[0090] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram of another virtual display interface provided in the embodiments of this disclosure. According to the initial position coordinates (x, y, z), length / width dimensions, and orientation angle parsed from the control commands, a corresponding rectangular planar grid can be directly generated, and a virtual screen material and display layer can be loaded to form a virtual display interface, that is, a virtual screen in space. The orientation of the virtual reality interface is adapted to the user's field of vision.

[0091] In one embodiment, the system may receive only the location information from the user's control commands and adaptively match the interface size based on the user's current field of view and operating space. For example, it may generate a 16:9 aspect ratio plane suitable for single-user viewing by default and make the interface orientation always adjust synchronously with the user's head rotation.

[0092] In one embodiment, a basic virtual display interface can be generated first based on control commands. Users can then adjust the size, position, and orientation of the interface in real time through subsequent supplementary control commands, such as gesture zooming and controller dragging. The virtual display interface state is updated synchronously until the user confirms the final form.

[0093] When no target object is detected in the user's operating space, the system receives control commands from the user via gestures or the terminal controller. These commands include interface size parameters, initial position coordinates, and orientation angle information. Based on the initial position coordinates in the command, an initial rectangular planar grid is generated at the corresponding position in the user's operating space. The length and width dimensions of the grid are adjusted based on the interface size parameters to ensure that the plane orientation is consistent with the user's field of vision, thus forming a virtual display interface. At the same time, the user can adjust the scaling ratio, spatial position, and orientation of the interface in real time through subsequent gesture operations.

[0094] It provides the function of creating independent virtual display interfaces in scenarios without target objects, enriching the application scenarios of interface creation, and supports users to customize interface parameters and adjust them in real time, meeting users' personalized needs for display interfaces in open virtual spaces.

[0095] In one embodiment, after creating a virtual display interface matching the contour of the target object on the target surface area, the virtual display interface creation method further includes: In response to the failure to detect the target resource in the virtual display interface, the virtual display interface is configured as the creation interface to respond to the user's creation instructions; In one embodiment, the target resource is digital content selected by the user from the associated resource library and displayed on the virtual display interface, including but not limited to multimedia resources such as audio and video, pictures, and document resources such as PPT and Word documents.

[0096] In one embodiment, the creation interface refers to the functional form of the virtual display interface when no target resource is detected. It has the ability to receive user creation instructions and render creation content in real time. That is, the virtual display interface is an interactive digital whiteboard.

[0097] In one embodiment, the creation instruction is an operation instruction that the user triggers a creation behavior on the creation interface, including but not limited to gesture drawing instructions, controller drawing instructions, and pen stroke parameter adjustment instructions, such as color and thickness selection.

[0098] In one embodiment, when it is detected that there are no target resources on the virtual display interface, a preset drawing and rendering module is directly loaded to initialize the interface as a white canvas. At the same time, the gesture recognition function is activated to allow users to simulate drawing on the interface with their fingers or controllers and render the drawing trajectory in real time.

[0099] In one embodiment, after the XR terminal device switches to the creation mode, a virtual parameter panel pops up. The user can select the pen type, such as pencil, brush, airbrush, color, thickness, etc., through gestures or controllers. The creation function is configured according to the user's selected parameters, responds to the user's creation instructions, and renders the effect according to the configuration.

[0100] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram of a creative interface provided in an embodiment of the present disclosure. After the virtual display interface is constructed, it is a white panel. When there are no resources displayed on it, it is a canvas. Users can use their devices to draw on the panel. All operations are synchronous, meaning that other users can also see this process and participate in the painting and create together.

[0101] In response to the detection of a target resource in the virtual display interface, the virtual display interface is controlled to display the target resource in a rendering mode corresponding to the type of the target resource, based on the type of the target resource.

[0102] In one embodiment, the rendering mode refers to the display method adapted by the virtual display interface according to the type of the target resource. Different types of resources correspond to exclusive rendering logic, such as video playback rendering and document pagination display rendering, to ensure that the resource display effect is adapted to its attributes.

[0103] In one embodiment, if the detected target resource is audio or video, parameters such as resource resolution and frame rate are identified, and the video is played in full-screen or user-specified area mode, supporting interactive operations such as pause, fast forward, and volume adjustment; if it is an image resource, it is displayed by adaptive scaling according to the interface size, supporting gesture scaling and rotation viewing.

[0104] In one embodiment, if the detected target resource is a document such as PPT or Word, the terminal calls the document parsing and rendering module to convert the document content into a format adapted to the virtual display interface, displays it page by page number, supports gesture-based page turning and clicking to jump to a specified page, and preserves the document's layout style and font format.

[0105] After creating a virtual display interface on the target surface area, the interface is monitored in real time to see if the user-submitted target resource exists. If no target resource is detected, the interface's interaction mode is switched to creation mode, the drawing and rendering module is loaded, and basic creation parameters such as stroke thickness and color selection are configured, enabling the interface to receive user drawing operation commands and render drawing trajectories in real time. If a target resource is detected, the resource type is identified. When the resource is audio or video, it is displayed in full-screen or specified area playback mode. When the resource is a document or image, it is displayed in pagination or full-screen mode adapted to the interface size, ensuring that different types of resources can be presented in an appropriate rendering mode.

[0106] This application enables the virtual display interface to adaptively switch between creation and resource display functions, satisfying users' need for free creation when resources are unavailable, while also providing suitable display methods for different types of resources, thus enhancing the interface's versatility and ease of use.

[0107] In summary, the solution provided in this public disclosure is as follows: First, in response to the user's creation operation, the system detects whether a target object exists within the user's operating space. Upon confirming the presence of a target object, it acquires the surface geometry data of the target object. Based on this surface geometry data, it determines the target surface region. Finally, it creates a virtual display interface in the target surface region that matches the outline of the target object. This breaks the limitations of traditional fixed-form virtual display interfaces, allowing them to flexibly adapt to target objects with different outlines, thus improving the adaptability of the virtual display interface in spatial usage.

[0108] Secondly, by using pixel-level visibility analysis to eliminate occlusion areas, the virtual display interface is generated only in the user's unobstructed visible area, avoiding the problem of incomplete interface display caused by occlusion. This is especially suitable for scenarios with high requirements for display integrity, such as video playback.

[0109] To implement the virtual display interface creation method provided in this disclosure, this disclosure also provides a virtual display interface creation apparatus, such as... Figure 5 As shown. Figure 5 This is a schematic diagram of a virtual display interface creation device provided in an embodiment of the present disclosure. The virtual display interface creation device 500 includes: The detection unit 501 is used to detect whether a target object exists in the user's operation space in response to the user's creation operation. The acquisition unit 502 is used to acquire the surface geometry data of the target object in response to the existence of the target object in the user operation space. The determining unit 503 is used to determine the target surface region based on surface geometry data; Creation unit 504 is used to create a virtual display interface on the target surface area that matches the outline of the target object.

[0110] In one embodiment, the acquisition unit 502 is specifically used for: In response to the existence of a target object in the user's operating space, determine whether the target object is a virtual object; In response to the fact that the target object is a virtual object, the surface geometry data of the target object is obtained from the 3D model data of the target object; In response to the fact that the target object is not a virtual object, acquire the point cloud data of the target object's surface; The point cloud data is processed for 3D reconstruction to generate the surface geometry data of the target object.

[0111] In one embodiment, the determining unit 503 is specifically used for: Based on surface geometry data, determine the normal direction of the target object's surface; Based on the normal direction and the user's viewpoint direction, the outer surface of the target object is determined as the target surface region.

[0112] In one embodiment, the creation unit 504 is specifically used for: In response to the user's occlusion exclusion selection operation, a visibility analysis is performed on each pixel in the target surface region to exclude invisible areas that are occluded from the user's current viewpoint; Determine the visible area based on the target surface area and the invisible area; Create a virtual display interface in the visible area that matches the outline of the target object.

[0113] In one embodiment, the creation unit 504 is specifically used for: In response to the absence of a target object within the user's operating space, a virtual display interface is created within the user's operating space based on the user's control commands.

[0114] In one embodiment, the virtual display interface creation device further includes a control unit, which is used for: In response to the failure to detect the target resource in the virtual display interface, the virtual display interface is configured as the creation interface to respond to the user's creation instructions; In response to the detection of a target resource in the virtual display interface, the virtual display interface is controlled to display the target resource in a rendering mode corresponding to the type of the target resource, based on the type of the target resource.

[0115] It should be noted that the virtual display interface creation device provided in the above embodiments is only illustrated by the division of the above program modules when creating a virtual display interface. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the virtual display interface creation device can be divided into different program modules to complete all or part of the processing described above. In addition, the virtual display interface creation device provided in the above embodiments and the virtual display interface creation method provided in this disclosure belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0116] Figure 6 This is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 6 As shown, the electronic device 600 includes at least one processor 602; and a memory 601 communicatively connected to the at least one processor 602; wherein the memory 601 stores instructions executable by the at least one processor 602, the instructions being executed by the at least one processor 602 to implement the steps of the virtual display interface creation method of the present disclosure embodiments.

[0117] Optionally, the electronic device may specifically be a virtual display interface creation device according to the embodiments of this application, and the electronic device may implement the corresponding processes implemented by the virtual display interface creation device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0118] It is understood that the electronic device also includes a communication interface 603. Various components in the electronic device are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designated all buses as Bus System 604.

[0119] It is understood that memory 601 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 601 described in this embodiment of the invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0120] The methods disclosed in the above embodiments can be applied to or implemented by processor 602. Processor 602 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 602 or by instructions in software form. Processor 602 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 602 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically memory 601. Processor 602 reads information from memory 601 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0121] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0122] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the steps of the virtual display interface creation method of the present invention.

[0123] Optionally, the computer-readable storage medium can be applied to the virtual display interface creation apparatus in the embodiments of this application, and the computer instructions cause the computer to execute the corresponding processes implemented by the virtual display interface creation apparatus in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0124] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the virtual display interface creation method provided in this embodiment of the invention.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0126] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0127] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0128] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0129] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0130] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for creating a virtual display interface, characterized in that, include: In response to the user's creation action, detect whether the target object exists within the user's operating space; In response to the presence of a target object in the user operation space, the surface geometry data of the target object is acquired; Based on the surface geometry data, the target surface region is determined; A virtual display interface matching the outline of the target object is created on the target surface area.

2. The method according to claim 1, characterized in that, The step of acquiring the surface geometry data of the target object in response to the existence of a target object in the user operating space includes: In response to the existence of a target object in the user operation space, determine whether the target object is a virtual object; In response to the fact that the target object is a virtual object, the surface geometry data of the target object is obtained from the three-dimensional model data of the target object; In response to the fact that the target object is not a virtual object, the point cloud data of the surface of the target object is acquired; The point cloud data is subjected to three-dimensional reconstruction processing to generate the surface geometric data of the target object.

3. The method according to claim 1, characterized in that, Determining the target surface region based on the surface geometry data includes: Based on the surface geometry data, the normal direction of the target object's surface is determined; Based on the normal direction and the user's viewpoint direction, the outer surface of the target object is determined as the target surface region.

4. The method according to claim 1, characterized in that, Creating a virtual display interface that matches the outline of the target object on the target surface area includes: In response to the user's occlusion exclusion selection operation, a visibility analysis is performed on each pixel in the target surface region to exclude invisible areas that are occluded from the user's current viewpoint; Based on the target surface area and the invisible area, determine the visible area; A virtual display interface matching the outline of the target object is created in the visible area.

5. The method according to claim 1, characterized in that, After detecting whether a target object exists within the user's operation space in response to the user's creation operation, the method further includes: In response to the absence of a target object within the user's operating space, a virtual display interface is created within the user's operating space based on the user's control commands.

6. The method according to claim 1, characterized in that, After creating a virtual display interface matching the contour of the target object in the target surface area, the method further includes: In response to the absence of a target resource detected in the virtual display interface, the virtual display interface is configured as a creation interface to respond to the user's creation instructions; In response to detecting a target resource in the virtual display interface, the virtual display interface is controlled to display the target resource in a rendering mode corresponding to the type of the target resource, based on the type of the target resource.

7. A virtual display interface creation device, characterized in that, include: The detection unit is used to detect whether a target object exists in the user's operation space in response to the user's creation operation. The acquisition unit is configured to acquire the surface geometric data of the target object in response to the existence of the target object in the user operation space. A determining unit is used to determine a target surface region based on the surface geometry data; A creation unit is used to create a virtual display interface in the target surface area that matches the outline of the target object.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images