Menu interface display method and device, medium and product

By using virtual reality technology and AR devices, a virtual extended screen is created to display the menu interface, solving the problem of menus obscuring the game screen and improving the user experience and the realism of the game.

CN119034197APending Publication Date: 2024-11-29MIGU COMIC CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411269296.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In existing technologies, menu interfaces obscure the game screen, leading to a reduced user experience, and in some scenarios, it is impossible to add a display screen to solve this problem.

Method used

By acquiring real-time images of the physical display screen using virtual reality technology, creating a 3D model, and displaying the menu interface on the virtual extended screen without obstructing the game screen, AR devices are used for interactive adjustments.

Benefits of technology

It improves the user's visual experience and immersive experience, solves the problem of space limitations, eliminates the need for additional displays, and enhances the realism and interactivity of the game.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119034197A_ABST
    Figure CN119034197A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of augmented reality, and provides a menu interface display method and device, a medium and a product, and the method comprises the steps: obtaining a real-time shot image containing a physical display screen based on a virtual reality technology; performing three-dimensional modeling on the physical display screen based on the real-time shot image to obtain a three-dimensional reconstruction model corresponding to the physical display screen; a virtual expansion screen is created based on the three-dimensional reconstruction model, the position of the virtual expansion screen is set in a preset range of a display picture frame of the physical display screen, and the virtual expansion screen is displayed in the physical display screen; and displaying a menu interface in the display picture frame on the virtual extension screen, and not displaying the menu interface in the display picture frame. Therefore, the menu interface can be displayed outside the display picture frame, and the visual effect and immersive experience of a user are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of augmented reality technology, and in particular to a method, device, medium and product for displaying a menu interface. Background Art

[0002] With the development of game consoles (such as PS5, Xbox One, PC, Switch, etc.), the graphics of console games are becoming more and more realistic. Currently, some console games can reach 4k60fps. For some mainstream games, although the graphics of the games are becoming more and more realistic, due to the characteristics of the games, some menu interfaces (maps, health bars, subtitles, menu bars, etc.) have to be added to the game interface, which reduces the realism of the game, and the menu interface will block part of the game's viewing angle, thereby reducing the user's visual effect.

[0003] The existing solution is usually to add a new display screen and display the game interface on another display screen. However, for scenarios such as hanging the monitor on the wall in the living room, adding a new display screen is inconvenient, and it is easy to be unable to accommodate two displays due to space limitations, thus failing to solve the above-mentioned problem and reducing the user's gaming experience. Summary of the invention

[0004] The embodiments of the present application provide a method, device, medium and product for displaying a menu interface to solve the technical problem in the related art that the display screen is blocked by the menu, thereby reducing the user experience.

[0005] To solve the above technical problems, the embodiments of the present application provide the following aspects:

[0006] In a first aspect, an embodiment of the present application provides a method for displaying a menu interface, the method comprising:

[0007] Acquire real-time captured images including physical display screens based on virtual reality technology;

[0008] Performing three-dimensional modeling on the physical display screen based on the real-time captured image to obtain a three-dimensional reconstruction model corresponding to the physical display screen;

[0009] Creating a virtual extended screen based on the three-dimensional reconstructed model, setting the position of the virtual extended screen within a preset range of a display frame of the physical display screen, wherein the virtual extended screen is displayed on the physical display screen;

[0010] The menu interface in the display picture frame is displayed on the virtual extended screen, and the menu interface is not displayed in the display picture frame.

[0011] Optionally, after creating the virtual extended screen based on the three-dimensional reconstruction model, the method further includes:

[0012] Based on the three-dimensional reconstructed model, aligning the center of the virtual extended screen with the center of the physical display screen;

[0013] Adjusting the geometric features of the virtual extended screen so that the geometric features of the virtual extended screen are completely consistent with the geometric features of the display picture frame, or making the geometric features of the virtual extended screen proportionally enlarged to the geometric features of the display picture frame, to obtain an adjusted virtual extended screen;

[0014] Setting the position of the virtual extended screen within a preset range of the display frame of the physical display screen includes:

[0015] The position of the adjusted virtual extended screen is set within a preset range of the display frame of the physical display screen.

[0016] Optionally, setting the position of the virtual extended screen within a preset range of a display frame of the physical display screen includes:

[0017] The virtual extended screen is positioned behind the display frame of the physical display screen and is located on the non-display side of the physical display screen.

[0018] Optionally, setting the position of the virtual extended screen behind the display frame of the physical display screen and located on the non-display side of the physical display screen includes:

[0019] The depth value of the pixel point in the display picture frame is adjusted to be smaller than the depth value of the corresponding pixel point in the virtual extended screen, so as to set the position of the virtual extended screen behind the display picture frame of the physical display screen and on the non-display side of the physical display screen.

[0020] Optionally, displaying the menu interface in the display picture frame on the virtual extended screen includes:

[0021] receiving an adjustment instruction from a user for a menu interface displayed on the virtual extended screen, and adjusting the menu interface according to the adjustment instruction, wherein the adjustment includes at least one of the following: moving, scaling;

[0022] Displaying the adjusted menu interface;

[0023] The adjusted menu interface does not block the target display screen in the display screen frame.

[0024] Optionally, the method further includes:

[0025] receiving a conversion instruction from a user for a target object in the target display screen;

[0026] According to the conversion instruction, convert the target object into a virtual target object;

[0027] The virtual target object is displayed on the virtual extended screen.

[0028] Optionally, before displaying the menu interface in the display picture frame on the virtual extended screen, the method further includes:

[0029] Extracting files related to the menu interface, wherein the files related to the menu interface include at least one of the following: text, image, icon, and style sheet of the menu interface;

[0030] The content of the related file is rendered to the virtual extended screen based on the Static Mesh component, so as to display the menu screen on the virtual extended screen.

[0031] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of a method for displaying a menu interface as described in the first aspect are implemented.

[0032] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for displaying a menu interface as described in the first aspect are implemented.

[0033] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of a method for displaying a menu interface as described in the first aspect.

[0034] Therefore, the method shown in the embodiment of the present application can display the menu interface outside the display screen, which can improve the user's visual effect and make the effect of other people watching the display screen on the spot more realistic, which can improve the immersive experience of users and viewers, and there is no need to add a new display screen, which can solve technical problems such as use in specific scenarios and space limitations. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0036] Figure 1 A flowchart of a method for displaying a menu interface provided by an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a virtual extended screen provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of outputting a menu interface to a virtual extended screen for display provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of using a virtual hand model to move and / or scale a menu interface within a virtual expansion area provided in an embodiment of the present application;

[0040] Figure 5 A structural block diagram of a menu interface display device provided in an embodiment of the present application;

[0041] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] Figure 1 A method for displaying a menu interface according to an embodiment of the present application is shown. Figure 1 As shown, the method includes:

[0044] Step S101: acquiring a real-time captured image including a physical display screen based on virtual reality technology;

[0045] Step S102: Perform three-dimensional modeling on the physical display screen based on the real-time captured image to obtain a three-dimensional reconstruction model corresponding to the physical display screen;

[0046] Step S103: creating a virtual extended screen based on the three-dimensional reconstruction model, and setting the position of the virtual extended screen within a preset range of a display screen frame of the physical display screen;

[0047] Among them, the virtual extended screen is displayed on the physical display screen;

[0048] Step S104: Display the menu interface in the display picture frame on the virtual extended screen, and do not display the menu interface in the display picture frame.

[0049] In step S101 and step S102, a real-time captured image including a physical display screen may be obtained based on virtual reality technology. Virtual reality (VR) is a computer technology that simulates real feelings and experiences through a virtual environment generated by a computer, allowing users to interact with the virtual environment as if they were actually there. Virtual reality technology generally includes the following aspects:

[0050] 3D graphics: Use computer-generated 3D graphics to simulate virtual environments, including buildings, landscapes, characters, etc. Head-mounted display: Use head-mounted display to project 3D graphics into the user's eyes so that the user can see the virtual environment. Interactive devices: Use interactive devices such as handles, gloves, sensors, etc. to allow users to interact with the virtual environment, such as touching, grabbing, throwing, etc.

[0051] Specifically, AR (Augmented Reality) devices (such as AR glasses or head-mounted devices) can be used to obtain real-time camera images containing real displays in real time, and the target scene can be modeled using the SFM (Structure from Motion) algorithm or the SLAM (Simultaneous Localization and Mapping) algorithm to perform three-mode modeling, and store key frames, camera poses, 2D feature points and their matching 3D map points and other information. When performing positioning, the user inputs an image, and based on the image retrieval method, the most similar key frame is found, and the input image is matched with the similar key frame for 2D feature matching and 3D information association. Finally, based on the PNP (Perspective-n-Point, 3D to 2D point pair object motion positioning method) algorithm, the reprojection error is optimized and minimized, the camera pose is estimated, and the positioning is completed.

[0052] In a possible implementation, after creating the virtual extended screen based on the three-dimensional reconstruction model in step S103, the method further includes: aligning the center of the virtual extended screen with the center of the physical display screen based on the three-dimensional reconstruction model, and adjusting the geometric features of the virtual extended screen so that the geometric features of the virtual extended screen are completely consistent with the geometric features of the display screen frame, or making the geometric features of the virtual extended screen a proportional enlargement of the geometric features of the display screen frame, to obtain the adjusted virtual extended screen; correspondingly, in step S103, setting the position of the virtual extended screen within a preset range of the display screen frame of the physical display screen includes:

[0053] The position of the adjusted virtual extended screen is set to be within a preset range of a display picture frame of the physical display screen.

[0054] In a possible implementation, setting the position of the virtual extended screen within a preset range of the display screen frame of the physical display screen includes: setting the position of the virtual extended screen behind the display screen frame of the physical display screen and on the non-display side of the physical display screen. And setting the position of the virtual extended screen behind the display screen frame of the physical display screen and on the non-display side of the physical display screen includes:

[0055] The depth value of the pixel in the display frame is adjusted to be smaller than the depth value of the corresponding pixel in the virtual extended screen, so as to set the position of the virtual extended screen behind the display frame of the physical display screen and on the non-display side of the physical display screen.

[0056] It should be noted that virtual expansion screen materials can be placed based on the three-dimensional reconstructed model data: First, use Unreal Engine (a game and visual effects development tool) to create a new plane object in the three-dimensional model, which is named "VirtualDisplaySurface (a hypothetical object name, which represents a plane object in Unreal Engine, usually used to represent or simulate the surface of a display device.)" by default. In the properties of the "VirtualDisplaySurface" object, set its material to "VirtualDisplay (a material name, which is preset or user-defined, and is used to give display characteristics to the surface of an object. In Unreal Engine, the material defines the surface appearance of the object, including its color, texture, reflectivity, etc.)" material. The "VirtualDisplay" material is a preset material. In the "VirtualDisplay" material, you can select the "Emissive (In computer graphics, Emissive refers to a component of the material's luminous properties, which defines the brightness and color emitted by the object's material itself.)" color channel and set it to the color and brightness of the virtual display. Use the Unreal Engine scene editor to create a new blueprint class named "VirtualDisplayContent" by default (this is a blueprint class defined in Unreal Engine. Blueprint is a visual scripting system of UnrealEngine used to create game logic and interactivity.)". In the "VirtualDisplayContent" blueprint, add a "Static Mesh" component to display the virtual screen. In the scene editor, instantiate the "VirtualDisplayContent" blueprint to the "VirtualDisplaySurface" object to generate a virtual extended screen. Secondly, the position of the real display and the display frame are identified from the real-time image. According to the 3D reconstructed model, the center of the virtual extended screen is aligned with the center of the real display and adjusted to an appropriate size. The virtual extended screen can be enlarged proportionally based on the ratio of the game interface frame or based on a preset ratio. Then, the plane where the virtual extended screen is located is aligned with the plane of the display frame, and the depth value of the display frame part (the distance information from each point in the scene in the image to the camera) is set to Di=Di-1 when calculating the occlusion relationship, so that the display frame is always displayed on the virtual extended screen (such as Figure 2 shown) before.

[0057] In a possible implementation, in step S104, before displaying the menu interface in the display frame on the virtual extended screen, the method further includes: extracting files related to the menu interface; rendering the contents of the files related to the virtual extended screen based on the Static Mesh component to display the menu screen on the virtual extended screen. The files related to the virtual extended screen include at least one of the following: text, images, icons, and style sheets of the menu interface.

[0058] And step S104, displaying the menu interface in the display picture frame on the virtual extended screen includes: receiving an adjustment instruction from the user for the menu interface displayed on the virtual extended screen, and adjusting the menu interface according to the adjustment instruction, wherein the adjustment includes at least one of the following: moving, scaling; displaying the adjusted menu interface; wherein the adjusted menu interface does not block the target display picture in the display picture frame.

[0059] In addition, in step S104, not displaying the menu interface in the display picture frame includes: setting the transparency of the menu interface in the display picture frame to zero.

[0060] Step S104 is explained with a specific application scenario. Optionally, the display screen frame is a game interface frame, and the size of the game interface frame is enlarged based on the original game interface (including the menu interface), such as enlarging the original game interface so that the menu interface is moved out of the original game interface, and the outer frame of the enlarged game interface at this time is used as the size of the virtual expansion frame. Optionally, the color of the virtual expansion frame can be set to black. Generally, displays have black borders on the outside of the screen. Setting the background color of the virtual expansion frame to black makes it more in line with the visual habits of the human eye. In the "VirtualDisplay" material, you can select the "Emissive" color channel and set it to black.

[0061] When the menu interface needs to be output to the virtual expansion frame, the menu interface file is obtained from the game file (the menu interface file can be annotated in advance by manual annotation, or by obtaining the files contained in all output images except the bottom layer output in the output interface), and the Static Mesh component is used to display the obtained menu interface file on the screen (such as Figure 3 and set the transparency of the menu interface on the original display to 0.

[0062] Optionally, the menu interface is displayed in a window on the virtual extended screen, and the menu interface is displayed based on a preset display strategy.

[0063] The preset display strategies are as follows:

[0064] 1. A virtual expansion frame is enlarged based on the original game interface (including the menu interface), and the virtual expansion frame is output as the output screen according to the display position of the original menu interfaces. Each menu interface is enlarged and output to the virtual expansion frame based on the ratio of the virtual expansion frame to the game screen, and each menu interface is located outside the original game interface.

[0065] 2. Enlarge the virtual expansion frame in proportion to the ratio of the game interface frame, and output the virtual expansion frame as the output screen according to the display position of the original menu interfaces. If a part of the output menu interface is located in the game interface, the menu interface is scaled so that the menu interface is moved outside the game interface. Further, scaling the menu interface includes: taking the corner point of the menu interface frame farthest from the game interface as the scaling base point, and scaling the menu interface in the direction of the scaling base point.

[0066] 3. Display the menu interface based on the preset position, set a coordinate outside the game interface as the placement coordinate, align the corner point of the menu interface frame close to the game interface side with the placement coordinate, and complete the placement of the menu interface.

[0067] In addition, the system time can be displayed in the virtual expansion box. Since most console games do not display the system time in the game screen (displaying the system time in the game screen affects the gaming experience), players need to exit the game or use external devices (such as mobile phones) to check the time, which is inconvenient. Displaying the system time in the virtual expansion box can make it easier for players to check the time. Optionally, the AR device and the game console are connected to the same network for file transfer, thereby ensuring the efficiency of file transfer and reducing the freeze of the menu interface.

[0068] In a possible implementation, the menu interface may be adjusted according to the needs of different users. For example, the menu interface may be adjusted based on the user's gestures (eg, Figure 4As shown). Specifically, the user's hand is placed in the display perspective of the AR device, and the user's hand is virtualized by the AR device to display a virtual hand model; the menu interface in the virtual expansion area is moved and scaled by the virtual hand model, such as: if the thumb and index finger of the virtual hand model are placed on a menu interface and moved left and right, the menu interface is moved; if the thumb and index finger of the virtual hand model are placed on a menu interface and opened outward, the menu interface is enlarged; if the thumb and index finger of the virtual hand model are placed on a menu interface and close to each other, the menu interface is reduced; the above adjustment method can be used to move or scale some menu interfaces (such as maps, subtitles), thereby improving the gaming experience of players (such as myopic players). Preset voice control commands can also be used to control specific menu interfaces in the virtual expansion area, such as voice control commands such as enlarging subtitles, enlarging maps, reducing subtitles, and reducing maps, so that commonly used menu interfaces can be controlled more conveniently.

[0069] In a possible implementation, the method further includes: receiving a conversion instruction from a user for a target object in a target display screen; converting the target object into a virtual target object according to the conversion instruction; and displaying the virtual target object on the virtual extended screen.

[0070] It should be noted that the specific object in the game screen specified by the user can be displayed in the virtual expansion area in a 3D manner, which specifically includes the following steps:

[0071] 1. Obtain user instructions and determine the objects that need to be displayed in the virtual expansion area on the game screen. The user instructions can be voice, such as: displaying the characters or monsters in the game scene in the virtual expansion area, and using speech-to-text technology and NLP (Natural Language Processing) technology to understand the user's instruction intentions, including entity recognition (such as: characters).

[0072] 2. Starting from obtaining user instructions, multiple frames of game screens are obtained at fixed time intervals. Pre-trained semantic segmentation models, such as SAM (Segment Anything Model) and Mask R-CNN (Instance Segmentation Model), are used to perform semantic segmentation on all the acquired frames to obtain pixel-level semantic labels for each area, and the image area closest to the character label is segmented to obtain a collection of multiple character images without background, which can be recorded as In;

[0073] 3. Input the acquired image set In into a 3D reconstruction model that does not rely on camera pose estimation, such as GGRt (Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time), to reconstruct a 3D model of the character, output the image of the 3D model of the character through the GGRt renderer, and use the Static Mesh component to display the renderer output image in a virtual expansion area at a preset position.

[0074] Among them, GGRt is a method based on 3D Gaussian splashing, which realizes real-time new perspective synthesis and 3D scene rendering without camera pose estimation through a joint learning framework and an efficient caching mechanism. This method combines the Iterative Pose Optimization Network (IPO-Net) and the Generalized 3D Gaussian Model (G-3DG), which can robustly estimate relative pose information from image observations and perform high-resolution training and inference under memory constraints through the DBP (Delay backpropagation) mechanism.

[0075] 4. Utilizing the existing gesture tracking algorithm, when it is recognized that the virtual hand is within the displayed 3D model area, the rotation of the 3D model is controlled by moving the virtual hand model, thereby displaying the specific object in the game screen specified by the user in 3D within the virtual expansion area.

[0076] In summary, using AR devices to display the menu interface outside the game interface can improve the user's visual effect, and make the picture of others watching the scene or watching the live video more realistic, which can improve the immersive experience of players and viewers; there is no need to add a new display screen, which can solve the problems of use in specific scenes and space limitations; some menu interfaces (such as maps, subtitles) can be moved or scaled, thereby improving the gaming experience of players (such as myopic players), and providing more convenient and diverse AR interaction methods.

[0077] Figure 5 A display device for a menu interface provided according to an embodiment of the present application is shown, such as Figure 5 As shown, the device comprises:

[0078] An acquisition module 501 is used to acquire a real-time captured image including a physical display screen based on virtual reality technology;

[0079] An execution module 502 is used to perform three-dimensional modeling on the physical display screen based on the real-time captured image to obtain a three-dimensional reconstruction model corresponding to the physical display screen;

[0080] Creating a virtual extended screen based on the three-dimensional reconstruction model, and setting a position of the virtual extended screen within a preset range of a display frame of the physical display screen, wherein the virtual extended screen is displayed on the physical display screen;

[0081] On the virtual extended screen, the menu interface in the display picture frame is displayed, and the menu interface is not displayed in the display picture frame.

[0082] In a possible implementation, the execution module 502 is further configured to align the center of the virtual extended screen with the center of the physical display screen based on the three-dimensional reconstruction model after the virtual extended screen is created based on the three-dimensional reconstruction model;

[0083] Adjusting the geometric features of the virtual extended screen so that the geometric features of the virtual extended screen are completely consistent with the geometric features of the display picture frame, or making the geometric features of the virtual extended screen an equal-proportion enlargement of the geometric features of the display picture frame, to obtain an adjusted virtual extended screen;

[0084] The position of the virtual extended screen is set within the preset range of the display frame of the physical display, including:

[0085] The position of the adjusted virtual extended screen is set to be within a preset range of a display picture frame of the physical display screen.

[0086] In a possible implementation, the execution module 502 is further configured to set the position of the virtual extended screen to be behind the display frame of the physical display screen and to be located on the non-display side of the physical display screen.

[0087] In one possible implementation, the execution module 502 is further used to adjust the depth value of the pixel point in the display picture frame to be smaller than the depth value of the corresponding pixel point in the virtual extended screen, so as to set the position of the virtual extended screen to be behind the display picture frame of the physical display screen and located on the non-display side of the physical display screen.

[0088] In a possible implementation, the execution module 502 is further configured to receive an adjustment instruction from a user for the menu interface displayed on the virtual extended screen, and adjust the menu interface according to the adjustment instruction, wherein the adjustment includes at least one of the following: moving, scaling;

[0089] Display the adjusted menu interface;

[0090] The adjusted menu interface does not block the target display screen in the display screen frame.

[0091] In a possible implementation, the execution module 502 is further configured to receive a conversion instruction from a user for a target object in a target display screen;

[0092] According to the conversion instruction, the target object is converted into a virtual target object;

[0093] The virtual target object is displayed on the virtual extended screen.

[0094] In a possible implementation, the execution module 502 is further configured to extract files related to the menu interface before displaying the menu interface in the display frame on the virtual extended screen, wherein the files related to the menu interface include at least one of the following: text, image, icon, and style sheet of the menu interface;

[0095] The contents of the related files are rendered to the virtual extended screen based on the Static Mesh component to display the menu screen on the virtual extended screen.

[0096] In summary, using AR devices to display the menu interface outside the game interface can improve the user's visual effect, and make the picture of others watching the scene or watching the live video more realistic, which can improve the immersive experience of players and viewers; there is no need to add a new display screen, which can solve the problems of use in specific scenes and space limitations; some menu interfaces (such as maps, subtitles) can be moved or scaled, thereby improving the gaming experience of players (such as myopic players), and providing more convenient and diverse AR interaction methods.

[0097] The present application embodiment also provides an electronic device 60, such as Figure 6 As shown, it includes: a processor 601, a memory 602, and a program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor, the steps of the method for displaying the menu interface shown in the above embodiment are implemented.

[0098] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a method for displaying a menu interface shown in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0099] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of a method for displaying a menu interface shown in the above method embodiment and can achieve the same technical effect. To avoid repetition, they are not repeated here.

[0100] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0102] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for displaying a menu interface, characterized in that: The method comprises: Acquire real-time captured images including physical display screens based on virtual reality technology; Performing three-dimensional modeling on the physical display screen based on the real-time captured image to obtain a three-dimensional reconstruction model corresponding to the physical display screen; Creating a virtual extended screen based on the three-dimensional reconstructed model, setting the position of the virtual extended screen within a preset range of a display frame of the physical display screen, wherein the virtual extended screen is displayed on the physical display screen; The menu interface in the display picture frame is displayed on the virtual extended screen, and the menu interface is not displayed in the display picture frame.

2. The method according to claim 1, characterized in that: After creating the virtual extended screen based on the three-dimensional reconstruction model, the method further includes: Based on the three-dimensional reconstructed model, aligning the center of the virtual extended screen with the center of the physical display screen; Adjusting the geometric features of the virtual extended screen so that the geometric features of the virtual extended screen are completely consistent with the geometric features of the display picture frame, or making the geometric features of the virtual extended screen proportionally enlarged to the geometric features of the display picture frame, to obtain an adjusted virtual extended screen; Setting the position of the virtual extended screen within a preset range of the display frame of the physical display screen includes: The position of the adjusted virtual extended screen is set within a preset range of the display frame of the physical display screen.

3. The method according to claim 1, characterized in that Setting the position of the virtual extended screen within a preset range of the display frame of the physical display screen includes: The virtual extended screen is positioned behind the display frame of the physical display screen and is located on the non-display side of the physical display screen.

4. The method according to claim 3, characterized in that: The step of setting the position of the virtual extended screen behind the display frame of the physical display screen and located on the non-display side of the physical display screen includes: The depth value of the pixel point in the display picture frame is adjusted to be smaller than the depth value of the corresponding pixel point in the virtual extended screen, so as to set the position of the virtual extended screen behind the display picture frame of the physical display screen and on the non-display side of the physical display screen.

5. The method according to claim 1, characterized in that The step of displaying the menu interface in the display picture frame on the virtual extended screen includes: receiving an adjustment instruction from a user for a menu interface displayed on the virtual extended screen, and adjusting the menu interface according to the adjustment instruction, wherein the adjustment includes at least one of the following: moving, scaling; Displaying the adjusted menu interface; The adjusted menu interface does not block the target display screen in the display screen frame.

6. The method according to claim 5, characterized in that The method further comprises: receiving a conversion instruction from a user for a target object in the target display screen; According to the conversion instruction, convert the target object into a virtual target object; The virtual target object is displayed on the virtual extended screen.

7. The method according to any one of claims 1 to 6, characterized in that Before displaying the menu interface in the display picture frame on the virtual extended screen, the method further includes: Extracting files related to the menu interface, wherein the files related to the menu interface include at least one of the following: text, image, icon, and style sheet of the menu interface; The content of the related file is rendered to the virtual extended screen based on the Static Mesh component, so as to display the menu screen on the virtual extended screen.

8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the method for displaying a menu interface as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the program is executed by the processor, the steps of the method for displaying a menu interface according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that It comprises computer instructions, which, when executed by a processor, implement the steps of the method for displaying a menu interface as described in any one of claims 1-7.