Method and apparatus for searching 2D applications in virtual space

Abstract

The application relates to the technical field of virtual display, and provides a search method and equipment for a 2D application in a virtual space, which renders and displays a user interface of the 2D application on a virtual desktop in a 3D space, copies layout information of an embedded window of an original search bar in the user interface, generates a floating window of an extended search bar independent of the user interface, and associates events of the floating window of the extended search bar with the embedded window of the original search bar, so that the content input in the extended search bar is synchronized to the original search bar. Since the position and size of the floating window in the 3D space change along with the user's line of sight, the position of the window of the virtual input method keyboard also changes along with the user's line of sight, so that the virtual input method is still visible when the user's line of sight changes, and the size of the floating window is adaptively adjusted along with the user's line of sight, the user can clearly determine the content in the original search bar through the content in the extended search bar, and the interactive experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of virtual display, and provides a search method and device for a 2D application in a virtual space. BACKGROUND

[0002] A virtual desktop is an important basic function in an AR / VR product. Through customized expansion of rendering display and interaction, the virtual desktop provides an intuitive and reliable interface display and interaction form for 2D applications in 3D display of the AR / VR product, increases the type of display resources supported by the AR / VR product, and expands the use range of the AR / VR product.

[0003] Currently, when the AR / VR product displays a user interface of a 2D application on a virtual desktop, each function window in the user interface is displayed as a whole. However, the area occupied by a search window in the entire user interface is usually small, and when a user wants to interact through the search window, input is inconvenient.

[0004] To solve the above problem, the related art usually adjusts the size of the user interface of the 2D application displayed on the virtual desktop, or adjusts the distance between the user interface of the 2D application and the eyes of a user, to realize zoom display of the search window. When the content input in the search window is completed, the user interface of the 2D application is restored, the entire interaction process is inconvenient to operate, the virtual display performance of the AR / VR product is not fully utilized, and the interaction experience is poor. SUMMARY

[0005] Embodiments of the present application provide a search method and device for a 2D application in a virtual space, to improve the convenience of interaction with the 2D application in the virtual space, thereby improving user experience.

[0006] In one aspect, the present application provides a search method for a 2D application in a virtual space, comprising:

[0007] starting a 2D application, and rendering and displaying a user interface of the 2D application on a virtual desktop in a 3D space;

[0008] in response to an input operation on an original search window in the user interface, obtaining layout information of an embedded window of the original search window in the user interface and copying the layout information, generating a floating window of an extended search window according to the copied layout information, and changing the floating window following a user's line of sight;

[0009] moving a focus to the extended search window in the floating window, and associating the embedded window with the floating window in terms of events;

[0010] display the virtual input method on the virtual desktop according to the first position of the floating window on the virtual desktop.

[0011] According to the event association relationship, the content input in the extended search bar by using the virtual input method is synchronized into the original search bar.

[0012] According to the content input in the original search bar, search is performed in the 2D application.

[0013] In another aspect, the embodiments of the present application provide a virtual display device, comprising a processor, a memory and a communication interface, the communication interface, the memory and the processor are connected through a bus, the memory stores a computer program, and the processor executes the following operations according to the computer program:

[0014] In response to a start operation of a 2D application, a virtual desktop is created in a 3D space, and a user interface of the 2D application is rendered and displayed on the virtual desktop.

[0015] After receiving an input operation on an original search bar in the user interface through the communication interface, layout information of an embedded window of the original search bar in the user interface is obtained and copied, a floating window of the extended search bar is generated according to the copied layout information, and the floating window changes following the user's line of sight.

[0016] The focus is moved to the extended search bar of the floating window, and the embedded window is associated with the floating window in terms of events.

[0017] The floating window is rendered and displayed on the virtual desktop independently of the user interface, and a window of a keyboard of a virtual input method is determined to be at a second position on the virtual desktop according to a first position of the floating window on the virtual desktop, and the virtual input method is rendered and displayed at the second position on the virtual desktop.

[0018] According to the event association relationship, the content input in the extended search bar by using the virtual input method is synchronized into the original search bar.

[0019] According to the content input in the original search bar, search is performed in the 2D application.

[0020] In another aspect, an embodiment of the present application provides a computer readable storage medium storing computer executable instructions for causing a computer device to perform the method for searching 2D application in virtual space provided by an embodiment of the present application.

[0021] The method and device for searching 2D application in virtual space provided by an embodiment of the present application render and display the user interface of the 2D application on the virtual desktop in 3D space, when receiving the input operation on the original search bar in the user interface, generate the floating window of the extended search bar according to the layout information of the embedded window of the user interface in the original search bar, and render and display the floating window of the extended search bar on the virtual desktop independently of the user interface. Since the floating window can change following the user's line of sight in 3D space, and the position of the window of the virtual input method keyboard on the virtual desktop is determined according to the position of the floating window on the virtual desktop, the virtual input method is still visible when the user's line of sight changes, thereby ensuring the normal search of the 2D application in the virtual space and improving the interactive experience. Meanwhile, the embedded window in the user interface is event-associated with the floating window, so that when the virtual input method is used to input in the extended search bar of the floating window where the current focus is located, the input content can be synchronized to the original search bar in the user interface. Therefore, when the user's line of sight changes, the user can clearly see the input content in the original search bar in the user interface through the input content displayed in the extended search bar, thereby ensuring the accuracy of the input content in the original search bar and further improving the interactive experience. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without any creative labor.

[0023] Figure 1 The scene schematic diagram in which the virtual display device provided by an embodiment of the present application displays the 2D application;

[0024] Figure 2 The flowchart of the method for searching 2D application in virtual space provided by an embodiment of the present application;

[0025] Figure 3 The effect diagram of displaying the user interface of the 2D application on the virtual desktop in 3D space provided by an embodiment of the present application;

[0026] Figure 4A The schematic diagram of the generated floating window of the extended search bar provided by an embodiment of the present application;

[0027] Figure 4B Another extended search bar generated by the embodiment of the present application provides a floating window diagram;

[0028] Figure 5A The rendering display hierarchy of the Activity component in the Android system provided by the embodiment of the present application;

[0029] Figure 5B The rendering display schematic diagram of the Activity component in the Android system provided by the embodiment of the present application;

[0030] Figure 6 The method flowchart of generating the floating window of the extended search bar provided by the embodiment of the present application;

[0031] Figure 7 The effect diagram of the floating window of the extended search bar and the independent display of the 2D application interface provided by the embodiment of the present application;

[0032] Figure 8 The rendering display method flowchart of the floating window of the extended search bar provided by the embodiment of the present application;

[0033] Figure 9A The position adjustment mode diagram of the floating window of the extended search bar provided by the embodiment of the present application;

[0034] Figure 9B Another position adjustment mode diagram of the floating window of the extended search bar provided by the embodiment of the present application;

[0035] Figure 10 The display effect diagram of the user interface, the floating window of the extended search bar and the window of the virtual input method keyboard on the virtual desktop provided by the embodiment of the present application;

[0036] Figure 11 The diagram of adjusting the size of the floating window of the extended search bar on the virtual desktop provided by the embodiment of the present application;

[0037] Figure 12 The synchronization effect diagram of the input content in the extended search bar and the original search bar provided by the embodiment of the present application;

[0038] Figure 13 The search result display effect diagram provided by the embodiment of the present application;

[0039] Figure 14 The update diagram of the floating window of the extended search bar provided by the embodiment of the present application;

[0040] Figure 15 The structure diagram of the virtual display device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0042] In the field of virtual display, most applications in AR / VR and other virtual display devices are 3D applications. To support 2D applications, a virtual desktop function is provided, which is used to display 2D applications.

[0043] like Figure 1 The diagram shown is a scenario of a virtual display device displaying a 2D application according to an embodiment of this application. In this scenario, the virtual display device creates a 2D virtual desktop and displays the user of the 2D application on the virtual desktop. The user interacts with the 2D application in 3D space by operating the controller of the virtual display device.

[0044] Currently, when displaying the user interface of a 2D application on a virtual desktop, all the functional windows in the user interface are displayed as a whole. However, the search bar window usually occupies a small area of ​​the user interface. When users want to interact through the search bar, it causes problems such as small font size and unclear input content, making searching very inconvenient.

[0045] The related technology achieves scaling of the search bar by adjusting the size of the user interface of the 2D application displayed on the virtual desktop, or by adjusting the distance between the virtual desktop used to display the 2D application's user interface and the user's eye. Once the content in the search bar is entered, the 2D application's user interface is restored. This search method processes the entire user interface of the 2D application, failing to fully utilize the virtual display performance of AR / VR and other virtual display devices. Furthermore, the entire interaction process is inconvenient and provides a poor user experience.

[0046] When a user inputs characters into a search bar using an application-embedded input method, the characters are displayed in a small window, which makes it difficult for the user to see the input characters and find errors in the input characters in time, resulting in incorrect search results and affecting the user's interactive experience. Therefore, in some mainstream AR / VR products, a virtual input method with an input box function is supported for searching. The virtual input method constructs a blank input box above the input keyboard to display the content input into the search bar, and the content in the input box is synchronized to the search bar of the 2D application, thereby implementing the search method of the 2D application displayed on the virtual desktop.

[0047] However, the above method mainly solves the problem of unclear input characters in the search bar of the user interface of the 2D application displayed on the virtual desktop, but still has the following two problems: 1) Since the position of the user interface of the 2D application is fixed relative to the 3D space, the position of the window of the search bar in the user interface of the 2D application on the virtual desktop is also fixed and cannot rotate with the head rotation. The position of the window of the virtual input method in the 3D space is determined according to the position of the window of the search bar in the user interface of the 2D application, and therefore the window of the virtual input method (including the keyboard and the input box) can only be displayed at a fixed position on the virtual desktop. When the head rotates and the user's line of sight deviates from the window of the search bar in the user interface, the user may not be able to see the virtual input method and the corresponding input box, resulting in a failed search; 2) When using the virtual input method to input content into the search bar of the 2D application, the user's visual angle moves (such as moving downward, upward, leftward, rightward, etc.), so that the focus is focused on the keyboard or the input box of the virtual input method. In this way, the user cannot completely determine whether the content input by the virtual input method has been input into the search bar of the 2D application.

[0048] In view of this, the embodiments of the present application provide a search method for a 2D application in a virtual space. The method can dynamically extract the window of the search bar in the user interface of the 2D application displayed on the virtual desktop and independently display the window of the search bar on the virtual desktop, thereby fully utilizing the virtual display performance of the AR / VR virtual display device. Since the extracted window of the search bar is independent of the entire user interface of the 2D application, the extracted window of the search bar can change flexibly in the 3D space according to the change of the user's line of sight, and at the same time, the window of the virtual input method (including the keyboard and the input box) in the 3D space also changes according to the change of the window of the search bar, making it more convenient for the user to input search content and greatly improving the user's interactive experience with the 2D application in the virtual space.

[0049] Referring to Figure 2A flowchart of a search method of a 2D application in a virtual space provided by an embodiment of the present application is executed by a virtual display device implementing AR / VR technology, and mainly includes the following steps:

[0050] S201: Start a 2D application, and render and display a user interface of the 2D application on a virtual desktop in a 3D space.

[0051] The virtual desktop in the virtual display device is equivalent to a two-dimensional screen in the 3D space, and can be used as a user interface display carrier of the 2D application. Therefore, the virtual display device supports displaying resources of the 2D application. When a user wants to view the resources of the 2D application on the virtual display device, the user interacts with the virtual display device through a handle or a gesture, starts the 2D application on the virtual display device, and the virtual display device renders and displays a user interface of the started 2D application on a virtual screen.

[0052] As shown in Figure 3 , an effect diagram of displaying a user interface of a 2D application on a virtual desktop in a 3D space provided by an embodiment of the present application is provided. The user interface 300 includes multiple function windows, such as an embedded window 301 of a search function, an embedded window 302 of a navigation bar, and an embedded window 303 of resource display, and the like. The user can input content into an original search bar 3011 of the user interface 300 to interact with the 2D application in the virtual space.

[0053] S202: In response to an input operation on the original search bar in the user interface, obtain layout information of an embedded window of the original search bar in the user interface, generate a floating window of the original search bar according to the copied layout information, and change the floating window following the user's line of sight.

[0054] When the user wants to search for resources in the 2D application, the user clicks the original search bar in the user interface of the 2D application (for example, 3011 in Figure 3 ) through an interactive manner, so as to position the focus in the original search bar to input the resources to be searched. After the virtual display device receives the input operation of the user on the original search bar in the user interface triggered by the user, the virtual display device obtains and copies the layout information of the embedded window of the original search bar in the user interface by querying the focus.

[0055] Optionally, in some embodiments, the obtained layout information includes at least the size of the original search bar in the user interface (such as parameters such as shape, length, and width), the texture of the original search bar (such as parameters such as the color of the original search bar border and the color of the original search bar background), the objects contained in the original search bar (such as text and images), the size of the objects (such as parameters such as the font size of the text and the size of the image), the texture of the objects (such as parameters such as the color of the text and the color of the image), and the positional relationship between the objects and the original search bar (such as the distance between the text and the left border of the original search bar, and the distance between the image and the left border of the original search bar).

[0056] Furthermore, after obtaining the layout information of the original search bar embedded in the user interface window, a floating window of the extended search bar independent of the user interface is generated based on the obtained layout information.

[0057] For example, such as Figure 4A The diagram shown is a schematic of a floating window generated based on the layout information of the embedded window of the original search bar in the copied user interface, according to an embodiment of this application. The floating window 400 includes an extended search bar 401 and objects 402 contained in the extended search bar. Since the floating window is generated based on the layout information of the embedded window of the copied original search bar, the data of the extended search bar 401 (such as size, texture, etc.) and the data of the objects 402 contained in the extended search bar 401 (such as size, texture, position, etc.) are consistent with the embedded window of the original search bar 3011 in the user interface.

[0058] Optionally, in order to ensure the integrity of the floating window of the extended search bar, in some embodiments, the layout information obtained and copied also includes the size of the associated controls (such as images and symbols) of the original search bar, the texture (such as color) of the associated controls, and the positional relationship (such as orientation and distance) between the associated controls and the original search bar.

[0059] For example, such as Figure 4B The diagram shown is a schematic of another floating window generated based on the layout information of the embedded window of the original search bar in the copied user interface, according to an embodiment of this application. The floating window 400 includes an extended search bar 401, an object 402 contained in the extended search bar, and an associated control 403 of the extended search bar. The data of the extended search bar 401 (such as size, texture, etc.), the data of the object 402 contained in the extended search bar 401 (such as size, texture, position, etc.), and the data of the associated control 403 of the extended search bar are consistent with the embedded window of the original search bar 3011 in the user interface.

[0060] In the embodiments of the present application, by acquiring and copying the layout information of the embedded window of the original search box in the user interface, a floating window of the extended search bar in the 3D space independent of the user interface is generated, so that the floating window can change according to the change of the user's line of sight, facilitating the user to input, thereby improving the interactive experience with the 2D application in the 3D space.

[0061] In the embodiments of the present application, the generation of the floating window of the extended search bar can be rendered and displayed by using the components of the virtual display device system.

[0062] Taking the Android system as an example of the system of the virtual display device, the floating window of the extended search bar can be drawn based on the acquired layout information by using the Activity component.

[0063] The Activity component is one of the four most basic and most commonly used components in the Android system, and as a carrier of the application program, it represents a complete user interface and provides a window to draw various views. When the Activity is started, a view is set by calling the setContentView method, and this view is the interface for interacting with the user and can handle events for interacting with the user.

[0064] In Android system, the interface that users see is displayed based on Window (that is, window). Window is an abstract concept from the perspective of Window Manager Service (system window management service), and all Views in Android system are presented through Window, so there is Window wherever there is View. Activity is the outermost container of the entire interface, each Activity corresponds to an abstract PhoneWindow (mobile screen), each PhoneWindow corresponds to a DecorView (decorated view) and a ViewRootImpl (view directory corresponding class), PhoneWindow is the most basic window system in Android system, and acts as a medium between Activity and DecorView. DecorView is the top-level View, which generally contains a vertical LinearLayout (linear layout) inside, and there are two parts above and below in this LinearLayout, the upper part is TitleView (title interface), and the lower part is ContentView (content interface). After PhoneWindow is created, setWindowManager is called to set the Window Manager (window management class) that manages Window. ViewRootImpl establishes the connection between DecorView and Window.

[0065] The rendering display hierarchy of Activity component is as shown in Figure 5A , in which PhoneWindow is the container of the outermost interface, DecorView is the root node of the interface, LinearLayout is the linear layout below DecorView, ViewStub (remaining view) is used to load the content of the interface, FrameLayout is the framework layout, and setContentView(...) is used to display the real view content.

[0066] Generally, View is rendered by Android system parsing the layout file of Activity component, and Android system can also copy a layout file and render it in other windows, as shown in Figure 5B , mView is rendered based on part of the data in the layout file of the copied PhoneWindow, and the input control (EditText) in Android system is used to obtain user interaction data, and View events and logic are processed according to the interaction data.

[0067] In the embodiment of the present application, the layout file of the Activity component of the user interface is parsed to obtain the layout information of the embedded view of the original search bar in the user interface and copy the layout information. Based on the copied layout information, a floating window is generated and rendered to obtain a floating view embedded in the original search bar in the user interface, so as to realize the extended display of the search bar. Figure 1

[0068] The specific implementation process is described below with reference to Figure 6 , mainly including the following steps:

[0069] S601: Obtain the layout file of the Activity component corresponding to the user interface of the opened 2D application.

[0070] S602: Parse the layout file to obtain the layout information of each control in the user interface.

[0071] Each control includes a search bar, a picture, text, resources, etc., and the layout information concent includes the size, texture, position of each control, and the structural relationship between the controls.

[0072] S603: According to the current focus position, extract the data of the original search bar control and the associated control in the layout information in the user interface.

[0073] The association relationship between the original search bar control and other controls in the user interface can be determined by the structural relationship between the controls in the layout information.

[0074] For example, other controls in the same horizontal direction as the original search bar control are taken as the associated controls of the original search bar.

[0075] By extracting the data of the controls associated with the original search bar in the embedded view window, the integrity of the window is ensured.

[0076] S604: Construct a new window, and render the window according to the extracted data of the original search bar control and the associated control to obtain the floating window of the extended search bar.

[0077] S203: Move the focus to the extended search bar in the floating window, and associate the embedded window with the floating window.

[0078] After the floating window of the extended search bar is generated, the focus is moved from the original search bar in the user interface to the extended search bar in the floating window, so as to realize the input operation of the extended search bar.

[0079] ​Since the floating window is independent of the user interface of the 2D application, in order to realize the operation of the 2D application through the operation of the floating window, the embodiment of the application associates the embedded window with the floating window in events, so that the content input in the extended search bar is input in the original search bar synchronously, thereby realizing the association of the floating window with the user interface, and realizing the interaction with the 2D application through the interaction with the floating window.

[0080] S204: rendering and displaying the floating window independently of the user interface on the virtual desktop, and determining the second position of the window of the keyboard of the virtual input method on the virtual desktop according to the first position of the floating window on the virtual desktop, and rendering and displaying the window of the keyboard of the virtual input method at the second position on the virtual desktop.

[0081] The virtual display desktop is virtual in the 3D space and can be extended, thus, in the embodiment of the application, the virtual display performance of the virtual display device is fully utilized to render and display the floating window of the extended search bar independently of the user interface of the 2D application on the virtual desktop.

[0082] As shown in Figure 7 , it is an effect diagram of the independent display of the floating window of the extended search bar and the 2D application interface provided by the embodiment of the application, in which Figure 7 , the floating window 400 of the extended search bar is consistent with the content of the embedded window 301 of the original search bar in the user interface, and the two are displayed at different positions on the virtual desktop in the 3D space.

[0083] Optionally, the first position of the floating window of the extended search bar on the virtual desktop is determined according to the user's line of sight in the 3D space and the position of the user interface.

[0084] For example, the center point of the user interface on the virtual desktop and the center of the point of view of the user's line of sight on the virtual desktop can be set as the first position of the floating window of the extended search bar on the virtual desktop, and the floating window of the extended search bar is rendered and displayed at the position.

[0085] In a specific implementation, the rendering and display process of the floating window of the extended search bar is as shown in Figure 8 , mainly including the following steps:

[0086] S801: starting the 2D application and entering the virtual desktop.

[0087] S802: displaying the user interface of the started 2D application on the virtual desktop.

[0088] S803: when the user interface contains the original search bar, extracting the layout information of the embedded window of the original search bar in the user interface.

[0089] S804: Get the position 1 of the user interface on the virtual desktop, and the position 2 of the current user's gaze.

[0090] S805: Determine the first position of the small window based on position 1 and position 2.

[0091] S806: Render the small window at the first position based on the extracted layout information to obtain and display the floating window of the extended search bar.

[0092] In the embodiments of this application, since the first position of the floating window of the extended search bar on the virtual desktop is based on the user's line of sight, the first position of the floating window of the extended search bar on the virtual desktop will also change when the user's line of sight in 3D space changes during the search process.

[0093] In one optional implementation, when the user's gaze moves left or right, the first position of the floating window of the extended search bar on the virtual desktop is adjusted by: determining the angle between the user's gaze and the vertical center line of the user interface, and adjusting the floating window within the field of view of the first angle. Here, the user's gaze is the center line of the horizontal field of view of the human eye.

[0094] like Figure 9A The image shows a diagram illustrating how to adjust the position of the extended search bar's floating window on the virtual desktop. Figure 9A As shown, at the first moment, the user's line of sight (in) Figure 9A (represented by a thin dotted line in the middle) and the vertical center line of the user interface (in...) Figure 9A The first included angle (represented by a double-dotted line) is the FOV. At this point, the floating window of the extended search bar is located at position 3 on the virtual desktop. Figure 9A (represented by a dashed line), which is the center of the first included angle FOV. The user's line of sight moves from left to right; at the second moment, the user's line of sight (in...) is... Figure 9A The first angle between the extended search bar's floating window (represented by a thick dotted line) and the vertical center line of the user interface is FOV′. At this point, the extended search bar's floating window is located at position 4 on the virtual desktop. Figure 9A (Represented by a solid line in the middle), which is the center of the first included angle FOV′.

[0095] In one optional implementation, when the user's gaze moves up and down, the first position of the floating window of the extended search bar on the virtual desktop is adjusted by: determining a second angle between the user's gaze and the horizontal center line of the user interface, and adjusting the floating window within the field of view of the second angle. Here, the user's gaze is the center line of the vertical field of view of the human eye.

[0096] like Figure 9B The image shows a diagram illustrating how the position of the floating window for another extended search bar on the virtual desktop is adjusted.Figure 9B As shown, at a first time, the second angle between the user's line of sight (indicated by a thin single-dot line in Figure 9B ) and the horizontal center line of the user interface (indicated by a thin double-dot line in Figure 9B ) is FOV, at which time the floating window of the extended search bar is located at position 5 (indicated by a dashed line in Figure 9B ) on the virtual desktop, which is the center of the second angle FOV. The user's line of sight moves from bottom to top, at a second time, the second angle between the user's line of sight (indicated by a thick single-dot line in Figure 9B ) and the horizontal center line of the user interface is FOV', at which time the floating window of the extended search bar is located at position 6 (indicated by a solid line in Figure 9B ) on the virtual desktop, which is the center of the second angle FOV'.

[0097] In the embodiments of the present application, it is considered that when the user's line of sight changes, if the user's line of sight deviates far from the user interface, the floating window of the extended search bar and the user interface can not be simultaneously within the range of the field of view of the human eye, therefore, when the first angle between the user's line of sight and the vertical center line of the user interface is greater than a preset angle threshold, or when the second angle between the user's line of sight and the horizontal center line of the user interface is greater than a preset angle threshold, the floating window of the extended search bar is hidden. Optionally, the preset angle threshold is 60 degrees.

[0098] When the focus moves to the extended search bar of the floating window, the virtual input method is started, the window of the keyboard of the virtual input method is at a second position on the virtual desktop, which is determined according to the first position of the floating window of the extended search bar on the virtual desktop, and the window of the keyboard of the virtual input method is rendered and displayed at the second position on the virtual desktop.

[0099] For example, the second position of the window of the keyboard of the virtual input method on the virtual desktop is directly below the first position of the floating window of the extended search bar on the virtual desktop, and is separated from the first position by a preset distance.

[0100] As Figure 10 shown, the display effect diagram of the user interface, the floating window of the extended search bar and the window of the keyboard of the virtual input method on the virtual desktop provided by the embodiments of the present application, wherein the floating window 400 of the extended search bar is independent of and consistent with the embedded window 301 of the original search bar in the user interface, the window 500 of the keyboard of the virtual input method for inputting content into the extended search bar where the focus is located is displayed adjacent to below the floating window 400.

[0101] In the embodiments of the present application, when the user's line of sight changes during the search interaction with the 2D application in the 3D space, the first position of the floating window of the extended search bar on the virtual desktop changes with the change of the user's line of sight. Since the second position of the window of the virtual input method keyboard on the virtual desktop is determined according to the first position of the floating window of the extended search bar on the virtual desktop, the second position of the window of the virtual input method keyboard on the virtual desktop also changes with the change of the user's line of sight, maintaining the visibility of the virtual input method, thereby ensuring the normal search of the 2D application in the virtual space and improving the interactive experience.

[0102] In the embodiments of the present application, in order to ensure that the user can see the complete data of the user interface, the extended search bar and the virtual input method, the window of the user interface, the window of the virtual input method keyboard and the window of the extended search bar do not overlap with each other in the virtual desktop.

[0103] Considering that the user's line of sight moves forward and backward in addition to up and down and left and right in the 3D space, in some embodiments, the change of the floating window of the extended search bar on the virtual desktop also includes following the forward and backward movement of the user's line of sight. When the user's line of sight moves forward and backward, the distance between the eyes and the user interface (i.e. the distance between the eyes and the virtual desktop) also changes. According to the principle of far small and near large, the size of the floating window of the extended search bar on the virtual desktop can be adjusted.

[0104] Specifically, when the user's line of sight moves forward and backward, the distance difference between the eyes and the user interface before and after the movement is determined, and the floating window of the extended search bar displayed on the virtual desktop is scaled and adjusted according to the distance difference.

[0105] For example, before the user's line of sight moves forward and backward, the distance between the eyes and the user interface is d1, and after the user's line of sight moves forward and backward, the distance between the eyes and the user interface is d2. The distance difference between the eyes and the user interface before and after the movement of the user's line of sight is Δd = d1 - d2. At this time, the scaling ratio for adjusting the floating window of the extended search bar displayed on the virtual desktop is ratio = 1 + Δd / d1.

[0106] In some embodiments, when the floating window of the extended search bar displayed on the virtual desktop is scaled and adjusted in proportion, at least one of the window of the user interface and the window of the virtual input method keyboard on the virtual desktop can also be scaled and adjusted in the same proportion.

[0107] As shown in FIG. 1, a schematic diagram for adjusting the size of the floating window of the extended search bar on the virtual desktop is provided in the embodiments of the present application. Figure 11 Figure 11 ​Figure (a) shows the display size of the user interface and the floating window of the extended search bar on the virtual desktop before the user's gaze moves forward or backward. The size of the user interface and the floating window of the extended search bar on the virtual desktop changes as the user's gaze moves backward. Figure 11 As shown in (b) above. By comparison, it can be seen that when the user's gaze moves backward, the distance between the human eye and the user interface is greater. Based on the principle that things appear smaller when they are farther away and larger when they are closer, the user interface displayed on the virtual desktop and the floating window of the extended search bar are shrunk and adjusted.

[0108] In some embodiments, when the distance between the user's eye and the user interface is greater than a preset distance threshold after moving forward and backward, the floating window of the extended search bar is hidden, and the keyboard window of the virtual input method is also hidden.

[0109] In this embodiment, the extended capabilities of the virtual desktop in the virtual display device are utilized to generate a floating window for an extended search bar based on the layout information of the embedded window of the original search bar in the user interface. This floating window is rendered and displayed independently of the user interface on the virtual desktop. The focus is then moved to the extended search bar to activate the virtual input method. Since the size and position of the floating window in 3D space can change with the user's gaze, and the position of the virtual input method's keyboard window on the virtual desktop is determined by the position of the floating window, the keyboard window remains visible even when the user's gaze changes. This ensures normal searching of 2D applications in the virtual space and improves the interactive experience. Furthermore, because the size of the floating window of the extended search bar changes according to the user's gaze movement, it allows for scaling, ensuring the user can clearly see the content entered into the extended search bar and improving search accuracy.

[0110] S205: Based on the event association, the content entered in the extended search bar using the virtual input method will be synchronized to the original search bar.

[0111] This application does not impose any restrictions on the synchronization method of the input content in the extended search bar and the original search bar. For example, the input content in the extended search bar can be synchronized to the original search bar word by word in real time, or it can be synchronized to the original search bar all at once after the input content in the extended search bar is completed.

[0112] like Figure 12 The image shown illustrates the synchronized effect of input content in the expanded search bar and the original search bar provided in this embodiment of the application. Figure 12 In this process, the embedded window of the original search bar in the user interface of the 2D application is extracted, and a floating window of the extended search bar is generated and displayed independently. When "a good life" is entered into the extended search bar using the virtual input method keyboard, "a good life" will also be entered into the original search bar of the user interface simultaneously.

[0113] In this embodiment, after the floating window of the extended search bar is generated, the focus moves from the original search bar in the 2D application's user interface to the extended search bar, and the virtual input method is activated. Based on the current focus position, the content entered using the virtual input method will enter the extended search bar of the floating window. Since the floating window of the extended search bar has been associated with the embedded window of the original search bar in the user interface through an event association, the content entered into the extended search bar will be synchronously entered into the original search bar of the user interface based on the event association relationship. This enables operation on the floating window, which is equivalent to operation on the 2D application. Because the floating window can flexibly display itself on the virtual desktop according to the user's line of sight and is associated with the embedded window of the original search bar in the user interface through an event association, when the user's line of sight changes, viewing the content entered in the extended search bar of the floating window ensures that the content entered in the original search bar of the user interface is also present, guaranteeing the clarity and accuracy of the input content. Compared to directly searching the original search bar in the 2D application, this improves the experience of interacting with 2D applications in 3D space.

[0114] It should be noted that in this embodiment of the application, after the input is completed, the floating window of the extended search bar and the window of the virtual input method keyboard are no longer displayed on the virtual desktop.

[0115] S206: Search within the 2D application based on the content entered in the original search bar.

[0116] The content entered in the extended search bar using the virtual input method is synchronized to the original search bar of the 2D application's user interface through event association. Therefore, the 2D application can query the resources that the user wants to watch based on the content entered in the original search bar and return them to the user for viewing through the user interface.

[0117] like Figure 13 The image shown is a screenshot of the search results display effect provided in this application embodiment. When the user enters "a better life" in the extended search bar using the virtual input method keyboard, the search bar is synchronized to the original search bar of the user interface. After clicking the search button on the user interface, the 2D application returns the resources for "a better life".

[0118] The method for searching 2D application in virtual space provided by the embodiments of the present application renders and displays the user interface of the 2D application on a virtual desktop in 3D space, and when an input operation on the original search bar in the user interface is received, a floating window of the extended search bar is generated according to the layout information of the embedded window of the original search bar in the user interface, and is rendered and displayed on the virtual desktop independently of the user interface. Since the floating window can change following the user's line of sight in 3D space, and the position of the window of the virtual input method keyboard on the virtual desktop is determined according to the position of the floating window on the virtual desktop, the virtual input method is still visible when the user's line of sight changes, thereby ensuring normal search of the 2D application in the virtual space and improving the interactive experience. Meanwhile, the embedded window in the user interface and the floating window are event-associated, so that when the virtual input method is used to input in the extended search bar of the floating window where the current focus is located, the input content can be synchronized to the original search bar in the user interface. Therefore, when the user's line of sight changes, the user can clearly see the input content in the original search bar in the user interface through the input content displayed in the extended search bar, thereby ensuring the accuracy of the input content in the original search bar and further improving the interactive experience.

[0119] In some embodiments, when the embedded window of the original search bar in the user interface changes during the search process, the layout information is updated, so that the generated floating window of the extended search bar is also refreshed synchronously.

[0120] For example, as shown in Figure 14 When the text in the embedded window of the original search bar in the user interface changes from "easy moment" to "seeking the voice of the heavens", the text in the extended search bar in the floating window also changes from "easy moment" to "seeking the voice of the heavens".

[0121] Based on the same technical concept, the embodiments of the present application provide a virtual display device which can implement the steps of the method for searching 2D application in virtual space in the above embodiments and achieve the same technical effects.

[0122] Referring to Figure 15 , the virtual display device comprises a processor 1501, a memory 1502 and a communication interface 1503, the communication interface 1503, the memory 1502 and the processor 1501 are connected through a bus 1504, the memory 1502 stores a computer program, and the processor 1501 executes the following operations according to the computer program:

[0123] in response to a start operation of a 2D application, creating a virtual desktop in 3D space and rendering and displaying a user interface of the 2D application on the virtual desktop;

[0124] By the communication interface 1503, after receiving the input operation on the original search bar in the user interface, the layout information of the embedded window of the original search bar in the user interface is acquired and copied, and according to the copied layout information, the floating window of the expanded search bar is generated, which changes following the user's line of sight;

[0125] The focus is moved to the expanded search bar of the floating window, and the embedded window is event-associated with the floating window;

[0126] The floating window is independently rendered and displayed on the virtual desktop, and according to the first position of the floating window on the virtual desktop, the window of the keyboard of the virtual input method is determined at the second position of the virtual desktop, and the virtual input method is rendered and displayed at the second position of the virtual desktop;

[0127] According to the event association relationship, the content input in the expanded search bar by the virtual input method is synchronized to the original search bar;

[0128] According to the content input in the original search bar, search is performed in the 2D search space.

[0129] Optionally, the processor 1501 implements the manner in which the floating window changes following the user's line of sight at least includes:

[0130] When the user's line of sight moves left and right, a first included angle between the user's line of sight and the vertical center line of the user interface is determined, and the floating window is adjusted within the field of view range of the first included angle;

[0131] When the user's line of sight moves up and down, a second included angle between the user's line of sight and the horizontal center line of the user interface is determined, and the floating window is adjusted within the field of view range of the second included angle.

[0132] Optionally, when the first included angle is greater than a preset included angle threshold, the floating window of the expanded search bar is hidden;

[0133] When the second included angle is greater than a preset included angle threshold, the floating window of the expanded search bar is hidden.

[0134] Optionally, the manner in which the floating window changes following the user's line of sight at least further includes:

[0135] When the user's line of sight moves forward and backward, a distance difference value between the eyes before and after the movement and the user interface is determined, and according to the distance difference value, the floating window of the expanded search bar displayed on the virtual desktop is scaled and adjusted.

[0136] Optionally, when the distance between the human eye and the user interface is greater than a preset distance threshold, the floating window of the extended search bar is hidden.

[0137] Optionally, the layout information at least includes: a size of the original search bar, a texture of the original search bar, an object contained in the original search bar, a size of the object, a texture of the object, and a positional relationship between the object and the original search bar.

[0138] Optionally, the layout information further includes: a size of an associated control of the original search bar, a texture of the associated control, and a structural relationship between the associated control and the original search bar.

[0139] Optionally, the user interface, the window of the keyboard of the virtual input method, and the floating window of the extended search bar do not overlap with each other in the virtual desktop.

[0140] It should be noted that, Figure 15 This is only an example, and the virtual display device necessary for the virtual display device to execute the steps of the search method for the 2D application in the virtual space provided by the embodiments of the present application is shown. The virtual display device not shown also includes a speaker, a receiver, a camera, a power supply, a display screen, left and right eye lenses, and other conventional AR / VR product devices.

[0141] The processor involved in the embodiments of the present application Figure 15 may be a central processing unit (CPU), a general-purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0142] The embodiments of the present application also provide a computer-readable storage medium for storing some instructions, which can complete the search method for the 2D application in the virtual space in the foregoing embodiments when executed.

[0143] The embodiments of the present application also provide a computer program product for storing a computer program, which is used to execute the search method for the 2D application in the virtual space in the foregoing embodiments.

[0144] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one embodiment, the present application can be implemented in software and can be stored on a computer readable medium, which can include random access memory (RAM), read only memory (ROM), magnetic disk or optical disk, or the like. The software implementation can comprise one or more computer program components embodied on one or more computer readable medium(s).

[0145] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0146] These computer program instructions can also be stored in a computer readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0148] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for searching 2D applications in a virtual space, characterized by, The method comprises: opening a 2D application, and rendering and displaying a user interface of the 2D application on a virtual desktop in a 3D space; in response to an input operation on an original search bar in the user interface, obtaining layout information of an embedded window of the original search bar in the user interface and copying the layout information, generating a floating window of an extended search bar according to the copied layout information, and changing the floating window following a user's line of sight; moving a focus to the extended search bar in the floating window, and associating the embedded window with the floating window in terms of events; rendering and displaying the floating window of the extended search bar on the virtual desktop independently of the user interface, and determining a window of a keyboard of a virtual input method in a second position on the virtual desktop according to a first position of the floating window on the virtual desktop, and rendering and displaying the virtual input method in the second position on the virtual desktop; synchronizing content input in the extended search bar in the virtual input method to the original search bar according to the event association relationship; and searching in the 2D application according to the content input in the original search bar.

2. The method of claim 1, wherein, The manner in which the floating window changes following the user's line of sight at least comprises: when the user's line of sight moves left and right, determining a first included angle between the user's line of sight and a vertical center line of the user interface, and adjusting the floating window within a visual range of the first included angle; when the user's line of sight moves up and down, determining a second included angle between the user's line of sight and a horizontal center line of the user interface, and adjusting the floating window within a visual range of the second included angle.

3. The method of claim 2, wherein, when the first included angle is greater than a preset included angle threshold, hiding the floating window of the extended search bar; when the second included angle is greater than a preset included angle threshold, hiding the floating window of the extended search bar.

4. The method of claim 1, wherein, The manner in which the floating window changes following the user's line of sight at least further comprises: when the user's line of sight moves forward and backward, determining a distance difference between the human eyes before and after the movement and the user interface, and adjusting the floating window of the extended search bar displayed on the virtual desktop according to the distance difference.

5. The method of claim 4, wherein, when the distance between the human eyes and the user interface is greater than a preset distance threshold, hiding the floating window of the extended search bar.

6. The method of any one of claims 1-5, wherein, The layout information at least comprises: a size of the original search bar, a texture of the original search bar, an object contained in the original search bar, a size of the object, a texture of the object, and a positional relationship between the object and the original search bar.

7. The method of claim 6, wherein, The layout information further comprises: a size of an associated control of the original search bar, a texture of the associated control, and a structural relationship between the associated control and the original search bar.

8. The method of any one of claims 1-5, wherein, The user interface, the window of the keyboard of the virtual input method, and the floating window of the extended search bar do not overlap with each other in the virtual desktop.

9. A virtual display device, comprising: The device comprises a processor, a memory, and a communication interface, the communication interface, the memory, and the processor are connected through a bus, the memory stores a computer program, and the processor executes the following operations according to the computer program: In response to a start operation of a 2D application, a virtual desktop is created in a 3D space, and a user interface of the 2D application is rendered and displayed on the virtual desktop; After receiving an input operation on an original search bar in the user interface through the communication interface, layout information of an embedded window of the original search bar in the user interface is obtained and copied, a floating window of an extended search bar is generated according to the copied layout information, and the floating window changes following the user's line of sight; The focus is moved to the extended search bar of the floating window, and the embedded window is event-associated with the floating window; The floating window is rendered and displayed on the virtual desktop independently of the user interface, and a window of a virtual input method keyboard is determined at a second position on the virtual desktop according to a first position of the floating window on the virtual desktop, and the virtual input method is rendered and displayed at the second position on the virtual desktop; According to the event association relationship, the content input in the extended search bar by using the virtual input method is synchronized to the original search bar; The content input in the original search bar is searched in the 2D application.

10. The virtual display device of claim 9, wherein, The floating window changes following the user's line of sight in at least the following ways: When the user's line of sight moves left and right, a first included angle between the user's line of sight and a vertical center line of the user interface is determined, and the floating window is adjusted within a visual range of the first included angle; When the user's line of sight moves up and down, a second included angle between the user's line of sight and a horizontal center line of the user interface is determined, and the floating window is adjusted within a visual range of the second included angle.

Images