A cross-dimensional interaction method, device and equipment for hybrid applications

Abstract

The present specification relates to the field of human-computer interaction technology, and in particular to a cross-dimensional interaction method, device and equipment for hybrid applications. The hybrid application includes content of multiple different dimensions, and the method includes: receiving an operation instruction from a user for a first content of a first dimension in the hybrid application; processing the first content according to the operation instruction; obtaining a second content of a second dimension according to an interaction logic based on the processed first content; and displaying the second content of the second dimension in the hybrid application. Through the embodiments of the present specification, a hybrid application is constructed based on content of multiple different dimensions, and the hybrid application can have content of multiple different dimensions, and cross-dimensional interaction of content of different dimensions in the hybrid application is achieved, providing users with a better user experience and solving the problem that cross-dimensional interaction of hybrid applications cannot be performed in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a cross-dimensional interaction method, device and equipment for hybrid applications. Background Art

[0002] In the current multi-tasking scenario of three-dimensional space, the dimensions of the content rendered by each application are two-dimensional or three-dimensional. Due to the limitations of the display characteristics of each application, cross-dimensional interaction of hybrid applications is not possible.

[0003] There is an urgent need for a cross-dimensional interaction method for hybrid applications, so as to solve the problem that cross-dimensional interaction of hybrid applications cannot be performed in the prior art. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the embodiments of this specification provide a cross-dimensional interaction method, device and equipment for hybrid applications, which realize the construction of a hybrid application based on content of multiple different dimensions. The hybrid application can have multiple different dimensional contents, and cross-dimensional interaction of different dimensional contents in the hybrid application is realized, providing users with a better user experience and solving the problem that cross-dimensional interaction of hybrid applications cannot be performed in the prior art.

[0005] In order to solve any of the above technical problems, the specific technical solutions of this specification are as follows:

[0006] On the one hand, an embodiment of the present specification provides a cross-dimensional interaction method of a hybrid application, wherein the hybrid application includes content of multiple different dimensions, and the method includes:

[0007] receiving an operation instruction from a user for a first content of a first dimension in the hybrid application;

[0008] Processing the first content according to the operation instruction;

[0009] Based on the processed first content, obtaining second content of a second dimension according to interaction logic;

[0010] The second content of the second dimension is presented in the hybrid application.

[0011] Furthermore, the contents of different dimensions come from different content providers, and the contents of each dimension are rendered by the corresponding content provider;

[0012] Before receiving an operation instruction from a user for the first content of the first dimension in the hybrid application, the method further includes:

[0013] Receiving a first rendering picture corresponding to each dimensional content sent by each content provider;

[0014] At least two of the first rendering images are displayed in the hybrid application.

[0015] Furthermore, the method further includes: providing the user's observation angle information for each dimensional content to each content provider, so that each content provider renders the content of its own dimension according to the observation angle information to obtain the first rendering screen of each content provider.

[0016] Furthermore, based on the processed first content, obtaining the second content of the second dimension according to the interaction logic further includes:

[0017] Sending the processing result of the first content to a content provider corresponding to the second dimension of the operation instruction, so that the content provider processes the model of the second content stored in itself according to the processing result to obtain a second rendering picture of the second content;

[0018] The second rendered picture sent by the content provider is received, and the second rendered picture is displayed in the hybrid application.

[0019] Further, the interaction logic includes a corresponding relationship between the first content and the second content;

[0020] Processing the first content according to the operation instruction further includes:

[0021] Execute the interaction logic according to the operation instruction to obtain an interaction result, wherein the interaction result at least includes the attributes and posture of the second content after the interaction;

[0022] The interaction result is used as the processing result.

[0023] Further, after displaying the received first rendering picture in the hybrid application, the method includes:

[0024] For content of one dimension, the user's operation instructions for the content are converted according to the operation mode of the content provider corresponding to the content, and the converted operation instructions are sent to the content provider, so that the content provider performs corresponding logical processing on the content according to the converted operation instructions and updates the first rendering screen.

[0025] Furthermore, after converting the user's operation instruction for the content according to the operation mode of the content provider corresponding to the content, the method further includes:

[0026] The converted operation instruction is transformed from the world coordinates of the hybrid application to the local coordinate system of the content provider, and the transformed operation instruction is sent to the content provider.

[0027] Furthermore, contents of different dimensions are displayed in multiple windows in the hybrid application respectively.

[0028] Furthermore, the contents of different dimensions run on different hardware or system platforms, and the contents of each dimension constitute a complete application to complete a unified task.

[0029] Furthermore, interaction information between the first dimensional content and the second dimensional content is displayed on the same device.

[0030] On the other hand, an embodiment of the present specification further provides a cross-dimensional interaction device for a hybrid application, wherein the hybrid application includes content of multiple different dimensions, and the device includes:

[0031] An operation instruction receiving unit, configured to receive an operation instruction from a user for a first content of a first dimension in the hybrid application;

[0032] a logic processing unit, configured to process the first content according to the operation instruction;

[0033] A content generating unit, configured to obtain second content of a second dimension according to an interaction logic based on the processed first content;

[0034] The content display unit is used to display the second content of the second dimension in the hybrid application.

[0035] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0036] Finally, an embodiment of this specification also provides a computer program product, wherein the computer program product includes a computer program, and the computer program implements the above method when executed by a processor.

[0037] By using the embodiments of this specification, a hybrid application may include content of multiple different dimensions. First, an operation instruction of a user for a first content of a first dimension in the hybrid application is received. Then, the first content of the first dimension is processed according to the operation instruction. Then, based on the processed first content, the second content of the second dimension is obtained according to the predetermined cross-dimensional interaction logic. Finally, the second content of the second dimension is displayed in the hybrid application, realizing the interaction between content of different dimensions displayed in the same three-dimensional space, and enhancing the flexibility of the application. In addition, based on the hybrid application form including multiple different dimensions, the advantages of interaction and experience of content of different dimensions can be fully utilized, and the content of different dimensions cooperate with each other to provide a better comprehensive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 The figure shows a schematic diagram of a system for implementing a cross-dimensional interaction method of a hybrid application according to an embodiment of the present specification;

[0040] Figure 2 The figure shows a flow chart of a cross-dimensional interaction method of a hybrid application according to an embodiment of the present specification;

[0041] Figure 3 The figure shows the display process of the content of different dimensions when the content of different dimensions comes from different content providers in the embodiment of this specification;

[0042] Figure 4 The figure shows a process of obtaining the second content of the second dimension according to the interaction logic based on the processed first content in the embodiment of this specification;

[0043] Figure 5 The figure shows the process of processing the first content according to the operation instruction in the embodiment of this specification;

[0044] Figure 6 It is a schematic diagram of the structure of a cross-dimensional interactive device for implementing a hybrid application in this specification;

[0045] Figure 7 The figure shows a schematic diagram of the structure of a computer device according to an embodiment of the present specification.

[0046]

Description of the accompanying drawings

[0047] 101. Hybrid applications;

[0048] 102. The first content of the first dimension;

[0049] 103. The second content of the second dimension;

[0050] 601, operation instruction receiving unit;

[0051] 602, logic processing unit;

[0052] 603. content generation unit;

[0053] 604, content display unit;

[0054] 702. Computer equipment;

[0055] 704. Processing equipment;

[0056] 706. Storage resources;

[0057] 708, driving mechanism;

[0058] 710, input / output module;

[0059] 712. Input devices;

[0060] 714. Output devices;

[0061] 716. Presentation equipment;

[0062] 718. Graphical user interface;

[0063] 720, network interface;

[0064] 722. Communication link;

[0065] 724. Communication bus. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0067] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0068] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] like Figure 1 The diagram shows a schematic diagram of the implementation system of the cross-dimensional interaction method of the hybrid application in the first embodiment of the present specification, which can be applied to the XR scene, including: a hybrid application 101, a first content 102 of the first dimension, and a second content 103 of the second dimension. The dimensions of the first content 102 of the first dimension and the second content 103 of the second dimension are different, for example, the first dimension is two-dimensional, and the second dimension is three-dimensional, etc. The first content 102 of the first dimension and the second content 103 of the second dimension can be run in the hybrid application 101, that is, the hybrid application 101 is responsible for the logical processing and rendering of the first content 102 of the first dimension and the second content 103 of the second dimension. The first content 102 of the first dimension and the second content 103 of the second dimension can also be run on different hardware or system platforms, and the hardware or system platform corresponding to the content of each dimension processes the logical processing and rendering of the content of each dimension, and the hybrid application is used to display the rendering screen provided by each hardware or / system platform. The content of each dimension constitutes a complete hybrid application 101 to complete a unified task.

[0070] The hybrid application 101 is responsible for processing the interaction logic between the first content 102 of the first dimension and the second content 103 of the second dimension.

[0071] It should be noted that the dimension in the embodiments of this specification refers to the dimension of the content relative to the user in the display space. For example, two-dimensional content will not change adaptively with the change of the user's viewing angle, that is, when the user does not operate the two-dimensional content, the two-dimensional content can only provide the user with one viewing angle, such as the content displayed on the mobile phone and computer screen, which are all two-dimensional. Three-dimensional content can change adaptively with the change of the user's viewing angle, that is, when the user does not operate the three-dimensional content, the three-dimensional content can provide the user with multiple different viewing angles, such as the three-dimensional content displayed in XR glasses.

[0072] like Figure 1 As described above, the first content 102 of the first dimension in the hybrid application 101 is "a two-dimensional turtle with mushrooms on its back", and the "two-dimensional turtle with mushrooms on its back" will not change adaptively with the change of the user's viewing angle, but the user can manually adjust the "two-dimensional turtle with mushrooms on its back". The second content 103 of the second dimension is "a three-dimensional turtle with mushrooms on its back", and the "three-dimensional turtle with mushrooms on its back" can change adaptively with the change of the user's viewing angle.

[0073] In addition, it should be noted that Figure 1 What is shown is only one application environment provided by the present disclosure. In actual application, other application scenarios may also be included, and this specification does not limit them.

[0074] In order to solve the problems existing in the prior art, an embodiment of the present specification provides a cross-dimensional interaction method for a hybrid application, which realizes the construction of a hybrid application based on contents of multiple different dimensions. The hybrid application can contain contents of multiple different dimensions, and cross-dimensional interaction of contents of different dimensions in the hybrid application is realized, providing users with a better user experience and solving the problem that cross-dimensional interaction of hybrid applications cannot be performed in the prior art. Figure 2 The flowchart of a cross-dimensional interaction method of a hybrid application in an embodiment of this specification is shown. This figure describes the process of cross-dimensional interaction between different dimensional contents in a hybrid application, but it may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the system or device product is executed in practice, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 2 As shown, the method may be executed by a processor of an XR device, and may include:

[0075] Step 201: receiving an operation instruction from a user for a first content of a first dimension in the hybrid application;

[0076] Step 202: Processing the first content according to the operation instruction;

[0077] Step 203: Based on the processed first content, obtain second content of a second dimension according to an interaction logic;

[0078] Step 204: Display the second content of the second dimension in the hybrid application.

[0079] Through the method of the embodiment of this specification, a hybrid application can include content of multiple different dimensions. First, an operation instruction from a user for the first content of the first dimension in the hybrid application is received, and then the first content of the first dimension is processed according to the operation instruction. Then, based on the processed first content, the second content of the second dimension is obtained according to the predetermined cross-dimensional interaction logic, and finally the second content of the second dimension is displayed in the hybrid application, realizing the interaction between content of different dimensions in the same three-dimensional space. The flexibility of the application is enhanced. In addition, based on the hybrid application form including multiple different dimensions, the advantages of interaction and experience of content of different dimensions can be fully utilized, and the content of different dimensions cooperate with each other to provide a better comprehensive experience.

[0080] In the embodiments of the present specification, the first content of the first dimension and the second content of the second dimension can both be rendered and run by a hybrid application, that is, the first content of the first dimension and the second content of the second dimension can correspond to a model (that is, the original data of the content) in the hybrid application, and the hybrid application displays the model as content of different dimensions according to the dimensional requirements and runs and displays it. In this case, the hybrid application first receives the user's operation instructions for the first content of the first dimension, and then processes the model corresponding to the first content according to the operation instructions, and then generates or updates the second content of the second dimension according to the processed model, thereby realizing the display of multiple contents of different dimensions in the same hybrid application. Among them, the operation instruction can be the user's editing or adjustment of the first content, which is not limited by the embodiments of the present specification.

[0081] In an embodiment of the present specification, the multiple windows in the hybrid application respectively display and run contents of different dimensions. For example, two-dimensional content is displayed and run in a 2D window, and three-dimensional content is displayed and run in a 3D window. The arrangement relationship and interactive relationship between multiple windows can be customized by the hybrid application, and each window specifies the rendering content and interactive logic. The content can be local content, and the application completes the relevant processing of the picture and logic, or it can specify a remote content and set the link address of the remote content. Developers can use extension plug-ins to expand existing applications on an XR platform across devices and dimensions. In an existing application, the corresponding development can be completed by calling the interface corresponding to the provided window. The extended content also supports the creation of multiple windows, and each window can also specify a different source.

[0082] In the embodiments of this specification, a 3D window is a subspace that can be observed 360 degrees in a three-dimensional space. The subspace can be embedded into a space through a projection matrix. For example, the subspace defined by the window can be embedded into the entire user space through a projection matrix. In addition, the subspace can artificially set the display / interaction boundary. There are many specific settings, such as setting a rectangle (for example, a two-dimensional window), or determining the projection matrix (three-dimensional window) according to the size of the window, or the actual boundary can be determined by the content in the subspace (for example, the place where there is a window corresponding to the content is considered a subspace). The content and interaction logic displayed in the subspace are controlled by the application to which it belongs. A hybrid application can contain one or more windows, in which the displayed content is a three-dimensional scene or a two-dimensional scene. Each window has its own independent coordinate system. Multiple windows are unified in the same world coordinate system (that is, the coordinate system of the hybrid application corresponding to the three-dimensional space) by the hybrid application described in the embodiments of this specification.

[0083] In some other embodiments of this specification, a hybrid application may also include multiple sub-applications of different dimensions, which is not limited by the embodiments of this specification.

[0084] It can be understood that, using this method, the user can operate the first content of the first dimension in the window corresponding to the first dimension, obtain the second content of the second dimension based on the processed first content according to the interaction logic, and display the second content of the second dimension in the window corresponding to the second dimension, thereby achieving a natural cross-dimensional experience.

[0085] In some other embodiments of this specification, the first content of the first dimension and the second content of the second dimension can be run on different hardware / system platforms, and the hardware / system platforms of the respective dimensional contents process the respective dimensional contents. Specifically, according to one embodiment of this specification, the contents of the different dimensions come from different content providers, and the contents of each dimension are rendered by the corresponding content providers;

[0086] like Figure 3 As shown, before receiving an operation instruction from a user for the first content of the first dimension in the hybrid application, the method further includes:

[0087] Step 301: receiving a first rendering image corresponding to each dimensional content sent by each content provider;

[0088] Step 302: Display at least two of the first rendering images in the hybrid application.

[0089] In the embodiments of this specification, the content provider corresponding to each dimension of content stores the model of the respective dimension of content, generates the rendering screen of the respective dimension of content, and sends the generated rendering screen to the hybrid application respectively, which is displayed by the hybrid application, or can be displayed by the hybrid application in the window corresponding to the respective dimension. The hybrid application is responsible for the interaction logic between the contents of each dimension, for example, determining the effect on the content of another dimension based on the user's operation on the content of one dimension.

[0090] Further, according to an embodiment of this specification, Figure 4 As shown, based on the processed first content, obtaining the second content of the second dimension according to the interaction logic further includes:

[0091] Step 401: sending the processing result of the first content to a content provider corresponding to the second dimension of the operation instruction, so that the content provider processes a model of the second content stored in itself according to the processing result to obtain a second rendering picture of the second content;

[0092] Step 402: Receive the second rendering picture sent by the content provider, and display the second rendering picture in the hybrid application.

[0093] In the example of this specification, the hybrid application can send the processing results of the first content of the first dimension to the content provider corresponding to the second dimension. The content provider calculates the model of the second content stored in itself according to the processing results, obtains the rendering picture of the second content, and sends the rendering picture to the hybrid application for display.

[0094] For example, the content provider corresponding to the two-dimensional content in the hybrid application is a computer, and the content provider corresponding to the three-dimensional content is XR glasses (locally, the hybrid application runs in the XR glasses). The two-dimensional content provided by the computer and the three-dimensional content provided by the XR glasses are displayed in the three-dimensional space of the XR glasses. The computer may include an album application, and the two-dimensional content is pictures, videos, model preview images, etc. processed by the album application in the computer, and supports editing two-dimensional pictures, videos, and model preview images on the computer. The embodiments of this specification can display two-dimensional pictures, videos, model preview images, etc. sent by the computer in a hybrid application. Specifically, the two-dimensional pictures, videos, model preview images, etc. are sent to the XR glasses, and the XR glasses generate three-dimensional pictures, videos, and models corresponding to the two-dimensional pictures, videos, and model preview images, and then display them in the hybrid application. Users can better observe three-dimensional pictures, videos, and models in the XR glasses, thereby providing users with a better user experience.

[0095] However, because XR glasses (local) operate the three-dimensional content in XR glasses through virtual rays, etc., the experience of editing three-dimensional pictures, videos, models, etc. in XR glasses will be poor, and it is difficult to perform some precise operations. Therefore, through the hybrid application of the embodiments of this specification, the existing application ecology in the computer can be used to edit the two-dimensional pictures, videos, and model previews, and the corresponding three-dimensional pictures, videos, and models are generated based on the edited two-dimensional pictures, videos, and model previews, and displayed in the XR glasses.

[0096] In addition, the contents of each dimension can interact with each other, and the hybrid application handles the interaction logic. For example, when the three-dimensional picture displayed in the window corresponding to the XR glasses is operated, when the user clicks the edit button, the two-dimensional picture corresponding to the three-dimensional picture is opened in the window corresponding to the computer, and the user can use the computer to edit it in the window corresponding to the computer, and the three-dimensional picture corresponding to the edited two-dimensional picture is displayed in the window corresponding to the XR glasses. In addition, you can select a two-dimensional picture in the window corresponding to the computer, and issue operation instructions for displaying the two-dimensional picture in the window corresponding to the XR glasses by swiping up, dragging, etc. The hybrid application sends the two-dimensional picture to the XR glasses based on the selected two-dimensional picture according to the interaction logic (the hybrid application can also be executed locally, which is not limited in the embodiments of this specification) to generate a three-dimensional picture corresponding to the two-dimensional picture, and then display it in the window corresponding to the XR glasses. A natural cross-device and cross-dimensional experience is achieved.

[0097] Specifically, the two-dimensional image data can be used as a texture and pasted on a plane or a plate-like geometric body with thickness to create a three-dimensional image. The two-dimensional image can also be calculated using an existing algorithm for converting two-dimensional images into three-dimensional images to obtain a three-dimensional image, which is not limited in the embodiments of this specification.

[0098] According to an embodiment of the present specification, the interaction logic includes a corresponding relationship between the first content and the second content;

[0099] like Figure 5 As shown, processing the first content according to the operation instruction further includes:

[0100] Step 501: executing the interaction logic according to the operation instruction to obtain an interaction result, wherein the interaction result at least includes the attributes and posture of the second content after the interaction;

[0101] Step 502: taking the interaction result as the processing result.

[0102] In the embodiments of this specification, the interaction logic includes the correspondence between the first content and the second content, for example, operating a part of the first content is equivalent to operating a corresponding part of the second content, or adjusting a part of the first content is equivalent to adjusting a corresponding part of the second content. Figure 1 As shown, a correspondence between each part of the "two-dimensional little turtle with mushrooms on its back" and each part of the "three-dimensional little turtle with mushrooms on its back" is pre-established.

[0103] According to the operation on the first part of the specified first content in the operation instruction, the second part of the second content is determined according to the corresponding relationship, the attributes and posture of the second part of the second content are changed according to the operation (that is, the impact of the operation on the second content is obtained according to the interaction logic), and the second content with changed attributes and posture is displayed in the mixed space.

[0104] For example, when a user clicks a button in window A, the logic corresponding to the button is to control a model in window B to execute animation a effect or modify the color, etc. The hybrid application obtains the effect on the model in window B according to the interaction logic, and sends the effect to the content provider corresponding to window B. The content provider corresponding to window B responds to the effect, generates a rendering of animation a, and then sends it to window B for display.

[0105] For example, continue as Figure 1As shown, in the window corresponding to the "two-dimensional turtle with mushrooms on its back", if the color of the "mushrooms" in the "two-dimensional turtle with mushrooms on its back" is changed, the "mushrooms" in the "three-dimensional turtle with mushrooms on its back" will be adjusted to the same color according to the corresponding relationship.

[0106] In some other embodiments of this specification, when the performance of the device where the hybrid application is located is limited, the content with a large amount of calculation can be handed over to a remote server with stronger performance for processing, and the content provided by the remote server is displayed in the window corresponding to the remote server in the hybrid application. The remote server is responsible for processing and rendering the content. The local hybrid application transmits the interactive event of the user operating the content in the window to the remote server, and the remote server performs corresponding processing according to the interactive event, and renders the screen of the content, and transmits the rendered screen to the local hybrid application, and the rendered screen is used as part of the content of the hybrid application. Content with a small amount of calculation can also be processed and rendered locally, thereby reducing latency. Through the method of the embodiments of this specification, the advantages of different platforms can be fully utilized. For users, there is no need to care about the source of each part of the content in the hybrid application, thereby giving users a unified experience on the same device and improving user experience.

[0107] For example, two windows are created in a hybrid application, a two-dimensional window A and a three-dimensional window B. The content of two-dimensional window A is rendered locally because the amount of calculation is small, while the content of three-dimensional window B is processed by a remote device because the amount of calculation is large. Then the local generates two-dimensional content inside two-dimensional window A, and two-dimensional window A responds to the user's interactive events. Three-dimensional window B specifies a remote application link and establishes a connection with the remote device through the link. The hybrid application generates an interactive event corresponding to the content in three-dimensional window B according to the operation instructions and interactive logic in two-dimensional window A, and sends the interactive event to the remote device. The remote device contains the model of the three-dimensional content to be rendered and the corresponding application processing logic. After receiving the interactive event, the remote device executes the application processing logic and calculates the model of the three-dimensional content, and renders the three-dimensional content according to the calculated three-dimensional model, and sends the rendered three-dimensional content to the hybrid application for display in the three-dimensional window B of the hybrid application.

[0108] According to one embodiment of the present specification, the method further includes: providing the user's observation angle information for each dimensional content to each content provider, so that each content provider renders the content of its own dimension according to the observation angle information to obtain the first rendering screen of each content provider.

[0109] In the embodiments of this specification, for content (such as three-dimensional content) that needs to be rendered according to the user's viewing angle information, the hybrid application can also send the user's viewing angle information to the content provider corresponding to the content. The viewing angle information may include position, orientation, etc. The content provider renders the content according to the viewing angle information, thereby further improving the user experience.

[0110] According to one embodiment of the present specification, because the contents of different windows in a hybrid application can come from different content providers, such as computers, mobile phones, etc., the operating logic of these different content providers may also be different. However, because the contents provided by these content providers are displayed in the corresponding windows of the hybrid application as part of the hybrid application, when the user operates the contents of different windows, the user needs to constantly switch the operating logic. For example, when operating the contents in the window corresponding to the computer, it is necessary to operate the contents in the window according to the logic of the computer's mouse input. When operating the contents in the window corresponding to the mobile phone, it is necessary to operate the contents in the window according to the logic of the mobile phone, which creates a certain burden on the user. Therefore, in an embodiment of the present specification, after displaying the received first rendering screen in the hybrid application, the method includes:

[0111] For content of one dimension, the user's operation instructions for the content are converted according to the operation mode of the content provider corresponding to the content, and the converted operation instructions are sent to the content provider, so that the content provider performs corresponding logical processing on the content according to the converted operation instructions and updates the first rendering screen.

[0112] In the embodiments of the present specification, the user's operating instructions for the content of the content provider are converted according to the operating mode of the content provider and converted into the operating logic of the content provider, and then the converted operating instructions are sent to the content provider for processing. Therefore, there is no need to constantly switch the operating logic in the hybrid application. The user only needs to use the hybrid application's own operating logic to operate the content from different content providers in each window.

[0113] For example, the interaction of two-dimensional content on a computer is done using a mouse and keyboard, the interaction of two-dimensional content on a mobile phone is done using a touch screen, and hybrid applications are run in XR glasses, and the interaction of content in the XR glasses is done using a handle or gestures. These contents from different content providers are ultimately displayed in corresponding windows in the three-dimensional space of the XR glasses. Through the methods of the embodiments of this specification, the operating instructions using the handle or gestures in the XR glasses are converted according to the operating logic of the computer or mobile phone, so that users can use a unified operating logic to operate the content in the corresponding windows of different content providers. There is no need to convert the handle or gestures into a mouse or keyboard when operating the content in the corresponding window of the computer, nor is there any need to convert the handle or gestures into a touch screen when operating the content in the corresponding window of the mobile phone.

[0114] Furthermore, because the coordinate system of the space of the hybrid application is the world coordinate system, and the coordinate system of each window in the hybrid application is the local coordinate system, in order to improve the accuracy of converting the handle or handle of the hybrid application according to the operation logic of each window, according to an embodiment of the present specification, after converting the operation instruction of the user for the content according to the operation mode of the content provider corresponding to the content, the method further includes:

[0115] The converted operation instruction is transformed from the world coordinates of the hybrid application to the local coordinate system of the content provider, and the transformed operation instruction is sent to the content provider.

[0116] Exemplarily, when a user operates the two-dimensional content in the two-dimensional window of a computer through the handle of the XR glasses, the three-dimensional interaction information of the user's handle in the XR glasses is converted into the mouse and keyboard operation information of the computer. The three-dimensional interaction information includes the intersection position of the handle's ray and the content in the two-dimensional window and the handle's button information. The system automatically transforms the intersection position from the world coordinates of the hybrid application to the local coordinate system of the computer's two-dimensional window, converts the handle's ray intersection position into the mouse position of the content in the two-dimensional window, and converts the handle's button information into the mouse button information. This operation information is automatically translated into a click operation performed by pressing the mouse button on the computer's two-dimensional content. In this way, the computer can directly process this operation information, correctly complete the corresponding processing logic, and provide a rendering screen after the content is operated.

[0117] For example, when a user operates the two-dimensional content in the two-dimensional window of a mobile phone through the handle of the XR glasses, the three-dimensional interaction information of the user's handle in the XR glasses is converted into a touch operation of the mobile phone. The three-dimensional interaction information includes the intersection position of the handle's ray and the content in the two-dimensional window and the button information of the handle. The system automatically transforms the intersection position from the world coordinates of the hybrid application to the local coordinate system of the two-dimensional window of the mobile phone, converts the intersection position of the handle's ray into the position of the touch operation of the content in the two-dimensional window, and converts the button information of the handle into the touch down and up states in the touch operation. The operation information is then automatically translated into a touch screen click operation at the corresponding position on the mobile phone screen.

[0118] It should be noted that the above examples are merely illustrative. In addition to mouse clicks and mobile phone touches, they can also be converted into operations such as dragging, long pressing, and sliding according to the operation method of the content provider, so that users only need to focus on one interaction method to complete the interaction of content of different dimensions from different content providers.

[0119] Based on the same inventive concept, the embodiment of this specification also provides a hybrid application cross-dimensional interaction device, such as Figure 6 As shown, the hybrid application includes content of multiple different dimensions, and the device includes:

[0120] An operation instruction receiving unit 601 is used to receive an operation instruction from a user for a first content of a first dimension in the hybrid application;

[0121] A logic processing unit 602, configured to process the first content according to the operation instruction;

[0122] A content generating unit 603 is configured to obtain second content of a second dimension according to an interaction logic based on the processed first content;

[0123] The content display unit 604 is configured to display the second content of the second dimension in the hybrid application.

[0124] The beneficial effects obtained by the above-mentioned device are consistent with the beneficial effects obtained by the above-mentioned method, and will not be described in detail in the embodiments of this specification.

[0125] like Figure 7The structure diagram of the computer device of the embodiment of this specification is shown. The apparatus in this article can be the computer device in this embodiment, and the method of this article is executed. The computer device 702 may include one or more processing devices 704, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 702 may also include any storage resource 706, which is used to store any kind of information such as code, settings, data, etc. Non-limiting, for example, the storage resource 706 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any storage resource can use any technology to store information. Further, any storage resource can provide volatile or non-volatile retention of information. Further, any storage resource can represent a fixed or removable component of the computer device 702. In one case, when the processing device 704 executes an associated instruction stored in any storage resource or a combination of storage resources, the computer device 702 can perform any operation of the associated instruction. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any storage resources, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like.

[0126] The computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via input devices 712) and for providing various outputs (via output devices 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface (GUI) 718. In other embodiments, the input / output module 710 (I / O), the input device 712, and the output device 714 may not be included, and the computer device 702 may be used as a computer device in a network. The computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0127] The communication link 722 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0128] The embodiments of the present specification also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0129] The embodiments of the present specification also provide a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to execute the above method.

[0130] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0131] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0132] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0133] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0134] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0136] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0138] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A cross-dimensional interaction method for hybrid applications, characterized in that: The hybrid application includes content of multiple dimensions, the content of the different dimensions comes from different content providers, the content of each dimension is rendered by the corresponding content provider, the multiple windows in the hybrid application respectively display and run the content of different dimensions, the arrangement relationship and interaction relationship between the multiple windows are customized by the hybrid application, the content of each dimension can interact with each other, and the hybrid application handles the interaction logic; The method comprises: Receiving a first rendering picture corresponding to each dimensional content sent by each content provider; Displaying at least two of the first rendering images in the hybrid application; receiving an operation instruction from a user for a first content of a first dimension in the hybrid application; Processing the first content according to the operation instruction; Based on the processed first content, obtaining second content of a second dimension according to interaction logic; presenting the second content of the second dimension in the hybrid application; Among them, for content of one dimension, the user's operation instructions for the content are converted according to the operation mode of the content provider corresponding to the content, and the converted operation instructions are transformed from the world coordinates of the hybrid application to the local coordinate system of the content provider, and the transformed operation instructions are sent to the content provider, so that the content provider performs corresponding logical processing on the content according to the converted operation instructions and updates the first rendering screen, wherein the coordinate system of the space of the hybrid application is the world coordinate system, and the coordinate system of each window in the hybrid application is the local coordinate system.

2. The method according to claim 1, characterized in that The method further includes: providing the user's observation angle information for each dimensional content to each content provider, so that each content provider renders the content of its own dimension according to the observation angle information to obtain the first rendering screen of each content provider.

3. The method according to claim 1, characterized in that Based on the processed first content, obtaining second content of a second dimension according to the interaction logic further includes: Sending the processing result of the first content to a content provider corresponding to the second dimension of the operation instruction, so that the content provider processes the model of the second content stored in itself according to the processing result to obtain a second rendering picture of the second content; The second rendered picture sent by the content provider is received, and the second rendered picture is displayed in the hybrid application.

4. The method according to claim 3, characterized in that The interaction logic includes a corresponding relationship between the first content and the second content; Processing the first content according to the operation instruction further includes: Execute the interaction logic according to the operation instruction to obtain an interaction result, wherein the interaction result at least includes the attributes and posture of the second content after the interaction; The interaction result is used as the processing result.

5. The method according to claim 1, characterized in that Contents of different dimensions run on different hardware or system platforms, and the contents of each dimension form a complete application to complete a unified task.

6. The method according to claim 1, characterized in that Interaction information between the first content of the first dimension and the second content of the second dimension is displayed on the same device.

7. A hybrid application cross-dimensional interactive device, characterized in that: The hybrid application includes content of multiple dimensions, the content of the different dimensions comes from different content providers, the content of each dimension is rendered by the corresponding content provider, the multiple windows in the hybrid application respectively display and run the content of different dimensions, the arrangement relationship and interaction relationship between the multiple windows are customized by the hybrid application, the content of each dimension can interact with each other, and the hybrid application handles the interaction logic; The device comprises: An operation instruction receiving unit, configured to receive an operation instruction from a user for a first content of a first dimension in the hybrid application; a logic processing unit, configured to process the first content according to the operation instruction; A content generating unit, configured to obtain second content of a second dimension according to an interaction logic based on the processed first content; A content display unit, configured to display the second content of the second dimension in the hybrid application; The device is also used to receive the first rendering screen corresponding to each dimensional content sent by each content provider; display at least two of the first rendering screens in the hybrid application; for one dimensional content, convert the user's operation instructions for the content according to the operation mode of the content provider corresponding to the content, transform the converted operation instructions from the world coordinates of the hybrid application to the local coordinate system of the content provider, and send the transformed operation instructions to the content provider, so that the content provider performs corresponding logical processing on the content according to the converted operation instructions and updates the first rendering screen, wherein the coordinate system of the hybrid application space is the world coordinate system, and the coordinate system of each window in the hybrid application is the local coordinate system.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images