Interface interaction method

Abstract

The embodiment of the invention discloses an interface interaction method which comprises the steps that drawing attribute information of all components in source codes of a target interface and interaction event information corresponding to all the components are read, and the source codes are written in a component form; generating an interface canvas based on the drawing attribute information of each component, the interface canvas containing each component of the target interface; generating an interaction canvas based on the position attribute information and the size attribute information of each component; based on the interface canvas, rendering the target interface in the browser page; in response to the monitored event triggered based on the target interface, determining a pixel coordinate of an event triggering position in the interaction canvas; and based on the pixel coordinates, obtaining a target component for the event, and executing a response operation of an interaction event corresponding to the target component. According to the embodiment of the invention, the consistency of the user interfaces in the browser pages of the terminals can be realized, so that the consistency of event interaction response is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual reality technology, and in particular to an interface interaction method. Background Art

[0002] Web Extended Reality (WebXR) is a technology for creating virtual reality and augmented reality experiences on web pages and is used to develop cross-platform immersive web applications.

[0003] In the related art, the user interface (UI) is usually implemented in a Web browser based on codes written based on Document Object Model (DOM) elements; however, in WebXR mode, since virtual reality (VR) devices cannot render DOM elements, it is necessary to implement the user interface based on bitmap codes. Therefore, the user interface is implemented differently on multiple terminals, which may lead to inconsistency in the user interfaces of multiple terminals, and may lead to inconsistency in the interactive event responses of the user interfaces of multiple terminals. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure is proposed. An embodiment of the present disclosure provides an interface interaction method.

[0005] According to one aspect of an embodiment of the present disclosure, there is provided an interface interaction method, including:

[0006] Reading drawing attribute information of each component and interaction event information corresponding to each component in the source code of the target interface, wherein the source code is written in the form of components;

[0007] Based on the drawing attribute information of each component, an interface canvas is generated, in which each component of the target interface is contained; and based on the position attribute information and the size attribute information of each component, an interactive canvas is generated, in which the occupied areas of each component in the interactive canvas are respectively filled with different filling colors;

[0008] Based on the interface canvas, rendering the target interface in a browser page;

[0009] In response to monitoring an event triggered based on the target interface, determining the pixel coordinates of the event triggering position in the interactive canvas;

[0010] Based on the pixel coordinates, the target component targeted by the event is obtained, and a response operation of the interactive event corresponding to the target component is executed.

[0011] In some implementations of the present disclosure, generating an interface canvas based on drawing attribute information of each component includes:

[0012] Based on the size attribute information and position attribute information of each component in the source code and based on the typesetting engine, determine the actual size information and actual position information of each component;

[0013] The canvas interface function is called to draw the interface canvas based on the actual size information and actual position information of each component and other drawing attribute information except the size attribute information and the position attribute information.

[0014] In some other implementations of the present disclosure, generating an interactive canvas based on the position attribute information and the size attribute information of each component includes:

[0015] Call the canvas interface function to generate the initial interactive canvas based on the actual size information and actual position information of each component;

[0016] The occupied areas of the components in the initial interactive canvas are filled with different filling colors.

[0017] In some other implementations of the present disclosure, after reading the drawing attribute information of each component in the source code of the target interface, the method further includes:

[0018] A color mapping table is generated, wherein the color mapping table records the fill colors of the occupied areas of the components in the interactive canvas.

[0019] In some other implementations of the present disclosure, acquiring the target component targeted by the event based on the pixel coordinates includes:

[0020] Get the fill color corresponding to the pixel coordinates in the interactive canvas;

[0021] The component corresponding to the filling color in the color mapping table is determined as the target component.

[0022] In some other implementations of the present disclosure, in an immersive browser mode in a virtual reality device, rendering the target interface in a browser page based on the interface canvas includes:

[0023] Creating a plane geometry for carrying the interface canvas;

[0024] Mapping the interface canvas as a texture map onto the plane geometry;

[0025] Render the plane geometry.

[0026] In some other implementations of the present disclosure, in response to monitoring an event triggered based on the target interface, determining the pixel coordinates of the event triggering position in the interactive canvas includes:

[0027] Based on the method of triggering the event, a spatial ray is made to assist the event interaction;

[0028] Determining an intersection point between the ray and the plane geometric body;

[0029] Based on the position coordinates of the intersection point on the plane geometric body, the pixel coordinates corresponding to the intersection point in the interactive canvas are determined.

[0030] In some other implementations of the present disclosure, in a web browser, rendering the target interface in a browser page based on the interface canvas includes:

[0031] Adding an interface canvas element corresponding to the interface canvas in the document object model of the browser page;

[0032] The target interface is rendered based on an interface canvas element in a document object model of the browser page.

[0033] In some other implementations of the present disclosure, after reading the drawing attribute information of each component in the source code of the target interface and the interaction event information corresponding to each component, the method further includes:

[0034] Based on the interaction event information corresponding to each component, register the interaction events of each component;

[0035] The interaction events of each component are stored in the heap memory.

[0036] In some other implementations of the present disclosure, the step of executing a response operation of an interaction event corresponding to the target component includes:

[0037] Obtain an interaction event corresponding to the target component in the heap memory, and execute the interaction event.

[0038] According to another aspect of the present disclosure, there is provided an interface interaction device, comprising:

[0039] A reading module, used to read the drawing attribute information of each component and the interaction event information corresponding to each component in the source code of the target interface, wherein the source code is written in the form of components;

[0040] A generation module, for generating an interface canvas based on the drawing attribute information of each component, wherein the interface canvas contains each component of the target interface; and generating an interactive canvas based on the position attribute information and the size attribute information of each component, wherein the occupied area of ​​each component in the interactive canvas is filled with different filling colors respectively;

[0041] A rendering module, used for rendering the target interface in a browser page based on the interface canvas;

[0042] A coordinate determination module, configured to determine the pixel coordinates of the event triggering position in the interactive canvas in response to monitoring an event triggered based on the target interface;

[0043] An event response module is used to obtain the target component targeted by the event based on the pixel coordinates, and execute a response operation of the interactive event corresponding to the target component.

[0044] In some implementations of the present disclosure, the generating module includes:

[0045] A first determination submodule, configured to determine actual size information and actual position information of each component based on the size attribute information and position attribute information of each component in the source code and based on a typesetting engine;

[0046] The drawing submodule is used to call the canvas interface function to draw the interface canvas based on the actual size information and actual position information of each component and other drawing attribute information except the size attribute information and position attribute information.

[0047] In some other implementations of the present disclosure, the generating module includes:

[0048] The second determination submodule is used to call the canvas interface function to generate an initial interactive canvas based on the actual size information and actual position information of each component;

[0049] The filling submodule is used to fill the occupied area of ​​each component in the initial interactive canvas with different filling colors.

[0050] In some other implementations of the present disclosure, the device further includes:

[0051] The color mapping module is used to generate a color mapping table, in which the filling color of the occupied area of ​​each component in the interactive canvas is recorded.

[0052] In some other implementations of the present disclosure, the event response module includes:

[0053] A first acquisition submodule is used to acquire the fill color corresponding to the pixel coordinates in the interactive canvas;

[0054] The third determining submodule is used to determine the component corresponding to the filling color in the color mapping table as the target component.

[0055] In some other implementations of the present disclosure, in an immersive browser mode in a virtual reality device, the rendering module includes:

[0056] Create a submodule for creating a plane geometric body for carrying the interface canvas;

[0057] A mapping submodule, used for mapping the interface canvas as a texture map on the plane geometric body;

[0058] The first rendering submodule is used to render the plane geometric body.

[0059] In some other implementations of the present disclosure, the coordinate determination module includes:

[0060] A ray submodule, used to make a space ray for assisting event interaction based on the way of triggering the event;

[0061] An intersection point determination submodule, used to determine the intersection point between the ray and the plane geometric body;

[0062] The coordinate determination submodule is used to determine the pixel coordinates of the intersection point in the interactive canvas based on the position coordinates of the intersection point on the plane geometric body.

[0063] In some other implementations of the present disclosure, in a web browser, the rendering module includes:

[0064] An element adding submodule, used for adding an interface canvas element corresponding to the interface canvas in the document object model of the browser page;

[0065] The second rendering submodule is used to render the target interface based on the interface canvas element in the document object model of the browser page.

[0066] In some other implementations of the present disclosure, the device further includes:

[0067] An event registration module is used to register the interaction events of each component based on the interaction event information corresponding to each component;

[0068] The storage module is used to store the interaction events of each component in the heap memory.

[0069] In some other implementations of the present disclosure, the event response module is used to obtain the interaction event corresponding to the target component in the heap memory and execute the interaction event.

[0070] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores computer program instructions, and when the computer program instructions are executed, the above-mentioned interface interaction method is implemented.

[0071] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0072] a memory for storing a computer program product;

[0073] The processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, the above-mentioned interface interaction method is implemented.

[0074] According to another aspect of the present disclosure, a computer program product is provided, including computer program instructions, and when the computer program instructions are executed by a processor, the above-mentioned interface interaction method is implemented.

[0075] Based on the interface interaction method provided by the above embodiment of the present disclosure, when rendering the user interface in the browser, the drawing attribute information of each component in the source code written in the form of components of the target interface and the interactive event information corresponding to each component can be read; based on the drawing attribute information of each component, an interface canvas is generated, in which each component of the target interface is contained; and based on the position attribute information and size attribute information of each component, an interactive canvas is generated, and the occupied areas of each component in the interactive canvas are filled with different filling colors respectively; based on the interface canvas, the target interface is rendered in the browser page; in response to monitoring the event triggered based on the target interface, the pixel coordinates of the event triggering position in the interactive canvas are determined; based on the pixel coordinates, the target component targeted by the event is obtained, and the response operation of the interactive event corresponding to the target component is executed. Thus, the technical solution of the present disclosure can realize that when each terminal triggers the browser page opening operation, the interface canvas is automatically drawn based on the source code of the user interface, and the user interface is rendered. Since the user interface is generated in each terminal according to the same source code, the consistency of the user interface in the browser page of each terminal can be ensured, and then the consistency of the event interaction response can be ensured.

[0076] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0078] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0079] Figure 1A flowchart of an interface interaction method provided by an exemplary embodiment of the present disclosure;

[0080] Figure 2 A schematic diagram of a process of generating an interface canvas provided by an exemplary embodiment of the present disclosure;

[0081] Figure 3 A schematic diagram of a process for generating an interactive canvas provided by an exemplary embodiment of the present disclosure;

[0082] Figure 4 A schematic diagram of a process for rendering a target interface in an immersive browser mode of a virtual reality device provided by an exemplary embodiment of the present disclosure;

[0083] Figure 5 A schematic diagram of a process flow for event interaction provided by an exemplary embodiment of the present disclosure;

[0084] Figure 6 A schematic diagram of a flow chart of determining position coordinates corresponding to an interactive event in an immersive browser mode of a virtual reality device provided by an exemplary embodiment of the present disclosure;

[0085] Figure 7 It is a structural schematic diagram of another embodiment of the interface interaction device disclosed in the present invention;

[0086] Figure 8 It is a structural schematic diagram of another embodiment of the interface interaction device disclosed in the present invention;

[0087] Fig. 9 A structural diagram of an electronic device provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0088] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0089] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0090] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0091] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0092] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0093] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0094] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0095] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0096] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0097] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0098] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0099] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0100] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0101] SUMMARY OF THE DISCLOSURE

[0102] The technical solution provided by the embodiment of the present disclosure is used to solve the problem of inconsistent user interfaces on multiple terminals caused by different user interface implementation methods on multiple terminals and the problem of inconsistent event interaction responses caused by inconsistent user interfaces on multiple terminals. For example, the user interface 1 in the web browser of the mobile terminal is inconsistent with the user interface 1 in the WebXR mode of the virtual reality device (such as VR glasses). Through the technical solution of this embodiment, the interface development framework in the browser of each terminal draws the interface based on the same source code, so the consistency of the user interface of each terminal can be ensured, and because the source code of the user interfaces of different terminals is the same, the code reuse rate is high and the development cost is low.

[0103] Exemplary Methods

[0104] Figure 1 A flowchart of an interface interaction method provided by an exemplary embodiment of the present disclosure; the interface interaction method can be applied to electronic devices such as mobile terminals and virtual reality devices, and an interface development framework capable of executing the following technical solutions is installed in the browser of the electronic device. The interface interaction method includes the following steps:

[0105] In step 101, drawing attribute information of each component and interaction event information corresponding to each component in the source code of the target interface are read, and the source code is written in the form of components.

[0106] In the present disclosure, in response to a user triggering the opening of a browser page or switching to WebXR mode, the source code of the browser page can be read, and the source code of the target interface written in the form of a component can be obtained therefrom.

[0107] The target interface is used to indicate the user interface for human-computer interaction in the browser page.

[0108] In this embodiment, writing the source code of the target interface in the form of components helps to achieve modularization of different functions, thereby improving the maintainability, reusability and development efficiency of the code.

[0109] The drawing attribute information of the component is used to indicate the attribute information required for drawing the component on the canvas, which may include the size attribute information (including length and width), position attribute information, background color, bounding box width, bounding box color, bottom margin and other attributes of the component. The interactive event information of the component is used to indicate the interactive events designed for each component, such as click events (such as onClick={event=>{console.log(event);setCount(count=>count+1);), pointer exit events (such as onPointerLeave={event=>{setHover(false);), pointer enter events (onPointerEnter={event=>{setHover(true);), etc.

[0110] In this embodiment, the drawing attribute information of each component and the interaction event information corresponding to each component can be obtained one by one by traversing the source code of the target interface.

[0111] After acquiring the interaction event information of each component, each interaction event information of each component may be registered. The event interaction information may include an event response function and is stored in the heap memory for subsequent event interaction.

[0112] In step 102, an interface canvas is generated based on the drawing attribute information of each component; and an interaction canvas is generated based on the position attribute information and size attribute information of each component.

[0113] The interface canvas contains various components of the target interface. A canvas element may be created first, and then the interface canvas may be obtained by drawing various components on the canvas according to the drawing attribute information of each component by calling the interface function of the canvas.

[0114] The occupied areas of each component in the interactive canvas are filled with different fill colors. When drawing the interactive canvas, you can first create a canvas element, draw the component frames of each component at the corresponding positions of the canvas according to the position attribute information and size attribute information of each component, and then fill them with the corresponding colors.

[0115] In this embodiment, unique identification information may be generated for each component first, and a fill color may be generated for each component. The fill color is inconsistent with the fill color written in the source code, and the fill color is used to fill the color of each component in the interactive canvas. For example, a red color with RGB (255,0,0) is generated as the fill color for component 1, and a yellow color with RGB (255,255,0) is generated as the fill color for component 2. Then, the occupied area of ​​component 1 in the interactive canvas may be filled with red (255,0,0), and the occupied area of ​​component 2 may be filled with yellow (255,255,0).

[0116] In the specific implementation, different filling colors are randomly generated for different components, and the filling colors of each component are different. After the filling colors are generated, a color mapping table is created, and the color mapping table is used to record the filling colors of each component.

[0117] In step 103, based on the interface canvas, the target interface is rendered in the browser page.

[0118] In this embodiment, for different terminals, the target interface can be rendered on the browser page in different ways.

[0119] In the immersive browser mode (WebXR) of the virtual reality device, a 3D scene can be developed based on the Web Graphics Library (WebGL), and a plane geometry can be created in it. The interface canvas is used as a texture map of the plane geometry through mapping to achieve the rendering of the target interface. For details, see Figure 4 The embodiment shown.

[0120] In the normal browser mode, an interface canvas element corresponding to the interface canvas can be added to the document object model of the browser page; then, the target interface is rendered based on the interface canvas element in the document object model of the browser page.

[0121] By inserting the interface canvas as an element into the document object model (DOM) of the browser page, the target interface can be rendered when the browser renders the browser page through the DOM.

[0122] In this embodiment, in browsers of different terminals, the target interfaces rendered based on the same interface canvas are consistent.

[0123] In step 104, in response to monitoring an event triggered based on the target interface, the pixel coordinates of the event triggering position in the interactive canvas are determined.

[0124] The event triggered based on the target interface may be various events triggered based on components in the target interface, such as a click event, a pointer exit event, a pointer enter event, and the like.

[0125] In specific implementation, the user can trigger the above events through sight when wearing VR glasses, or trigger the above events through a handle.

[0126] In this embodiment, when an event is triggered in the normal browser mode, since the target interface and the interactive canvas are consistent in size and position, the pixel coordinates of the event triggering position in the interactive canvas can be determined accordingly based on the position of the event triggering position on the target interface.

[0127] In specific implementation, when a user triggers a click event in a browser page, the browser's event interface can directly read the position of the event triggering position in the browser page, and then obtain the position of the event triggering position in the target interface based on the position of the target interface in the browser page.

[0128] In immersive browser mode (WebXR) on VR devices, you can Figure 6 The implementation method of the embodiment determines the pixel coordinates of the event triggering position in the interactive canvas, which will not be described in detail here.

[0129] In step 105, based on the pixel coordinates, the target component targeted by the event is obtained, and a response operation of the interactive event corresponding to the target component is executed.

[0130] In some implementations, after the pixel coordinates of the event triggering position in the interactive canvas are determined, the color of the point in the interactive canvas can be determined based on the pixel coordinates, and then the target component where the event occurs can be determined based on the color.

[0131] In this embodiment, after obtaining the interaction event information of each component according to the source code of the target interface through step 101, the interaction event information of each component is registered and the registered interaction event information is stored in the heap memory. Therefore, the corresponding interaction event information of the target event can be obtained from the heap memory, and the response operation of the interaction event can be executed.

[0132] Exemplarily, if it is determined that the event trigger location is component 1 and the event is a click event, the event interaction information corresponding to the click event of component 1 can be obtained from the heap memory. The event interaction information may include an event response function, and the response operation of the interaction event can be implemented by calling the function.

[0133] In the above steps 101 to 105, when rendering the user interface in the browser, the drawing attribute information of each component in the source code written in the form of components of the target interface and the interactive event information corresponding to each component can be read; based on the drawing attribute information of each component, an interface canvas is generated, in which each component of the target interface is contained; and based on the position attribute information and size attribute information of each component, an interactive canvas is generated, and the occupied areas of each component in the interactive canvas are filled with different filling colors respectively; based on the interface canvas, the target interface is rendered in the browser page; in response to monitoring the event triggered based on the target interface, the pixel coordinates of the event triggering position in the interactive canvas are determined; based on the pixel coordinates, the target component targeted by the event is obtained, and the response operation of the interactive event corresponding to the target component is executed. Thus, the technical solution disclosed in the present invention can realize that when each terminal triggers the browser page opening operation, the interface canvas is automatically drawn based on the source code of the user interface, and the user interface is rendered. Since the user interface is generated in each terminal according to the same source code, the consistency of the user interface in the browser page of each terminal can be ensured, and then the consistency of the event interaction response can be ensured.

[0134] In order to better illustrate the interface interaction scheme of the present disclosure, another embodiment is used below for illustration.

[0135] Figure 2 A schematic diagram of a process for generating an interface canvas provided by an exemplary embodiment of the present disclosure; this embodiment takes how to generate an interface canvas as an example for exemplary description, such as Figure 2 As shown, the following steps are included:

[0136] In step 201, based on the size attribute information and position attribute information of each component in the source code and based on the typesetting engine, the actual size information and actual position information of each component are determined.

[0137] In some implementations, the layout engine refers to an engine for laying out and layouting components in the target interface, such as the yoga engine, the WebKit engine, the Servo engine, and the like.

[0138] The size attribute information of each component includes attributes such as width and height that are used to indicate the size of the component, and the position attribute information may include attributes such as top, right, bottom, and left that are used to locate the component. The actual size information and actual position information of the component are used to indicate the actual length and width information of the component and the actual pixel coordinates in the canvas.

[0139] In step 202, a canvas interface function is called to draw an interface canvas based on the actual size information and actual position information of each component and other drawing attribute information except the size attribute information and the position attribute information.

[0140] In some implementations, the canvas interface function can be called to draw the frame of each component on the canvas based on the actual size information and actual position information of the component, and the overall drawing of each component can be completed based on other drawing attribute information, such as border color, font and other attribute information.

[0141] Through the above steps 201 to 202, the interface canvas is drawn according to the drawing attribute information of the component, which is helpful for subsequently rendering the target interface in the browser page according to the interface canvas.

[0142] Figure 3 FIG. 1 is a flow chart of generating an interactive canvas provided by an exemplary embodiment of the present disclosure. In this embodiment, how to generate an interactive canvas is used as an example for exemplary description. Figure 3 As shown, the following steps are included:

[0143] In step 301, a canvas interface function is called to generate an initial interactive canvas based on the actual size information and actual position information of each component.

[0144] The actual size and position information of each component can be obtained by referring to Figure 2 The implementation of step 201 in the illustrated embodiment will not be described in detail here.

[0145] In some implementations, the canvas interface function can be called to draw the component frame of each component on the canvas according to the actual size information and actual position information of the component to obtain an initial interactive canvas. In the initial interactive canvas, each component only draws the component frame, and does not draw the component according to other attribute information of the component.

[0146] In step 302, the occupied areas of the components in the initial interactive canvas are filled with different filling colors.

[0147] In some implementations, when traversing each component in the source code of the target interface, unique identification information may be generated for each component in advance, and a fill color may be generated for each component. The fill color is inconsistent with the fill color written in the source code, and the fill color is used to fill the color of each component in the interactive canvas. For example, a red color with an RGB value of (255,0,0) is generated as a fill color for component 1, and a yellow color with an RGB value of (255,255,0) is generated as a fill color for component 2. Then, the occupied area of ​​component 1 in the interactive canvas may be filled with red (255,0,0), and the occupied area of ​​component 2 may be filled with yellow (255,255,0).

[0148] In the specific implementation, different filling colors are randomly generated for different components, and the filling colors of each component are different. After the filling colors are generated, a color mapping table is created, and the color mapping table is used to record the filling colors of each component.

[0149] After the initial interactive canvas is created, the occupied areas of each component in the initial interactive canvas can be filled with the fill colors recorded in the color mapping table to obtain the interactive canvas.

[0150] Through the above steps 301 to 302, an implementation method for drawing an interactive canvas is disclosed. The interactive canvas provides a basis for interaction and response to events triggered in the target interface.

[0151] Figure 4 A flowchart of rendering a target interface in an immersive browser mode of a virtual reality device is provided for an exemplary embodiment of the present disclosure; this embodiment takes how to render a target interface in an immersive browser mode of a virtual reality device as an example for exemplary description, such as Figure 4 As shown, the following steps are included:

[0152] In step 401, a plane geometric body for carrying an interface canvas is created.

[0153] The plane geometric body is a geometric plane body in three-dimensional space, which is used to carry the interface canvas.

[0154] In the specific implementation, in WebXR, a three-dimensional scene is created based on WebGL, and a plane geometry is created. The plane geometry can be constructed by using 4 vertices and 2 triangles; the plane geometry can also be constructed by using more mesh triangles.

[0155] In some implementations, the user can set the relevant information of the plane geometry, such as size and position, by means of buttons in the browser client interface or through related menu instructions, and then create the plane geometry in the three-dimensional scene.

[0156] In step 402, the interface canvas is mapped onto the plane geometry as a texture map.

[0157] In the specific implementation, the UV coordinates of the four vertices of the plane geometric body are specified to be [0,0], [0,1] [1,1] [1,0] respectively, and the interface canvas is pasted on the plane geometric body through texture mapping, and the spatial coordinates of each surface point on the plane geometric body corresponding to the coordinates of each pixel point on the interface canvas are stored.

[0158] In step 403, the plane geometry is rendered.

[0159] Through the above steps 401 to 403, it is possible to render the target interface in the immersive browser mode of the virtual reality device. By creating a plane geometric body in the three-dimensional scene and mapping it, it is possible to accurately display the target interface in the immersive browser mode.

[0160] Figure 5 An exemplary embodiment of the present disclosure provides a flow chart of event interaction; this embodiment uses how to perform event interaction according to the pixel coordinates of the event trigger position in the interactive canvas as an example to illustrate, Figure 5 As shown, the following steps are included:

[0161] In step 501, the fill color corresponding to the pixel coordinates in the interactive canvas is obtained.

[0162] When drawing the interactive canvas, the occupied area of ​​each component (corresponding to different pixel coordinate areas) has been filled with the corresponding color, so the filling color can be determined according to the pixel coordinates corresponding to the event trigger position in the interactive canvas.

[0163] In step 502, the component corresponding to the fill color in the color mapping table is determined as the target component.

[0164] The color mapping table records the fill color of each component, so the component corresponding to the fill color at the pixel coordinate can be obtained from the color table.

[0165] In the above steps 501 to 502, by setting the fill color for the occupied area of ​​each component when drawing the interactive canvas, the target component targeted by the event is quickly obtained according to the fill color of the trigger position when an event is triggered. This is efficient and can improve event response efficiency.

[0166] Figure 6 An exemplary embodiment of the present disclosure provides a flow chart of determining the position coordinates corresponding to an interaction event in an immersive browser mode of a virtual reality device. This embodiment takes how to determine the position coordinates corresponding to an interaction event in an immersive browser mode of a virtual reality device as an example for illustrative description. Figure 6 As shown, the following steps are included:

[0167] In step 601, based on the way of triggering the event, a spatial ray is made to assist the event interaction.

[0168] Among them, the event triggering method is used to indicate the method used by the user to trigger the event. For example, the user can trigger the above event through sight when wearing VR glasses, or the user can trigger the above event through a handle.

[0169] The spatial ray used to assist event interaction is a spatial ray used to determine the event triggering position.

[0170] In some implementations, if the user triggers the above event through sight while wearing VR glasses, a ray is drawn along the sight direction with the middle position of the left and right screens of the VR glasses as the starting point as the above space ray. In other implementations, if the user triggers the above event through a handle, a space ray is drawn along the direction of the handle with the handle position as the starting point.

[0171] In some implementations, when an event is triggered in VR three-dimensional space, VR glasses and handle hardware can directly provide ray data such as the handle pointing and line of sight direction.

[0172] In step 602, the intersection point of the ray and the plane geometry is determined.

[0173] In this embodiment, the intersection point between the ray and the surface of the plane geometric body is calculated to obtain the spatial coordinates of the intersection point.

[0174] In step 603, based on the position coordinates of the intersection point on the plane geometric body, the pixel coordinates corresponding to the intersection point in the interactive canvas are determined.

[0175] When the interface canvas is pasted on a plane geometric body, the spatial coordinates of each surface point on the plane geometric body corresponding to the pixel coordinates of each pixel point on the interface canvas are stored. Therefore, according to the determined spatial coordinates of the intersection of the ray and the plane geometric body, the pixel coordinates of the corresponding pixel point on the interface canvas can be calculated.

[0176] In the above steps 601 to 603, in the immersive browser mode of the virtual reality device, when an event is received, a spatial ray is made to obtain the intersection of the ray and the plane geometric body, obtain the spatial coordinates of the intersection, and then obtain the coordinates of the corresponding pixel points on the interface canvas, and determine the pixel coordinates of the interactive event occurring in the target interface, which is helpful for the subsequent determination of the target component where the interaction occurs and realize event interaction.

[0177] Corresponding to the embodiments of the aforementioned interface interaction method, the present disclosure also provides corresponding embodiments of the interface interaction device.

[0178] Exemplary Devices

[0179] Figure 7 FIG. 1 is a schematic diagram of a structure of another embodiment of the interface interaction device disclosed in the present invention. The device is applied to an electronic device (such as a virtual reality device, a computer system, a mobile terminal), such as Figure 7 As shown, the device comprises:

[0180] The reading module 71 is used to read the drawing attribute information of each component and the interactive event information corresponding to each component in the source code of the target interface. The source code is written in the form of components.

[0181] The generating module 72 is used to generate an interface canvas based on the drawing attribute information of each component, in which each component of the target interface is contained; and to generate an interactive canvas based on the position attribute information and the size attribute information of each component, in which the occupied area of ​​each component in the interactive canvas is filled with different filling colors respectively;

[0182] A rendering module 73, used to render a target interface in a browser page based on the interface canvas;

[0183] A coordinate determination module 74, configured to determine the pixel coordinates of the event triggering position in the interactive canvas in response to monitoring an event triggered based on the target interface;

[0184] The event response module 75 is used to obtain the target component targeted by the event based on the pixel coordinates, and execute the response operation of the interactive event corresponding to the target component.

[0185] Figure 8 FIG. 1 is a structural diagram of another embodiment of the interface interaction device disclosed in the present invention. Figure 8 As shown, in Figure 7 Based on the illustrated embodiment, in some embodiments, the generating module 72 includes:

[0186] The first determination submodule 721 is used to determine the actual size information and actual position information of each component based on the size attribute information and position attribute information of each component in the source code and based on the typesetting engine;

[0187] The drawing submodule 722 is used to call the canvas interface function to draw the interface canvas based on the actual size information and actual position information of each component and other drawing attribute information except the size attribute information and the position attribute information.

[0188] In some embodiments, the generation module 72 includes:

[0189] The second determination submodule 723 is used to call the canvas interface function to generate an initial interactive canvas based on the actual size information and actual position information of each component;

[0190] The filling submodule 724 is used to fill the occupied area of ​​each component in the initial interactive canvas with different filling colors.

[0191] In some embodiments, the apparatus further comprises:

[0192] The color mapping module 76 is used to generate a color mapping table, in which the filling color of the occupied area of ​​each component in the interactive canvas is recorded.

[0193] In some embodiments, the event response module 75 includes:

[0194] The first acquisition submodule 751 is used to acquire the fill color corresponding to the pixel coordinates in the interactive canvas;

[0195] The third determining submodule 752 is used to determine the component corresponding to the filling color in the color mapping table as the target component.

[0196] In some embodiments, in the immersive browser mode in the virtual reality device, the rendering module 73 includes:

[0197] A creation submodule 731 is used to create a plane geometric body for carrying the interface canvas;

[0198] The mapping submodule 732 is used to map the interface canvas onto the plane geometry as a texture map;

[0199] The first rendering submodule 733 is used to render a plane geometric body.

[0200] In some embodiments, the coordinate determination module 74 includes:

[0201] The ray submodule 741 is used to make a space ray for assisting event interaction based on the trigger event;

[0202] The intersection point determination submodule 742 is used to determine the intersection point between the ray and the plane geometric body;

[0203] The coordinate determination submodule 743 is used to determine the pixel coordinates of the intersection point in the interactive canvas based on the position coordinates of the intersection point on the plane geometric body.

[0204] In some embodiments, in a web browser, the rendering module 73 includes:

[0205] The element adding submodule 734 is used to add an interface canvas element corresponding to the interface canvas in the document object model of the browser page;

[0206] The second rendering submodule 735 is used to render the target interface based on the interface canvas element in the document object model of the browser page.

[0207] In some embodiments, the apparatus further comprises:

[0208] An event registration module 77, used to register the interaction events of each component based on the interaction event information corresponding to each component;

[0209] The storage module 78 is used to store the interaction events of each component in the heap memory.

[0210] In some embodiments, the event response module 75 is used to obtain the interaction event corresponding to the target component in the heap memory and execute the interaction event.

[0211] The modules and units in the device disclosed in the present invention can also be divided into finer granularity according to actual needs, and can be specifically set according to actual needs.

[0212] The device of the embodiment of the present disclosure can be used to implement the methods of the above-mentioned embodiments of the present disclosure. The two correspond to each other in specific implementation, and the specific implementations of related parts are referenced to each other, which will not be repeated here.

[0213] Exemplary electronic devices, computer program products, and computer-readable storage media

[0214] An embodiment of the present disclosure further provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, the interface interaction method of any of the above embodiments of the present disclosure is implemented.

[0215] Below, reference Fig. 9 An electronic device according to an embodiment of the present disclosure is described, in which a device for implementing the method according to an embodiment of the present disclosure may be integrated. Fig. 9 A structural diagram of an electronic device provided in an exemplary embodiment of the present disclosure, such as Fig. 9 As shown, the electronic device includes one or more processors 91, one or more computer-readable storage media memory 92, and a computer program stored in the memory and executable on the processor. When executing the program in the memory 92, the above interface interaction method can be implemented.

[0216] Specifically, in practical applications, the electronic device may further include components such as an input device 93 and an output device 94, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Those skilled in the art will appreciate that Fig. 9 The structure of the electronic device shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or certain components, or different component arrangements.

[0217] in:

[0218] The processor 91 can be a central processing unit (CPU) or other forms of processing units with interface interaction capabilities and / or instruction execution capabilities. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 92, and calling data stored in the memory 92, thereby monitoring the electronic device as a whole.

[0219] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 91 may run the program instructions to implement the interface interaction methods of the various embodiments of the present disclosure above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0220] The input device 93 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0221] The output device 94 can output various information to the outside, including the determined distance information, direction information, etc. The output device 94 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0222] The electronic device may also include a power supply for supplying power to various components, which may be logically connected to the processor 91 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply may also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0223] Of course, to simplify, Fig. 9 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0224] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the interface interaction method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0225] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0226] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the interface interaction method according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0227] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0228] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0229] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0230] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0231] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0232] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. An interface interaction method, characterized in that: include: Reading drawing attribute information of each component and interaction event information corresponding to each component in the source code of the target interface, wherein the source code is written in the form of components; Based on the drawing attribute information of each component, an interface canvas is generated, in which each component of the target interface is contained; and based on the position attribute information and the size attribute information of each component, an interactive canvas is generated, in which the occupied areas of each component in the interactive canvas are respectively filled with different filling colors; Based on the interface canvas, rendering the target interface in a browser page; In response to monitoring an event triggered based on the target interface, determining the pixel coordinates of the event triggering position in the interactive canvas; Based on the pixel coordinates, the target component targeted by the event is obtained, and a response operation of the interactive event corresponding to the target component is executed.

2. The method according to claim 1, characterized in that: The generating of the interface canvas based on the drawing attribute information of each component includes: Based on the size attribute information and position attribute information of each component in the source code and based on the typesetting engine, determine the actual size information and actual position information of each component; The canvas interface function is called to draw the interface canvas based on the actual size information and actual position information of each component and other drawing attribute information except the size attribute information and the position attribute information.

3. The method according to claim 2, characterized in that The generating of the interactive canvas based on the position attribute information and the size attribute information of each component includes: Call the canvas interface function to generate the initial interactive canvas based on the actual size information and actual position information of each component; The occupied areas of the components in the initial interactive canvas are filled with different filling colors.

4. The method according to any one of claims 1 to 3, characterized in that: After reading the drawing attribute information of each component in the source code of the target interface, the method further includes: A color mapping table is generated, wherein the color mapping table records the fill colors of the occupied areas of the components in the interactive canvas.

5. The method according to claim 4, characterized in that The acquiring, based on the pixel coordinates, a target component targeted by the event, comprises: Get the fill color corresponding to the pixel coordinates in the interactive canvas; The component corresponding to the filling color in the color mapping table is determined as the target component.

6. The method according to any one of claims 1 to 5, characterized in that: In the immersive browser mode of the virtual reality device, rendering the target interface in the browser page based on the interface canvas includes: Creating a plane geometry for carrying the interface canvas; Mapping the interface canvas as a texture map onto the plane geometry; Render the plane geometry.

7. The method according to claim 6, characterized in that In response to monitoring an event triggered based on the target interface, determining the pixel coordinates of the event triggering position in the interactive canvas includes: Based on the method of triggering the event, a spatial ray is made to assist the event interaction; Determining an intersection point between the ray and the plane geometric body; Based on the position coordinates of the intersection point on the plane geometric body, the pixel coordinates corresponding to the intersection point in the interactive canvas are determined.

8. The method according to any one of claims 1 to 5, characterized in that: In a web browser, rendering the target interface in a browser page based on the interface canvas includes: Adding an interface canvas element corresponding to the interface canvas in the document object model of the browser page; The target interface is rendered based on an interface canvas element in a document object model of the browser page.

9. The method according to any one of claims 1 to 8, characterized in that: After reading the drawing attribute information of each component in the source code of the target interface and the interaction event information corresponding to each component, the method further includes: Based on the interaction event information corresponding to each component, register the interaction events of each component; The interaction events of each component are stored in the heap memory.

10. The method according to claim 9, characterized in that The executing the response operation of the interaction event corresponding to the target component includes: Obtain an interaction event corresponding to the target component in the heap memory, and execute the interaction event.

Images