Display device and browser operation method

Abstract

The invention relates to a display device and a browser operation method. The device comprises a browser configured to display a browser page; the communication module is configured to be in communication connection with terminal equipment; a virtual touchpad matched with the browser is installed in the terminal equipment; the user input interface is configured to receive an operation instruction triggered by a user through the virtual touch panel; the browser is configured to perform operation type identification on the operation instruction in response to the operation instruction to obtain the operation type of the operation instruction; determining an operation response object matched with the operation type from candidate response objects of the browser; when the operation response object comprises the user interface component of the browser, sending the operation instruction to the user interface component based on a message transmission mechanism of the browser; and executing the operation instruction through the user interface component. The operation convenience of the browser can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a browser operation method. Background Art

[0002] When a display device plays media such as video and audio, it can be presented through a browser. For example, the display device can present an application interface for media playback through a browser, and detailed media information can be displayed in the media playback application interface.

[0003] When a user accesses a web page or views media resources through a browser in a display device, the user usually needs to perform various page operations such as sliding, clicking, or page zooming.

[0004] In traditional technology, users are accustomed to using the remote control of the display device to operate the browser page. However, the remote control generally requires pressing buttons one by one to control the cursor in the browser page to move, which is cumbersome and easily reduces the convenience of browser operation. Summary of the invention

[0005] The present application provides a display device and a browser operation method to solve the problem of low convenience of browser operation in the prior art.

[0006] In a first aspect, some embodiments provide a display device, comprising: a browser, configured to display a browser page; a communication module, configured to communicate with a terminal device; a virtual touchpad matching the browser is installed in the terminal device; a user input interface, configured to receive operation instructions triggered by a user through the virtual touchpad; the browser is configured to: in response to the operation instruction, identify the operation type of the operation instruction to obtain the operation type of the operation instruction; determine an operation response object that matches the operation type from the candidate response objects of the browser; when the operation response object includes a user interface component of the browser, send the operation instruction to the user interface component based on the message transmission mechanism of the browser; and execute the operation instruction through the user interface component.

[0007] Through the above display device, the user can control the browser page in the display device through the virtual touchpad installed in the mobile terminal. There is no need to perform cumbersome button operations through the remote control, which effectively improves the convenience of browser operation. In addition, different operation response objects are configured for different types of operations, so that various types of operations can be processed and responded to in a targeted manner, thereby improving the response rate of operation instructions. When the operation response object includes the user interface component of the browser, it involves the message transmission inside the browser, so the operation instruction can be further sent to the user interface component based on the message transmission mechanism of the browser to increase the transmission rate of the operation instruction inside the browser, thereby improving the response rate of the operation instruction, making the response process of the operation instruction smoother and more convenient, thereby improving the convenience of browser operation.

[0008] In the second aspect, some embodiments further provide a browser operation method, which is applied to the display device provided in the first aspect, and the method includes: in response to an operation instruction triggered by a user through a virtual touchpad, identifying the operation type of the operation instruction to obtain the operation type of the operation instruction; determining an operation response object that matches the operation type from the candidate response objects of the browser; when the operation response object includes a user interface component of the browser, sending the operation instruction to the user interface component based on the message transmission mechanism of the browser; and executing the operation instruction through the user interface component.

[0009] According to the above-mentioned browser operation method, the user can control the browser page in the display device through the virtual touchpad installed in the mobile terminal. There is no need to perform cumbersome button operations through the remote control, which effectively improves the convenience of browser operation. In addition, different operation response objects are configured for different types of operations, so that various types of operations can be processed and responded to in a targeted manner, thereby improving the response rate of operation instructions. When the operation response object includes the user interface component of the browser, it involves the message transmission inside the browser, so the operation instruction can be further sent to the user interface component based on the message transmission mechanism of the browser to increase the transmission rate of the operation instruction inside the browser, thereby improving the response rate of the operation instruction, making the response process of the operation instruction smoother and more convenient, thereby improving the convenience of browser operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0012] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0013] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0014] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0015] Figure 5 A schematic diagram of displaying media data on a browser page provided in some embodiments of the present application;

[0016] Figure 6 A virtual touchpad page of a mobile terminal provided in some embodiments of the present application;

[0017] Figure 7 A text input page of a virtual touchpad provided in some embodiments of the present application;

[0018] Figure 8 A schematic diagram of a communication connection between a mobile terminal and a display device provided in some embodiments of the present application;

[0019] Fig. 9 A schematic diagram of a sliding operation flow diagram provided for some embodiments of the present application;

[0020] Fig.10 A schematic diagram of a click operation flow provided for some embodiments of the present application;

[0021] Fig.11 A schematic diagram of a flow chart of a page zoom operation provided in some embodiments of the present application;

[0022] Fig.12 Installation prompt information of the virtual touchpad provided in some embodiments of the present application;

[0023] Fig.13 A schematic diagram of display device connection in a mobile terminal provided in some embodiments of the present application;

[0024] Fig.14 A schematic diagram of connecting a verification code in a mobile terminal provided in some embodiments of the present application;

[0025] Fig.15 A schematic diagram showing a default mouse provided in some embodiments of the present application;

[0026] Fig.16A flowchart of the interaction between various devices during the browser operation process provided in some embodiments of the present application;

[0027] Fig.17 A flowchart of a browser operation method provided for some embodiments of the present application. DETAILED DESCRIPTION

[0028] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0029] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0030] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0031] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0032] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0033] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0034] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through the mobile terminal 300 and the control device 100. For example, the control device 100 may be a remote controller, a handle, or the like.

[0035] The mobile terminal 300 can be used as a control device for performing human-computer interaction between a user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0036] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks.

[0037] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function of a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0038] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0039] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0040] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0041] In some embodiments, the display 260 includes a display function component for presenting a picture, a touch component connected to the display 260, and a driving component for driving the image display. The display 260 is used to receive the image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0042] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0043] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0044] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0045] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0046] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0047] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0048] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0049] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0050] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0051] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0052] In some embodiments, Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0053] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0054] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0055] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0056] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0057] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0058] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0059] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0060] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0061] The operating system can be divided into different modules or layers according to the functions implemented.

[0062] like Figure 4 As shown, Figure 4 Some embodiments of the present application provide Figure 1 Schematic diagram of software configuration in the display device. In some embodiments, the system of the display device 200 can be divided into three layers, namely, application layer, middleware layer and hardware layer from top to bottom.

[0063] The application layer mainly includes commonly used applications on TV and application frameworks. Common applications are mainly applications developed based on browsers, such as HTML5 APPs and native APPs.

[0064] The Application Framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces of these functions (toolbars, status bars, menus, dialog boxes).

[0065] Native apps can support online or offline, message push or local resource access.

[0066] The middleware layer includes various TV protocols, multimedia protocols, system components and other middleware. Middleware can use the basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, and can achieve the purpose of resource sharing and function sharing.

[0067] The hardware layer mainly includes the hardware abstraction layer (HAL) interface, hardware and drivers. The HAL interface is a unified interface for all TV chips to connect, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver.

[0068] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0069] When a display device plays video, audio and other media, it can be presented through a browser. For example, a display device can present an application interface for media playback through a browser, and detailed media information can be displayed in the media playback application interface. When a user accesses a web page or watches media through a browser in a display device, it is usually necessary to perform various page operations such as sliding, clicking, or page zooming. In traditional technology, users are accustomed to using a remote control of a display device to perform browser page operations. However, a remote control generally requires pressing buttons one by one to control the movement of the cursor in the browser page. The operation process is relatively cumbersome, which can easily reduce the convenience of browser page operations.

[0070] In some embodiments, a display device is provided, the display device comprising:

[0071] A browser is configured to display a browser page.

[0072] Among them, the browser is a browser application configured in the display device, and the user can search and browse media data through the browser application. The browser page is the display page of the browser, and the page content of the display page can specifically be media data. Media data is various forms of content resources in the media industry and various types of data and information related thereto. In terms of content form, media data includes different types of materials such as audio, video, pictures, and text. For example, TV program tapes, radio station audio files, news agency photo libraries, and text content of various publications. Media data can specifically be data stored locally on the display device, or data received through the network.

[0073] Optionally, when displaying media data, the browser may display it in a mixed arrangement of media data. Mixed arrangement of media data can be understood as mixed arrangement and display of various types of media data, and each type of media data may correspond to a display template, such as a pure text template, a 1:1 large picture plus information template, a 1:1 small picture plus information template, a 16:9 large picture plus information template, and a 16:9 small picture plus information template.

[0074] A pure text template refers to a template that only uses text elements and does not include any pictures or decorative elements. A 1:1 large picture plus information template refers to a template that uses a large picture of the same size as the original media picture as the background, and adds text or other information to it. A 1:1 small picture plus information template refers to a template that uses a small picture that is a result of scaling down the original media picture in a 1:1 ratio as the background, and adds text or other information to it. A 16:9 large picture plus information template refers to a template that uses an image that is amplified by the original media picture in a 16:9 aspect ratio as the background, and adds text or other information elements to it. A 16:9 small picture plus information template refers to a template that uses an image that is a result of scaling down the original media picture in a 16:9 aspect ratio as the background, and adds text or other information elements to it. Figure 5 The schematic diagram of displaying media data on a browser page in this embodiment is shown. Of course, in actual applications, other media display forms can be selected according to actual conditions, and other media display templates can also be defined, which is not limited in this embodiment.

[0075] Optionally, the browser includes an odin browser. Of course, in actual applications, other types of browsers, such as a vewd browser, may also be configured in the display device.

[0076] For example, when a user opens a browser in a display device for the first time, the browser will first display a pop-up window for installing a virtual touchpad on the main interface, so that the user can install the virtual touchpad in the mobile terminal based on the pop-up window. After the user successfully installs the virtual touchpad, the browser can display the main interface in full screen, which can include a media search bar and real-time media data display. The browser can control the changes of the main interface in real time based on the operation instructions initiated by the user on the virtual touchpad.

[0077] Optionally, the browser may also record the installation record of the virtual touchpad installed on each mobile terminal. In the case where the virtual touchpad has been installed on the mobile terminal, the browser page may not display the virtual touchpad installation pop-up window, but directly display the main interface. For example, after the user installs the virtual touchpad on a mobile terminal, the browser will record the model or other unique identifier of the mobile terminal and mark it as "virtual touchpad installed". The specific form of the mark is not limited, as long as it can indicate that the mobile terminal has installed the virtual touchpad.

[0078] The communication module is configured to communicate with the terminal device; the terminal device is installed with a virtual touchpad matching the browser.

[0079] Among them, the communication module is a communicator of the display device, which is used to communicate with the terminal device. The specific form of the communication module is not limited, as long as it can realize the communication connection between the display device and the mobile terminal. The virtual touchpad refers to an interface that can be operated by touch. The user can slide, click, zoom, etc. on the interface with his fingers to simulate mouse operations. There is a one-to-one correspondence between the virtual touchpad and the browser, that is, the virtual touchpad is only used to control the browser page corresponding to it.

[0080] For example, after the user installs the virtual touchpad on the mobile terminal, a communication connection with the display device is required to transmit the operation instruction. After the mobile terminal successfully establishes a communication connection with the display device, the touchpad page of the virtual touchpad can be displayed on the terminal page of the mobile terminal, such as Figure 6 As shown, the user can trigger various operation instructions in the operation area in the touchpad page, such as click, slide, or page zoom operation, etc. In actual application, the virtual touchpad page can also prompt the user that the virtual touchpad can only be used in the browser of the display device.

[0081] Optionally, when the user needs to search for media data, he can move the virtual mouse on the browser page to the search bar by moving it in the operation area of ​​the virtual touchpad page. Then the user can click in the operation area, and the virtual touchpad page can jump to the text input page, such as Figure 7 The text input page may include a text input bar and an operation area, and various command operations can also be performed in the operation area. When the user clicks the text input bar, the input keyboard will be displayed below the text input page. The content entered by the user on the text input page will be synchronously displayed in the search bar of the browser page.

[0082] Optionally, after receiving the text input content from the user, the browser can search for media data based on the text input content, and display the searched media data on the browser page. It should be noted that the browser is not limited to the channel for searching media data, and can query from the local media database or from the cloud.

[0083] Optionally, the mobile terminal and the display device can communicate with each other via MQTT (Message Queuing Telemetry Transport). Figure 8As shown in the figure, in the MQTT protocol, the mobile terminal can be used as a message publisher, that is, the party that publishes the operation instructions, and the display device can be used as a message subscriber, that is, the party that subscribes to the operation instructions. There is also a proxy server between the mobile terminal and the display device, which plays a role in message forwarding, that is, pushing the operation instructions of the mobile terminal to the display device, and the display device can receive these operation instruction messages.

[0084] The MQTT protocol supports two modes: "recording the token validity period" and "not recording the token validity period". Recording the token validity period means that when the communication connection is established, a token within the validity period will be exchanged between the mobile terminal and the display device. The token is used to verify the identities of both parties in subsequent sessions. And this token has a clear validity period. When the two parties reconnect during the validity period, there is no need to re-authenticate. Not recording the token validity period means that a complete authentication operation is required for each communication connection.

[0085] It should be noted that in actual applications, other communication mechanisms may also be selected to achieve the communication connection between the mobile terminal and the display device.

[0086] The user input interface is configured to receive operation instructions triggered by the user through the virtual touch pad.

[0087] The user input interface is an interface that can receive operation instructions initiated by the user. The operation instruction received by the user input interface may be one of a click instruction, a slide instruction, a page zoom instruction, etc. The user input interface may send the received operation instruction to the display device, which distributes it to the browser, and the browser may execute the operation instruction.

[0088] Optionally, the user input interface can first send the received operation instructions to the server (service) module of the display device. The server module can parse the operation instructions and obtain the operation events of the operation instructions. Then the server module will send the operation events to the DFB module. The DFB module can be understood as a module that specializes in processing mouse operation events. The DFB module will convert the operation events into actual actions on the browser page. Finally, the DFB module sends the converted message to the browser engine of the browser, which will display it. The browser engine, also known as the browser kernel, is a set of application software components that process markup languages, such as Hypertext Markup Language (HTML), Extensible Markup Language (XML), Extensible Stylesheet Language (XSL), image files, etc., and process formatting information, such as Cascading Style Sheets (CSS), etc., and display content.

[0089] For example, Fig. 9 As shown, take the sliding operation as an example. First, the user initiates a sliding operation instruction on the virtual touchpad of the mobile terminal. The sliding operation instruction will first enter the server module of the display device. The server module can be understood as a control center for managing various operation instructions. The server module will parse the sliding operation instruction to obtain the specific event content of the sliding operation instruction, such as the sliding range, sliding position, etc. The specific event content will be sent to the DFB module, and the DFB module will analyze the current mouse display position in the browser page based on the sliding range and sliding position. The DFB module will send this information to the browser engine, and the browser engine can directly display the mouse at the display position of the browser page.

[0090] The browser is configured to: in response to the operation instruction, identify the operation type of the operation instruction to obtain the operation type of the operation instruction.

[0091] The operation type refers to the type of operation instruction. It is understandable that the operation instruction may include but is not limited to a click operation instruction, a slide operation instruction, a page zoom instruction, etc. A click operation refers to clicking on a browser page, a slide operation refers to sliding a browser page, and a page zoom refers to reducing / enlarging a browser page.

[0092] In some embodiments, the operation type may include a first type of operation and a second type of operation. The first type of operation refers to an operation in which the position of the mouse in the browser page changes, such as a sliding operation. It can be understood that the sliding operation is an operation of sliding the browser page, which essentially involves the change of the position of the mouse in the browser page. The second type of operation refers to an operation in which there is page interaction and / or logical judgment, such as a click operation, page zooming, etc. Page interaction refers to the jump between different pages. When a user initiates a click operation instruction, a page jump is generally involved. For example, when a user clicks on a control or page element, the browser page will jump to the page corresponding to the control or page element, or display the display content corresponding to the control or page element. Logical judgment means that when executing the operation instruction, a logical judgment needs to be made. For example, when zooming the page, it is necessary to determine whether the current browser page belongs to a third-party page.

[0093] Exemplarily, when the browser receives an operation instruction forwarded by the display device, it can identify the type of the operation instruction. It can be judged based on the content of the operation event for the operation instruction sent by the display device. For example, when the content of the operation event includes the sliding range, sliding position, etc., the browser may consider the operation to be a sliding operation, and then confirm that the operation instruction belongs to the first type of operation. When the content of the operation event includes the address of the page to be zoomed, the browser may consider the operation to be a page zoom operation, and then confirm that the operation instruction belongs to the second type of operation. When the content of the operation event includes clicking on a control, etc., the browser may consider the operation to be a click operation, and then confirm that the operation instruction belongs to the second type of operation. Of course, in actual applications, other methods can also be selected to distinguish the types of operation instructions, and this embodiment does not limit it.

[0094] From the candidate response objects of the browser, determine the operation response object that matches the operation type.

[0095] Different operation types correspond to different operation response objects. Candidate response objects refer to the various component objects in the browser that respond to and execute operation instructions, such as the browser engine, the user interface component (UI) of the browser, etc. The operation response object is one or more component objects that ultimately respond to and execute the operation instruction initiated by the current user.

[0096] Exemplarily, there are multiple component objects in the browser that can be used to respond to and execute the operation instruction. The browser will perform response object matching according to the operation type of the operation instruction to filter out one or more component objects used to execute the operation instruction from the multiple component objects.

[0097] Optionally, the browser may pre-establish an association relationship between the operation type and the candidate response object, and the form of the association relationship is not limited. After identifying the operation type of the operation instruction, the browser may directly query the association relationship to determine the operation response object corresponding to the operation instruction.

[0098] When the operation response object includes a user interface component of the browser, the operation instruction is sent to the user interface component based on the message transmission mechanism of the browser.

[0099] Among them, the user interface component is also the UI component of the browser. The user interface component interacts with page elements in the browser page, such as buttons, input boxes, pop-up windows, etc., and can quickly and accurately obtain information about each page element. Since the user interface component needs to obtain the operation event of the operation instruction from the browser engine, in order to further improve the response rate of the operation instruction, the message transmission mechanism of the browser is introduced in this embodiment. The message transmission mechanism of the browser can be understood as a communication mechanism for message transmission between component objects inside the browser. Taking the Odin browser as an example, the message transmission mechanism of the Odin browser is the OMI communication mechanism, and the operation event of the operation instruction will be transmitted to the user interface component in the form of an OMI message. It should be noted that different browsers have different corresponding message transmission mechanisms.

[0100] For example, when the operation response object of the operation instruction includes the user interface component of the browser, it means that the message transmission between the browser engine and the user interface component is involved, that is, the browser engine obtains the operation event information from the DFB module and sends the operation event information to the user interface component. In this process, the browser engine sends the operation event information to the user interface component based on the message transmission mechanism of the browser.

[0101] like Fig.10 As shown, Fig.10 Take a click operation instruction as an example. After the DFB module sends the operation event information of the click operation instruction to the browser engine, the browser engine can send the operation event information to the user interface component through the message transmission mechanism of the browser, and the user interface component executes the click operation.

[0102] Optionally, the operation instruction can also be transmitted in other component objects inside the browser, and is not limited to the transmission between the browser engine and the user interface component. When the operation instruction is transmitted between other component objects, it can also be implemented based on the message transmission mechanism of the browser.

[0103] Execute operation instructions through user interface components.

[0104] Exemplarily, after the user interface component receives the operation event information for the operation instruction sent by the browser based on the message transmission mechanism, the user interface component can execute the operation instruction based on the operation event information.

[0105] In this embodiment, the user can control the browser page in the display device through the virtual touchpad installed in the mobile terminal. There is no need to perform cumbersome button operations through the remote control, which effectively improves the convenience of browser operation. In addition, different operation response objects are configured for different types of operations, so that various types of operations can be processed and responded to in a targeted manner, thereby improving the response rate of operation instructions. When the operation response object includes the user interface component of the browser, it involves the message transmission inside the browser, so the operation instruction can be further sent to the user interface component based on the message transmission mechanism of the browser to increase the transmission rate of the operation instruction inside the browser, thereby improving the response rate of the operation instruction, making the response process of the operation instruction smoother and more convenient, thereby improving the convenience of browser operation.

[0106] In some embodiments, in the process of determining the operation response object that matches the operation type, the browser is further configured as follows: when the operation type belongs to the first category of operations, the operation response object that matches the operation type is determined to be the browser engine; the first category of operations refers to operations in which the mouse position in the browser page changes; when the operation type belongs to the second category of operations, the operation response object that matches the operation type is determined to be the browser engine and the user interface component of the browser; the second category of operations refers to operations involving page interaction and / or logical judgment.

[0107] For example, if the operation type belongs to the first type of operation, that is, the operation of changing the mouse position, the corresponding operation response object is the browser engine. It can be understood that since the browser engine is usually responsible for the content rendering and content display of the browser page, including the position and style of the mouse, the first type of operation can be directly processed by the first type of operation, which can improve the response rate of the first type of operation.

[0108] Exemplarily, if the operation type belongs to the second type of operation, that is, an operation involving page interaction and / or logical judgment, the corresponding operation response object is a user interface component. Unlike the first type of operation, the second type of operation involves page elements. The user interface component interacts directly with the page elements of the browser page, so the user interface component can handle the second type of operation more accurately and improve the response rate of the first type of operation. It should be noted that since the user interface component needs to obtain the operation instruction through the browser engine, the operation response object of the second type of operation also needs to include the browser engine.

[0109] Optionally, the browser may pre-establish an association relationship between the first type of operation and the candidate response object, and an association relationship between the second type of operation and the candidate response object, and the form of the association relationship is not limited. If the browser recognizes that the operation instruction belongs to the first type of operation, it can directly query the association relationship between the first type of operation and the candidate response object, and determine the operation response object corresponding to the operation instruction. If the browser recognizes that the operation instruction belongs to the second type of operation, it can directly query the association relationship between the second type of operation and the candidate response object, and determine the operation response object corresponding to the operation instruction.

[0110] It should be noted that in practical applications, other operation types and the number of operation types may be defined according to actual conditions, and this embodiment does not limit this.

[0111] In this embodiment, considering that the first type of operation is an operation in which the mouse position changes, matching it with a browser engine as an operation response object can improve the response rate of the first type of operation. The second type of operation refers to an operation involving page interaction and / or logical judgment, which involves page elements, so matching it with a user interface component as an operation response object can improve the response rate of the second type of operation.

[0112] In some embodiments, when the browser executes an operation instruction through a user interface component, the browser is further configured to: determine whether the current browser page is a third-party page through the user interface component; if so, obtain the page zoom parameters of the page zoom instruction; and execute the page zoom instruction according to the page zoom parameters.

[0113] Among them, third-party pages refer to web pages developed by third parties, and can also be understood as pages that are not built into the browser, such as web pages of other video websites or shopping websites. Page zoom parameters refer to the corresponding parameters of page zoom operations, such as page zoom ratio, page address to be zoomed, etc.

[0114] It is understandable that the browser supports the display of third-party pages, that is, the user can access the pages of other external websites through the browser of the display device. In this case, page zooming will involve the zooming of the browser page and the zooming of the third-party web page. It should be noted that the browser of the display device does not support the zooming of built-in pages, but supports the zooming of third-party web pages. This is because the browser page needs to ensure a clear interface display, which requires an adaptive display device screen layout, so the page cannot be zoomed at will. Therefore, when performing a page zoom operation, the browser needs to make a third-party page judgment, that is, to judge whether the current browser page belongs to the browser's built-in page or a third-party web page.

[0115] Exemplarily, when the operation instruction is a page zoom instruction, it belongs to the second type of operation, so the browser engine will send the operation event information of the page zoom instruction to the user interface component. The user interface component will determine whether the current browser page belongs to a third-party page. If it belongs to a third-party page, it means that the page can be zoomed, and then the page zoom operation is performed on the current browser page according to the page zoom parameter. If the current browser page does not belong to a third-party page, that is, the current browser page is a page that comes with the browser, it means that the page cannot be zoomed, and the user interface component does not process the operation instruction.

[0116] Optionally, when determining whether the current browser page is a third-party page, the user interface component may determine based on page parameters of the third-party page, such as page title, page source, page content, page identifier, etc. Of course, other methods may also be used, such as obtaining the page code of the current browser page to determine whether it is a third-party page. This embodiment does not limit the specific method for determining a third-party page.

[0117] In this embodiment, a third-party page judgment is performed on the current browser page to determine whether the page zoom operation can be performed on the current browser page, thereby ensuring effective execution of the page zoom operation.

[0118] In some embodiments, when the browser executes operation instructions according to page zoom parameters, the browser is further configured to: send the page zoom parameters to the browser engine through the user interface component based on the browser's message transmission mechanism; and execute the page zoom instructions through the browser engine according to the page zoom parameters.

[0119] It should be noted that page zooming actually changes the display size of the page, so the page zooming operation can be performed by the browser engine. The judgment of third-party pages involves page elements, such as obtaining the page title, page source, page content, page logo, etc., which are performed by the user interface component.

[0120] For example, if the user interface component determines that the current browser page belongs to a third-party page, it will generate corresponding page zoom parameters including the page zoom ratio, the address of the page to be zoomed, etc. Since the page zoom operation is ultimately performed by the browser engine, the user interface component needs to return these page zoom parameters to the browser engine, which also requires a browser-based message transmission mechanism. After the browser engine receives these page zoom parameters, it can perform the page zoom operation on the current browser page.

[0121] Optionally, after obtaining the page zoom parameter, the browser engine can extract the page address to be zoomed in the page zoom parameter, and compare the page address with the address of the current browser page for consistency. If they are consistent, the current browser page is zoomed. If they are inconsistent, the page zoom operation is not performed. In this way, the page zoom parameter is tampered with during the transmission process, which causes the page zoom operation to fail, and the accuracy of the page zoom operation is ensured.

[0122] like Fig.11 As shown, after the browser engine sends the operation event information of the page zoom instruction to the user interface component based on the message transmission mechanism of the browser, the user interface component determines whether the current browser page belongs to a third-party page. If so, the user interface component returns the generated page zoom parameter to the browser engine based on the message transmission mechanism of the browser, and the browser engine zooms the current browser page. If not, the user interface component does not process it.

[0123] In this embodiment, when the user interface component sends the page zoom parameter to the browser engine, it can also be implemented based on the message transmission mechanism of the browser, which can increase the transmission rate of the page zoom parameter and thus increase the response rate of the page zoom operation.

[0124] In some embodiments, when determining whether the current browser page is a third-party page, the browser is further configured to: obtain the page identifier and page content of the current browser page; and determine whether the current browser page is a third-party page based on the page identifier and page content.

[0125] The page identifier refers to the unique identifier of the current browser page, such as the page address, page number, page code, etc. The page content refers to the information and elements displayed on the current browser page, such as pictures, text, video, audio, etc.

[0126] Exemplarily, when the user interface component makes a third-party page judgment, it can extract the unique identifier of the current browser page and the information and elements displayed on the current browser page. It is judged whether the unique identifier belongs to the identifier of the browser's native page. If not, it is considered that the current browser page belongs to a third-party page; if it does, it is considered that the current browser page does not belong to a third-party page. Or it is judged whether the information and elements displayed on the current browser page belong to the information and elements of the browser's native page. If not, it is considered that the current browser page belongs to a third-party page; if it does, it is considered that the current browser page does not belong to a third-party page.

[0127] Optionally, the user interface component may determine that the current browser page does not belong to a third-party page when the unique identifier of the current browser page belongs to the identifier of the browser's native page and the information and elements displayed by the current browser page belong to the information and elements of the browser's native page. The user interface component may determine that the current browser page belongs to a third-party page when the unique identifier of the current browser page does not belong to the identifier of the browser's native page and / or the information and elements displayed by the current browser page do not belong to the information and elements of the browser's native page.

[0128] Optionally, the user interface component can simultaneously obtain the page identifier and page content of the current browser page. Alternatively, the page identifier can be obtained first to determine whether it is an identifier of a page that comes with the browser. If it does not belong to the browser, it can be directly considered that the current browser page belongs to a third-party page, and the page content is no longer obtained. In the case of an identifier that belongs to a page that comes with the browser, the page content can continue to be obtained to determine whether the page content is information and elements of the browser's built-in page. Alternatively, the page identifier can be obtained first and then the page content. This embodiment does not limit the order in which the page identifier and page content are obtained.

[0129] For example, the identifiers of the browser's built-in page include A, B, and C, and the information displayed on the browser's built-in page includes information 1, information 2, and information 3. Assuming that the unique identifier of the current browser page is A, which is an identifier of the browser's built-in page, the current browser page can be considered to be a browser's built-in page. If the unique identifier of the current browser page is D, which is not an identifier of the browser's built-in page, the current browser page can be considered to be a third-party page. Similarly, assuming that the information of the current browser page is information 1, which is information displayed by the browser's built-in page, the current browser page can be considered to be a browser's built-in page. If the information of the current browser page is information 4, which is not information displayed by the browser's built-in page, the current browser page can be considered to be a third-party page.

[0130] It should be noted that, in actual applications, it is also possible to determine whether the current browser page is a third-party page based on other characteristic information of the current browser page, and this embodiment does not limit this.

[0131] In this embodiment, the user interface component obtains the page identifier and page content of the current browser page, and then determines whether the current browser page is a third-party page based on the page identifier and page content, thereby improving the accuracy and reliability of third-party page judgment, thereby improving the accuracy and reliability of page zooming operations.

[0132] In some embodiments, the browser is further configured to: display installation prompt information for installing a virtual touchpad on a terminal device on a browser page of the browser; in response to a communication connection request initiated by the terminal device to a display device after the virtual touchpad is installed based on the identified installation prompt information, display a connection verification code for the communication connection request; the connection verification code is used for communication between the terminal device and the display device.

[0133] The installation prompt information refers to the installation information of the virtual touchpad, which may include the installation QR code of the virtual touchpad, the page operations supported by the virtual touchpad, and the triggering methods of these page operations. Fig.12 As shown, Fig.12 The installation prompt information of the virtual touchpad is shown. Fig.12 The installation prompt information is displayed in the form of a pop-up window. Of course, in actual applications, other methods can be selected for display, and even jumping to a new page for display. This embodiment does not limit this. Fig.12 In the figure, the left side is the installation QR code of the virtual touchpad, and the right side prompts that the virtual touchpad supports click, double-click, slide, and zoom operations, and the corresponding icons are the triggering methods of these operations. For example, the triggering method of click is to click the operation area with a finger, double-click is to click the operation area twice with a finger, slide is to slide the finger up and down in the operation area, and zoom is to slide two fingers outward or inward in the operation area. Of course, in actual applications, other page operations and shortcuts for page operation triggering can also be configured, and this embodiment does not limit it.

[0134] A communication connection request refers to a communication connection request initiated by a mobile terminal to a display device to request a communication connection with the display device. It is understandable that after the user completes the installation of the virtual touchpad in the mobile terminal, if the browser page needs to be controlled through the virtual touchpad, a communication connection between the mobile terminal and the display device is also required. The connection verification code is a verification code used to achieve a communication connection between a terminal device and a display device, and can be a combination of one or more of the forms of letters, numbers, symbols, etc. The length of the verification code is not limited, and a four-digit verification code is selected in this embodiment.

[0135] Optionally, in order to facilitate users to find the installation entrance of the virtual touchpad more quickly, the installation entrance of the virtual touchpad can be placed in the toolbar of the browser. After the user clicks the installation entrance, the browser page can display the installation prompt information. It can also be placed in other obvious locations in the browser page, or the display size of the installation entrance can be enlarged. The specific method is not limited in this embodiment.

[0136] For example, after the user enters the main interface of the browser, the user can find the installation entry of the virtual touchpad in the toolbar of the browser page. After the user clicks the installation entry, the installation prompt information of the virtual touchpad will be displayed in the form of a pop-up window in the main interface of the browser. Through the installation prompt information, the user can know the various page operations for the browser page that the virtual touchpad can support, as well as the triggering method of each page operation. The user scans the installation QR code in the installation prompt information to download and install the virtual touchpad application.

[0137] After the virtual touchpad is successfully installed, the terminal page of the mobile terminal can display the display devices that can be connected to the current LAN, such as Fig.13 The user chooses to connect to the device whose name is currently displayed. After receiving the communication connection request initiated by the user, the display device will display a connection verification code on the browser page. At this time, a pop-up window for entering the verification code will be displayed on the mobile terminal page, as shown in the figure below. Fig.14 As shown. The user enters the connection verification code displayed on the browser page in the pop-up window to establish a communication connection between the mobile terminal and the display device. The mobile terminal can automatically jump to the virtual touchpad usage page.

[0138] Optionally, the user may also search for a virtual touchpad application APP in the application management center of the display device, and download and install it.

[0139] Optionally, the installation prompt information of the virtual touchpad can also be automatically displayed after the user enters the main interface of the browser, that is, the user does not need to click the installation entrance. The user can choose to scan the code to download or exit the pop-up window.

[0140] In this embodiment, by displaying the installation prompt information of the virtual touchpad on the browser page, the user can install the virtual touchpad more conveniently, thereby improving the convenience of installing the virtual touchpad. In addition, during the communication connection between the mobile terminal and the display device, the connection verification code is used to ensure the security of the communication connection between the mobile terminal and the display device.

[0141] In some embodiments, the installation QR code is applicable to terminal devices of Android system and terminal devices of iOS system. That is, no matter it is a mobile terminal of Android system or a terminal device of iOS system, the installation QR code can be scanned to realize the installation of the virtual touchpad.

[0142] In some embodiments, the browser is further configured to: display a default mouse on the browser page when the communication connection between the display device and the terminal device is disconnected; and the default mouse moves along with the focus frame on the browser page.

[0143] The default mouse refers to the default mouse on the browser page, and the virtual touchpad corresponds to the virtual mouse. The focus frame refers to the frame that displays the focus information or focus position on the browser page. Fig.15 As shown in the figure, the frame corresponding to the first page element is the focus frame, and the lower right corner of the focus frame is the default mouse on the browser page. The default mouse moves with the focus frame on the browser page, that is, if the focus frame moves from the first page element to the second page element, the default mouse will also move to the lower right corner of the second page element.

[0144] Exemplarily, when the communication connection between the display device and the terminal device is disconnected, that is, the user cannot control the browser page through the virtual touchpad. At this time, the browser can automatically display the default mouse, which is located in the lower right corner of the focus frame. When the user selects other focus frames through the remote control, the default mouse will also move with the movement of the focus frame. It should be noted that the switching of mouse styles is also applicable to wired mouse scenarios. When a wired mouse, that is, a USB mouse, is inserted into the display device, the browser displays a virtual mouse. When the wired mouse is unplugged from the display device, the browser displays the default mouse.

[0145] In this embodiment, when the communication connection between the mobile terminal and the display device is disconnected, the browser displays a default mouse, which can ensure the smoothness of the browser operation.

[0146] In some embodiments, the browser is further configured to: execute the operation instruction through the browser engine when the operation response object does not include the user interface component of the browser.

[0147] Exemplarily, when the operation response object does not include the user interface component of the browser, it means that the operation instruction belongs to the first type of operation, that is, the operation of changing the mouse position in the browser page. At this time, the browser engine directly executes the operation instruction, which can improve the response rate of the operation.

[0148] In a specific embodiment, Fig.16As shown, the display device is configured with an Odin browser, and there is a corresponding virtual touchpad application in the Odin browser. After the user enters the main interface of the Odin browser, the Odin browser can display the installation prompt information of the virtual touchpad, and the user can download and install the virtual touchpad in the mobile terminal based on the installation prompt information. After successfully installing the virtual touchpad, the user selects the current display device on the display device connection page for communication connection. After the display device receives the communication connection request sent by the mobile terminal, a four-digit verification code can be displayed on the Odin browser page. The user enters the verification code in the mobile terminal page to realize the communication connection between the mobile terminal and the display device. The display device and the mobile terminal are connected via the MQTT protocol.

[0149] The user can initiate click operations, sliding operations, page zooming operations, and text input operations on the browser page through the virtual touchpad. The operation instructions will be sent to the server module of the display device through the mobile terminal, and then parsed by the server module and sent to the DFB module. The DFB module generates operation event information and sends the operation event information to the browser engine of the Odin browser. The browser engine identifies the type of the operation instruction based on the operation event information. If the operation instruction belongs to a sliding operation, the browser engine, as the operation response object, performs a sliding operation on the browser page.

[0150] If the operation instruction is a click operation, the operation response object includes the user interface component. The browser engine sends the operation event information to the user interface component in the form of an ODI message based on the message transmission mechanism of the ODIN browser, and the user interface component performs the click operation on the browser page.

[0151] If the operation instruction belongs to a page zoom operation, the operation response object includes a user interface component. The browser engine sends the operation event information to the user interface component based on the message transmission mechanism of the Odin browser, and the user interface component obtains the page parameters of the current browser page. Based on the page parameters, it is determined whether the current browser page belongs to a third-party page. If it does, a page zoom parameter is generated, and based on the message transmission mechanism of the Odin browser, the page zoom parameter is sent to the browser engine, and the browser engine performs a page zoom operation on the browser page. If the current browser page does not belong to a third-party page, no processing is performed.

[0152] In this embodiment, the user can control the browser page in the display device through the virtual touchpad installed in the mobile terminal. There is no need to perform cumbersome button operations through the remote control, which effectively improves the convenience of browser operation. In addition, different operation response objects are configured for different types of operations, so that various types of operations can be processed and responded to in a targeted manner, thereby improving the response rate of operation instructions. When the operation response object includes the user interface component of the browser, it involves the message transmission inside the browser, so the operation instruction can be further sent to the user interface component based on the message transmission mechanism of the browser to increase the transmission rate of the operation instruction inside the browser, thereby improving the response rate of the operation instruction, making the response process of the operation instruction smoother and more convenient, thereby improving the convenience of browser operation.

[0153] Based on the same inventive concept, the embodiment of the present application also provides a browser operation method applied to the above-mentioned display device. The implementation solution provided by the method to solve the problem is similar to the implementation solution recorded in the above-mentioned display device embodiment, so the specific limitations in one or more browser operation embodiments provided below can refer to the above-mentioned limitations on the display device, and will not be repeated here.

[0154] In some embodiments, the present application further provides a browser operation method, which is applied to the above-mentioned display device. Fig.17 As shown, the browser operation method includes the following steps:

[0155] Step S1702, in response to an operation instruction triggered by a user through a virtual touchpad of a terminal device, identifying the operation type of the operation instruction to obtain the operation type of the operation instruction; a communication connection is established between the terminal device and the display device, and the virtual touchpad matches the browser of the display device;

[0156] Step S1704, determining an operation response object that matches the operation type from the candidate response objects included in the browser;

[0157] Step S1706, when the operation response object includes a user interface component of the browser, the operation instruction is sent to the user interface component based on the message transmission mechanism of the browser;

[0158] Step S1708, executing the operation instruction through the user interface component.

[0159] In some embodiments, determining the operation response object that matches the operation type includes: when the operation type belongs to the first category of operations, determining that the operation response object that matches the operation type is the browser engine; the first category of operations refers to operations in which the mouse position in the browser page changes; when the operation type belongs to the second category of operations, determining that the operation response object that matches the operation type is the browser engine and the user interface component of the browser; the second category of operations refers to operations involving page interaction and / or logical judgment.

[0160] In some embodiments, the operation instruction includes a page zoom instruction; executing the operation instruction through the user interface component includes: determining through the user interface component whether the current browser page belongs to a third-party page; if so, obtaining the page zoom parameter of the page zoom instruction; and executing the page zoom instruction according to the page zoom parameter.

[0161] In some embodiments, executing a page zoom instruction according to page zoom parameters includes: sending the page zoom parameters to a browser engine through a user interface component based on a message transmission mechanism of the browser; and executing the page zoom instruction according to the page zoom parameters through the browser engine.

[0162] In some embodiments, determining whether the current browser page is a third-party page includes: obtaining a page identifier and page content of the current browser page; and determining whether the current browser page is a third-party page based on the page identifier and page content.

[0163] In some embodiments, the browser operation method also includes: displaying installation prompt information for installing a virtual touchpad on a terminal device on a browser page of the browser; in response to a communication connection request initiated by the terminal device to a display device after the virtual touchpad is installed based on the identified installation prompt information, displaying a connection verification code for the communication connection request; the connection verification code is used for communication between the terminal device and the display device.

[0164] In some embodiments, the installation prompt information includes an installation QR code for the virtual touchpad; the installation QR code is applicable to terminal devices of Android system and terminal devices of iOS system.

[0165] In some embodiments, the browser operation method further includes: when the communication connection between the display device and the terminal device is disconnected, displaying a default mouse on the browser page; and the default mouse moves following the focus frame on the browser page.

[0166] In some embodiments, the browser operation method further includes: when the operation response object does not include a user interface component of the browser, executing the operation instruction through the browser engine.

[0167] In some embodiments, the present application further provides a browser operation device, which is applied to the above-mentioned display device. In this embodiment, the browser operation device includes:

[0168] An operation type identification module is used to identify the operation type of the operation instruction in response to the operation instruction triggered by the user through the virtual touch panel of the terminal device, and obtain the operation type of the operation instruction; the terminal device and the display device have established a communication connection, and the virtual touch panel matches the browser of the display device;

[0169] An operation response object determination module is used to determine an operation response object that matches the operation type from candidate response objects contained in the browser;

[0170] A message transmission module, used to send the operation instruction to the user interface component based on the message transmission mechanism of the browser when the operation response object includes the user interface component of the browser;

[0171] The operation instruction execution module is used to execute the operation instruction through the user interface component.

[0172] In some embodiments, the module for determining the operation response object is also used for: when the operation type belongs to the first category of operation, determining that the operation response object that matches the operation type is the browser engine; the first category of operation refers to an operation in which the mouse position in the browser page changes; when the operation type belongs to the second category of operation, determining that the operation response object that matches the operation type is the browser engine and the user interface component of the browser; the second category of operation refers to an operation involving page interaction and / or logical judgment.

[0173] In some embodiments, the operation instruction includes a page zoom instruction; the operation instruction execution module is also used to: determine whether the current browser page is a third-party page through the user interface component; if so, obtain the page zoom parameter of the page zoom instruction; and execute the page zoom instruction according to the page zoom parameter.

[0174] In some embodiments, the operation instruction execution module is also used to: send the page zoom parameter to the browser engine through the user interface component based on the message transmission mechanism of the browser; and execute the page zoom instruction according to the page zoom parameter through the browser engine.

[0175] In some embodiments, the operation instruction execution module is further used to: obtain the page identifier and page content of the current browser page; and determine whether the current browser page is a third-party page based on the page identifier and page content.

[0176] In some embodiments, the device is also used to: display installation prompt information for installing a virtual touchpad on a terminal device on a browser page of a browser; in response to a communication connection request initiated by the terminal device to a display device after the virtual touchpad is installed based on the identified installation prompt information, display a connection verification code for the communication connection request; the connection verification code is used for the terminal device to establish a communication connection with the display device.

[0177] In some embodiments, the installation prompt information includes an installation QR code for the virtual touchpad; the installation QR code is applicable to terminal devices of Android system and terminal devices of iOS system.

[0178] In some embodiments, the apparatus is further used to: display a default mouse on a browser page when the communication connection between the display device and the terminal device is disconnected; and the default mouse moves along with a focus frame on the browser page.

[0179] In some embodiments, when the operation response object does not include a user interface component of the browser, the operation instruction is executed through the browser engine.

[0180] Each module in the above-mentioned browser operating device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0181] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method steps are implemented.

[0182] In some embodiments, a computer program product is provided, comprising a computer program, which implements the above method steps when executed by a processor.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0184] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A display device, characterized in that: include: a browser, configured to display a browser page; A communication module configured to communicate with a terminal device; the terminal device is installed with a virtual touchpad matching the browser; A user input interface, configured to receive an operation instruction triggered by a user through the virtual touch panel; The browser is configured to: In response to the operation instruction, identifying the operation type of the operation instruction to obtain the operation type of the operation instruction; Determining an operation response object matching the operation type from candidate response objects of the browser; When the operation response object includes a user interface component of the browser, sending the operation instruction to the user interface component based on a message transmission mechanism of the browser; The operation instruction is executed through the user interface component.

2. The display device according to claim 1, characterized in that In the process of determining the operation response object matching the operation type, the browser is further configured to: In the case where the operation type belongs to a first type of operation, determining that the operation response object matching the operation type is a browser engine; the first type of operation refers to an operation in which a mouse position in the browser page changes; When the operation type belongs to the second type of operation, the operation response object matching the operation type is determined to be the browser engine and the user interface component of the browser; the second type of operation refers to an operation involving page interaction and / or logical judgment.

3. The display device according to claim 1, characterized in that The operation instruction includes a page zoom instruction; and the browser is further configured to: Determining whether the current browser page belongs to a third-party page through the user interface component; If yes, obtaining the page zoom parameter of the page zoom instruction; The page zoom instruction is executed according to the page zoom parameter.

4. The display device according to claim 3, characterized in that In the process of executing the operation instruction according to the page zoom parameter, the browser is further configured to: Sending the page zoom parameter to the browser engine through the user interface component based on the message transmission mechanism of the browser; The page zoom instruction is executed by the browser engine according to the page zoom parameter.

5. The display device according to claim 3, characterized in that In the process of determining whether the current browser page is a third-party page, the browser is further configured to: Get the page ID and page content of the current browser page; Based on the page identifier and the page content, it is determined whether the current browser page belongs to a third-party page.

6. The display device according to claim 1, characterized in that The browser is further configured to: Displaying, on the browser page of the browser, installation prompt information for installing a virtual touchpad on a terminal device; In response to the communication connection request initiated by the terminal device to the display device after installing the virtual touchpad based on the identified installation prompt information, a connection verification code for the communication connection request is displayed; the connection verification code is used for the terminal device to communicate with the display device.

7. The display device according to claim 6, characterized in that The installation prompt information includes an installation QR code for the virtual touch panel; the installation QR code is applicable to terminal devices of Android system and terminal devices of iOS system.

8. The display device according to claim 1, characterized in that The browser is further configured to: When the communication connection between the display device and the terminal device is disconnected, a default mouse is displayed on the browser page; the default mouse moves along with the focus frame on the browser page.

9. The display device according to claim 1, characterized in that: The browser is further configured to: When the operation response object does not include the user interface component of the browser, the operation instruction is executed through the browser engine.

10. A browser operation method, characterized in that: The method comprises: In response to an operation instruction triggered by a user through a virtual touchpad of a terminal device, an operation type identification is performed on the operation instruction to obtain an operation type of the operation instruction; the terminal device and the display device have established a communication connection, and the virtual touchpad matches the browser of the display device; Determining an operation response object matching the operation type from candidate response objects included in the browser; When the operation response object includes a user interface component of the browser, sending the operation instruction to the user interface component based on a message transmission mechanism of the browser; The operation instruction is executed through the user interface component.

Images