Display device and voice search method based on in-site page

Abstract

The display device and the voice search method based on an intrawebsite page provided in the application can, when displaying an intrawebsite page, respond to a voice search instruction sent by a user, acquire a DOM tree of the intrawebsite page, acquire candidate input label elements with a type attribute value of search and a type attribute value of text in the DOM tree, determine a target input label element in an activated state from the candidate input label elements, and then determine a search box on the intrawebsite page based on the target input label element to provide an input basis for filling in a keyword. It can be seen that the display device can automatically locate the search box on the intrawebsite page and fill in the keyword to complete corresponding search without additional interaction with the user in response to the voice search instruction of the user, so that the user operation can be simplified, the voice search efficiency of the intrawebsite page is improved, and the experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent display devices, and in particular to a display device and a voice search method based on an in-site page. BACKGROUND

[0002] A display device refers to a terminal device capable of outputting a specific display screen, and can be a smart television, a mobile terminal, a smart advertising screen, a projector, or the like. Taking a smart television as an example, a smart television is based on Internet application technology, has an open operating system and a chip, has an open application platform, and has a bidirectional man-machine interaction function, and integrates audio, entertainment, data, and other functions into one, and is used to meet the diversified and personalized needs of users.

[0003] In order to improve the experience of users, the display device provides a network search function for users, and users can perform network search through a television browser. In order to facilitate the interaction between users and the display device, the display device is configured to provide a voice interaction function for users, that is, the display device can collect and respond to voice instructions sent by users to perform corresponding functions. Thus, users can send voice instructions to the display device to perform network search in the television browser.

[0004] The television browser is a search software provided by the display device, and the display device stores related information of a search page provided by the television browser, such as the specific position of a search box on the search page. Therefore, when users send a voice search instruction based on the television browser, the display device can accurately identify the position of the search box based on the pre-stored related information of the search page, and input the keywords in the voice search instruction to complete the corresponding search work and enter the corresponding website. Users need to further search in the website, but these websites provide in-site pages by different operators, the forms of the in-site pages are diverse, and the display device is difficult to pre-know and store the related information of the in-site pages. Therefore, if users send a voice search instruction based on an in-site page, the display device cannot automatically locate the position of the search box in the in-site page, and the user needs to focus the focus on the search box, and then the display device can input the keywords in the search box to complete the search work in the website. SUMMARY

[0005] The present application provides a display device and a voice search method based on an in-site page, which automatically locates a search box on an in-site page by identifying a label element of the in-site page, without focusing the focus by the user, so as to simplify the interaction process between the user and the display device and improve the efficiency of voice search in the website.

[0006] In a first aspect, the present application provides a display device, comprising:

[0007] a display configured to display an in-site page;

[0008] a controller configured to:

[0009] receive a voice search instruction sent by a user when displaying the in-site page, the voice search instruction indicating to display a page corresponding to a keyword;

[0010] obtain a Document Object Model (DOM) tree of the in-site page in response to the voice search instruction;

[0011] obtain all candidate input elements in the DOM tree, the candidate input elements including an input element with a type attribute value of search or an input element with a type attribute value of text;

[0012] determine a target input element, the target input element being one of the candidate input elements in an activated state;

[0013] determine a search box of the in-site page, and fill the keyword in the search box to perform a search based on the keyword, the search box corresponding to the target input element.

[0014] In a second aspect, the present application provides a search method of an in-site page, applied to a display device, the method comprising:

[0015] receiving a voice search instruction sent by a user when displaying the in-site page, the voice search instruction indicating to display a page corresponding to a keyword;

[0016] obtaining a Document Object Model (DOM) tree of the in-site page in response to the voice search instruction;

[0017] obtaining all candidate input elements in the DOM tree, the candidate input elements including an input element with a type attribute value of search or an input element with a type attribute value of text;

[0018] determining a target input element, the target input element being one of the candidate input elements in an activated state;

[0019] determining a search box of the in-site page, and filling the keyword in the search box to perform a search based on the keyword, the search box corresponding to the target input element.

[0020] According to the technical solution, when the display device displays the webpage in the website, the display device can respond to the voice search instruction sent by the user, acquire the DOM tree of the webpage in the website, acquire the candidate input tag elements with the type attribute value of search and the type attribute value of text in the DOM tree, determine a target input tag element in an active state from the candidate input tag elements, and then determine the search box on the webpage in the website based on the target input tag element to provide an input basis for filling in the keyword. It can be seen that the display device can automatically locate the search box on the webpage in the website and fill in the keyword to complete the corresponding search in response to the voice search instruction of the user without additional interaction with the user, which can simplify the user operation, improve the voice search efficiency of the webpage in the website, and improve the experience of the user. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0022] Figure 1 The use scenario of the display device in the embodiments of the present application;

[0023] Figure 2 The hardware configuration block diagram of the control device in the embodiments of the present application;

[0024] Figure 3 The hardware configuration block diagram of the control device in the embodiments of the present application;

[0025] Figure 4 The hardware configuration diagram of the display device in the embodiments of the present application;

[0026] Figure 5 The software configuration diagram of the display device in the embodiments of the present application;

[0027] Figure 6 The flowchart of the display device responding to the voice search instruction based on the webpage in the embodiments of the present application;

[0028] Figure 7 The flowchart of the display device acquiring the candidate input tag elements in the embodiments of the present application;

[0029] Figure 8 The flowchart of the display device determining the target input tag element from the candidate input tag elements in the embodiments of the present application;

[0030] Figure 9A flowchart for determining the input label element to be transferred for the display device in the embodiments of the present application;

[0031] Figure 10 A schematic diagram of the relative position between the search box and the viewport in the embodiments of the present application;

[0032] Figure 11 A flowchart for adjusting the viewport for the display device in the embodiments of the present application. DETAILED DESCRIPTION

[0033] The embodiments will be described in detail below with reference to the drawings. When the description below refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of implementations consistent with some aspects of the present application as detailed in the claims.

[0034] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the implementations described next, and is not intended to limit the implementations of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0035] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or like objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0036] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not have to be limited to all components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0037] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware or / and software code capable of performing a function associated with that element.

[0038] Figure 1 A schematic diagram of the use scenario of the display device according to the embodiments. As shown in Figure 1 The display device 200 can communicate with the server 300 over the Internet, and the user can operate the display device 200 through the control device 100.

[0039] In some embodiments, the control device 100 can be a remote controller, and the communication between the remote controller and the display device includes at least one of infrared protocol communication or Bluetooth protocol communication, and other short-distance communication modes, to control the display device 200 in a wireless or wired manner. The user can control the display device 200 by inputting control instructions through at least one of a button on the remote controller, voice input, control panel input, and the like.

[0040] In some embodiments, the control device 100 can also be a mobile terminal, such as a mobile phone or the like, and the communication between the mobile terminal and the display device 200 includes at least one of Internet protocol communication or Bluetooth protocol communication, and other short-distance communication, long-distance communication modes. The user can control the display device 200 by inputting user instructions through at least one of a button on the mobile terminal, voice input, control panel input, and the like. Figure 2 An exemplary configuration block diagram of the control device 100 is shown as a remote controller. As shown, the control device 100 includes a controller, a communication interface, a user input / output interface, a memory, a power supply. Figure 2

[0041] Figure 3 An exemplary configuration block diagram of the control device 100 is shown as a mobile terminal. As shown, the control device 100 includes at least one of a radio frequency (RF) circuit, a memory, a display unit, a camera, a sensor, an audio circuit, a Wireless Fidelity (Wi-Fi) circuit, a processor, a Bluetooth circuit, and a power supply. Figure 3

[0042] Figure 4 A hardware configuration block diagram of the display device 200 according to an exemplary embodiment is shown.

[0043] In some embodiments, the display device 200 includes at least one of a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0044] In some embodiments, the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator can include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the control device 100 or the server 300 through the communicator 220.

[0045] ​​In some embodiments, the external device interface 240 can include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It can also be a composite input / output interface formed by a plurality of the above interfaces.

[0046] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different devices, i.e., the tuner demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0047] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations by storing various software control programs in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object displayed on the display 260, the controller 250 can perform an operation related to the object selected by the user command.

[0048] In some embodiments, the user can input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0049] In some embodiments, the "user interface" is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form that the user can accept. The commonly used form of the user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an icon, a window, a control, etc. displayed on the display screen of an electronic device, wherein the control can include at least one of the following visual interface elements: an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a widget, etc.

[0050] Referring to Figure 5 In some embodiments, the system is divided into four layers, from top to bottom, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the Android runtime and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0051] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0052] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications in the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0053] like Figure 5 As shown, the application framework layer in this embodiment includes managers, content providers, etc., wherein the managers include at least one of the following modules: ActivityManager, which interacts with all activities running in the system; LocationManager, which provides access to system location services for system services or applications; PackageManager, which retrieves various information related to application packages currently installed on the device; NotificationManager, which controls the display and clearing of notification messages; and WindowManager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0054] The user sends control commands to the display device 200 via the control device 100 to instruct the display device 200 to access the search page of the TV browser. For example, the user can press the "Up," "Down," "Left," and "Right" buttons on the remote control to move the focus to the TV browser icon or option, and press the "OK" button on the remote control to instruct the display of the TV browser's search page. Alternatively, the user can send a voice command to the display device 200 via the control device 100, such as saying "Open browser," to instruct the display device 200 to display the TV browser's search page. Or, the user can directly send a voice command to the display device 200, such as saying "Open browser," and the display device 200 will use a voice recognition device to collect and recognize the user's voice content to display the TV browser's search page. This embodiment does not limit the specific method by which the user instructs the display device 200 to display the TV browser's search page.

[0055] In this embodiment, the TV browser refers to a browser that has pre-stored the relevant information of the search page to the display device 200, and the relevant information of the search page at least includes the position information of the search box such as the address bar and the search bar, so that the display device 200 can automatically locate the search box based on the pre-stored relevant information of the search page.

[0056] After the display device 200 displays the search page of the TV browser, the user sends a voice search instruction based on the search page of the TV browser, for example, “open www. × × × (website). com”, and the display device 200 can automatically locate the search box based on the pre-stored relevant information of the search page of the TV browser in response to the voice search instruction, and input the content in the voice instruction into the search box to perform the corresponding search.

[0057] The response result of the display device 200 to the voice search instruction sent by the user based on the search page of the TV browser is to display the intranet page of the corresponding website. For example, a shopping website, a video website, etc. In this embodiment, the intranet page is provided by the server of the website, and the intranet page presents diversity based on user demand, operator demand, etc. and dynamically changes, and the display device 200 needs to request the relevant information of the intranet page from the server in real time. That is, the display device 200 cannot pre-store the relevant information of the intranet page, and accordingly, it is impossible to automatically locate the search box based on the pre-stored relevant information of the intranet page. Therefore, if the user wants to perform voice search based on the intranet page, the user needs to first interact with the display device 200 to locate the search box of the intranet page, and the user sends a voice search instruction based on the located search box, and the display device 200 responds to the voice search instruction to fill the keywords into the search box to perform the search.

[0058] The embodiment provides a voice search method based on an intranet page, and the display device 200 can automatically locate the search box of the intranet page to avoid additional interaction between the user and the display device 200 and improve the voice search efficiency of the intranet page. Referring to the voice search flow of the intranet page shown in the figure, the specific process is as follows: Figure 6

[0059] S601, when displaying the intranet page, receiving a voice search instruction sent by a user, the voice search instruction instructing to display a page corresponding to a keyword.

[0060] ​The user sends a control instruction to the display device 200 through the control device 100 to instruct the display device 200 to display a search page of the TV browser. The user sends a search instruction to the display device 200 based on the search page of the TV browser to instruct the display device 200 to display an intraweb page of a corresponding website. The above process can refer to the process of the user searching for an intraweb page through the TV browser disclosed above, and details are not repeated here.

[0061] The display device 200 receives the search instruction sent by the user and obtains the search content therefrom, for example, “open www.xxx (website).com”. The display device 200 obtains a Hyper Text Markup Language (HTML) document data packet corresponding to “xxx (website)” from a corresponding server in response to the search instruction. The display device 200 obtains page information constituting an intraweb page corresponding to “xxx (website)” by parsing the HTML document data packet, for example, each element displayed on the page, position information of each element on the page, display state information, etc. The display device 200 can present the HTML document data packet as a tree structure with elements, attributes, and text through an HTML Document Object Model (DOM), i.e., a DOM tree. For example, the HTML DOM converts each component in the HTML document data packet into a node, for example, a document node (document object) representing the entire document, an element node (element object) representing an element (tag), a text node (text object) representing text in the element (tag), an attribute node (attribute object) representing an attribute, an element (tag) having an attribute, a comment node (comment object) representing a comment, etc. There is a dependency relationship between these nodes. The HTML DOM constructs a DOM tree based on the dependency relationship between the nodes. The DOM tree has a corresponding relationship with the HTML web page. The display device 200 can render the corresponding HTML web page based on the DOM tree, i.e., display the intraweb page.

[0062] The user sends a voice search instruction to the display device 200 based on the intraweb page to instruct the display device 200 to display a page corresponding to a keyword. For example, the user inputs a voice search instruction such as “search page ***” to the microphone of the control device 100 and transmits the voice search instruction to the display device 200 through the control device 100 to instruct the display device 200 to display a page corresponding to the keyword “***”.

[0063] S602, in response to the voice search instruction, obtaining a file object model DOM tree of the intraweb page.

[0064] After the display device 200 receives the voice search instruction sent by the user, the display device 200 determines that the voice search instruction is a search instruction by identifying the keyword, such as "search", in the voice search instruction, that is, the display device 200 needs to locate the search box in the webpage in the website and fill the keyword into the located search box to perform subsequent search work.

[0065] As known from the above, the DOM tree has a corresponding relationship with the HTML page, that is, the webpage in the website in the embodiment. The display device 200 can obtain the DOM tree corresponding to the webpage in the website based on the corresponding relationship.

[0066] S603, obtain all candidate input tag elements in the DOM tree, the candidate input tag elements including an input tag element with a type attribute value of search or an input tag element with a type attribute value of text.

[0067] As known from the above, the element nodes in the DOM tree indicate the elements in the HTML page. The element corresponding to the search box is indicated by the element node indicating input in the DOM tree, that is, the input tag element. The input tag element can indicate multiple types of elements, such as a simple text input box and a search box. The specific type of the input tag element is described by the type attribute value. In the embodiment, the search box should obviously have a type attribute value of search. In some embodiments, the search box must also be able to fill in text, and therefore the search box also needs to have a type attribute value of text. Therefore, the display device 200 can locate the search box by the candidate input tag elements with the type attribute value of search and the type attribute value of text. Correspondingly, the display device 200 obtains the two types of input tag elements with the type attribute value of search and the type attribute value of text from the obtained DOM tree as the candidate input tag elements.

[0068] In some embodiments, the display device 200 can obtain all the candidate input tag elements in the DOM tree by referring to the process shown in FIG. 8A: Figure 7

[0069] S701, obtain all input tag elements in the DOM tree and the type attribute value of each input tag element.

[0070] The display device 200 finds and obtains all the input tag elements in the DOM tree according to the tag name "input", and identifies the type attribute value in each input tag element according to the attribute value name "type".​

[0071] S702, screen out the input tag elements with type attribute value of search and type attribute value of text from the all input tag elements.

[0072] For each input tag element, the type attribute value recognized by the display device 200 can include multiple types, such as search, text, read only, disabled, etc. As known from the above, the search box corresponds to the type attribute value of search or the type attribute value of text. Therefore, first, the input tag elements with the type attribute value of search or the type attribute value of text are screened out from the input tag elements as the candidate input tag elements.

[0073] S703, delete the input tag elements with read only attribute value or disabled attribute value from the candidate input tag elements to obtain the candidate input tag elements.

[0074] The candidate input tag elements also include attribute values describing the display state of the search box. The input tag elements in the state of not inputting text need to be removed from all candidate input tag elements to ensure that the candidate input tag elements screened out are all in the state of inputting text.

[0075] In this embodiment, the candidate input tag elements with attribute values of read only and disabled are removed. The candidate input tag element with the attribute value of read only indicates a state of being able to display default text or a cursor in the search box, but not being able to input text. The candidate input tag element with the attribute value of disabled indicates a state of not displaying any text or cursor in the search box, and not being able to input text. In addition to the attribute values indicating the state of not inputting text disclosed in this embodiment, the candidate input tag elements with other attribute values indicating the state of not inputting text also need to be removed. Thus, it can be ensured that the positioned search box is a search box that can provide the basis for text input, i.e. the search box can be normally filled with keywords, and the display device 200 can display the search box with keywords to effectively interact with the user.

[0076] S604, determine a target input tag element, which is one of the candidate input tag elements in the active state.

[0077] In the embodiment, the attribute value of the input tag element further includes an attribute value indicating a response state, for example, an attribute value active, and an attribute value inactive, wherein the attribute value active indicates that the input tag element can currently provide services, i.e., can obtain corresponding data in response to a search instruction, and the like, and vice versa, the attribute value inactive indicates that the input tag element cannot currently provide services, i.e., cannot obtain corresponding data in response to a search instruction, or can only be in a dormant state for data synchronization, and the like.

[0078] In the embodiment, a candidate input tag element in an active state is selected from the candidate input tag elements as a target input tag element, so that the located search box is a search box in a state that can respond to a search keyword instruction, to ensure that a page corresponding to the keyword can be displayed subsequently.

[0079] In some embodiments, the target input tag element can be determined from the candidate input tag elements according to the flow as shown in Figure 8

[0080] S801, a first array and a second array are constructed, the first array includes first type input tag elements, and the second array includes second type input tag elements, wherein the first type input tag elements are candidate input tag elements with a type attribute value of search, and the second type input tag elements are candidate input tag elements with a type attribute value of text.

[0081] According to the type attribute value, the candidate input tag elements are divided into two types, i.e., the first type input tag elements with a type attribute value of search, and the second type input tag elements with a type attribute value of text, and the first array is constructed with the first type input tag elements, for example, the first array can be named as a search element array, and the second array is constructed with the second type input tag elements, for example, the second array can be named as a text element array.

[0082] S802, a to-be-transferred input tag element is determined, the to-be-transferred input tag element is a second type input tag element with a specified attribute value containing a search string, wherein the specified attribute value includes an attribute value specified in the second type input tag element, and an attribute value of each parent node tag element within 4 levels of the second type input tag element in the DOM tree.

[0083] ​The second-type input tag elements in the second array are further divided according to the specified attribute values containing the search string. The second-type input tag elements do not directly show the search attribute in the type attribute value, but hide the search attribute in the specified attribute value. The to-be-migrated input tag elements carrying the search string in the specified attribute value are screened from the second-type input tag elements, so as to further divide the to-be-migrated input tag elements into the first-type input tag elements, further subdivide the candidate input tag elements, ensure that all candidate input tag elements with the search attribute can be identified, and ensure the comprehensiveness of the positioning of the search box range and the positioning accuracy of the search box.

[0084] In the embodiment, the specified attribute value includes the specified attribute value contained in the second-type input tag element itself and the attribute value of the parent node tag element within 4 levels in the DOM tree of the second-type input tag element. The to-be-migrated input tag element can be determined according to the flow as shown in FIG. 8B. Figure 9

[0085] S8021, removing the second-type input tag element with the attribute value login from the second array to obtain a compliant second-type input tag element.

[0086] For some second-type input tag elements, the attribute value login is carried, which indicates that the second-type input tag element is a text box for providing a login-related service. For example, the text box indicated by the second-type input tag element can input text, but after the text is input, the text box provides a login service based on the text, that is, the input text is a username, a password, and the like, instead of a keyword for searching. In order to avoid positioning the search box to these text boxes for login, the display device 200 first removes the second-type input tag element with the attribute value login from the second array. Further, if there are attribute values of the second-type input tag elements including text boxes indicating other functions, these second-type input tag elements are also removed from the second array, so as to ensure that the text boxes indicated by the second-type input tag elements in the second array are all search boxes for providing a search service. For convenience of description, the second-type input tag elements remaining in the second array after the removal work are referred to as compliant second-type input tag elements.

[0087] ​S8022, identify whether the attribute value specified in the compliant second-type input tag element carries the search string. If the attribute value carries the search string, the corresponding compliant second-type input tag element is determined as the input tag element to be transferred. If the attribute value does not carry the search string, it is determined whether the attribute value of each parent node tag element within 4 levels of the compliant second-type input tag element in the DOM tree carries the search string, and the compliant second-type input tag element carrying the search string is determined as the input tag element to be transferred.

[0088] According to the order of the second-type input tag elements added to the second array, the order of adding each second-type input tag element is generated, and based on the order of adding, each compliant second-type input tag element also has a corresponding order of adding. According to the order of adding, each compliant second-type input tag element is detected whether it is an input tag element to be transferred. In some embodiments, each compliant second-type input tag element can also be detected whether it is an input tag element to be transferred according to other orders.

[0089] The identification process of a compliant second-type input tag element is described as follows:

[0090] First, it is identified whether the attribute value specified in the compliant second-type input tag element carries the search string. In this embodiment, the specified attribute value can be id, class, name, etc. If it is identified that the specified attribute value carries the search string, it can be determined that the compliant second-type input tag element is an input tag element to be transferred. If it is not identified that the specified attribute value carries the search string, it is determined whether the attribute value of each parent node tag element within 4 levels of the compliant second-type input tag element in the DOM tree carries the search string. The specific process can be referred to the following flow:

[0091] If the attribute value of the parent node tag element within 4 levels in the DOM tree is identified to have the search string, it is determined that the compliant second-type input tag element is a to-be-migrated input tag element. If the attribute value of the parent node tag element within 4 levels in the DOM tree is not identified to have the search string, the compliant second-type input tag element is not a to-be-migrated input tag element. Alternatively, in the process of judging each level of parent node tag element, the tag name of the parent node tag element (the currently judged parent node tag element) is first obtained, and if the tag name is body or form, it is directly determined that the compliant second-type input tag element is not a to-be-migrated input tag element, and the subsequent level-by-level judgment process is terminated immediately.

[0092] Each compliant second-type input tag element in the second array is judged according to the above process to determine each to-be-migrated input tag element.

[0093] S803, migrating the to-be-migrated input tag element from the second array to the first array.

[0094] Each to-be-migrated input tag element is migrated from the second array to the first array in the judgment order, and each to-be-migrated input tag element corresponds to an addition order in the first array. In order to facilitate subsequent description, the first array after completing the migration of the to-be-migrated input tag element is referred to as the migrated first array, and the second array after completing the migration of the to-be-migrated input tag element is referred to as the migrated second array.

[0095] The input tag element in the migrated first array includes one or more types of the first-type input tag element with the type attribute value of search, the compliant second-type input tag element with the search string in the specified attribute value, and the compliant second-type input tag element with the search string in the attribute value of the parent node tag element within 4 levels in the DOM tree.

[0096] The input tag element in the migrated second array includes the compliant second-type input tag element with the type attribute value of text, without the search string in the specified attribute value, and without the search string in the attribute value of the parent node tag element within 4 levels in the DOM tree, or with the tag name of body or form.

[0097] S804, determining the target input tag element in the migrated first array.

[0098] Firstly, it is identified whether the number of input label elements in the first array after the transfer is 0. Based on the number of input label elements in the first array, the target input label element is determined in a corresponding manner, specifically as follows:

[0099] In an implementation manner, if the number of input label elements in the first array after the transfer is greater than 0, the target input label element is determined from the input label elements in the first array after the transfer. For example, the response state of each input label element is identified, and one input label element in an activated state is determined as the target input label element. If there are multiple input label elements in the activated state, one input label element in the activated state can be randomly selected as the target input label element; or a specified input label element can be selected as the target input label element, for example, the last input label element that is added to the first array after the transfer and is in the activated state is selected as the target input label element.

[0100] In some embodiments, if all the input label elements in the first array after the transfer are in the inactivated state, the first input label element that is added to the first array after the transfer is taken as the target input label element.

[0101] In an implementation manner, if the number of input label elements in the first array after the transfer is equal to 0, all the input label elements in the second array after the transfer are transferred to the first array to obtain the first array after the second transfer. If the number of input label elements in the second array after the transfer is also 0, it is indicated that there is no target input label element, i.e., there is no search box on the page in the website. The process of determining the target input label element in the first array after the second transfer can refer to the process of determining the target input label element in the first array after the transfer in the above implementation manner, which will not be described herein again.

[0102] S605, a search box of the page in the website is determined, and the keyword is filled in the search box to perform a search based on the keyword. The search box corresponds to the target input label element.

[0103] The target input tag element can be accurately determined by the above process, and the target input tag element indicates a search box on the webpage in the website. Based on the target input tag element, the display device 200 can locate the search box on the webpage in the website, fill in the keyword (e.g., " *** ") carried in the voice search instruction in the search box, and obtain the HTML document data packet corresponding to the keyword " *** " based on the search service provided by the search box, and display the corresponding next-level page.

[0104] According to the above technical solution, when displaying the webpage in the website, the display device can respond to the voice search instruction sent by the user, obtain the DOM tree of the webpage in the website, obtain the candidate input tag elements with the type attribute value of search and the type attribute value of text in the DOM tree, determine a target input tag element in an active state from the candidate input tag elements, and then determine the search box on the webpage in the website based on the target input tag element to provide an input basis for filling in the keyword. It can be seen that the display device can automatically locate the search box on the webpage in the website and fill in the keyword to complete the corresponding search in response to the voice search instruction of the user without additional interaction with the user, which can simplify the user operation, improve the voice search efficiency of the webpage in the website, and improve the user experience.

[0105] The viewport represents a currently visible computer graphics area, and the size of the viewport is usually the same as the size of the display window of the display device 200. When the overall size of the webpage in the website is larger than the size of the viewport, i.e., part of the webpage in the website cannot be displayed in the viewport, the position of the viewport needs to be moved to move the part of the webpage in the website to the viewport, and then the part of the webpage in the website is displayed.

[0106] In this embodiment, when the overall size of the webpage in the website is larger than the size of the viewport, part of the webpage in the website cannot be displayed in the viewport, and the search box is located on the part of the webpage in the website, the display device 200 can determine the target input tag element based on the search box, and then fill in the keyword in the search box based on the target input tag element. Figure 10Fig. 1 shows a schematic diagram of the position relationship between the search box and the viewport, wherein the search box located outside the viewport can be roughly divided into 8 categories, for example, the upper left area outside the viewport, such as search box A; the upper area outside the viewport, such as search box B; the upper right area outside the viewport, such as search box C; the left area outside the viewport, such as search box D; the right area outside the viewport, such as search box E; the lower left area outside the viewport, such as search box F; the lower area outside the viewport, such as search box G; and the lower right area outside the viewport, such as search box H. At this time, the user cannot know whether the display device 200 successfully locates the search box, whether the display device 200 fills in the keyword in the search box, and whether the filled keyword is correct, etc., resulting in poor interaction between the user and the display device 200.

[0107] The viewport can be adjusted by referring to the flow shown in Fig. 2 to display the search box in the adjusted viewport: Figure 11

[0108] S1101, obtaining first position information corresponding to the search box in the DOM tree and second position information corresponding to the current viewport.

[0109] The renderer tree is a part of the DOM tree for reflecting the rendering information of the element on the website page, and the rendering information includes position information, color information, shape information, etc. on the website page. Therefore, the display device 200 can obtain the first position information corresponding to the search box by obtaining the renderer tree corresponding to the search box from the DOM tree after locating the search box, and further obtain the first position information corresponding to the search box through the renderer tree. For example, the first position information can be represented as (left1, top1, width1, height1).

[0110] In this embodiment, the viewport of the display device 200 when receiving the voice search instruction sent by the user is called the current viewport, and the position information corresponding to the current viewport is called the second position information. The position of the viewport can be adjusted by the scroll bar 400 shown in Fig. 3, and thus the position information of the viewport is determined by the position information corresponding to the scroll bar. For example, the second position information can be represented as (left2, top2, width2, height2). Figure 9

[0111] S1102, judging whether the search box is located in the current viewport based on the first position information and the second position information.

[0112] By comparing the first position information and the second position information, the position of the search box relative to the current viewport can be determined. The specific position is as follows:

[0113] ​​(1) If left1 < left2, top1 < top2, it means the search box is in the top-left area outside the current viewport, such as Figure 10 Search box A in Fig. 1;

[0114] (2) If left1 < left2, and left1 + width1 < left2 + width2, top1 < top2, it means the search box is in the top area outside the current viewport, such as Figure 10 Search box B in Fig. 1;

[0115] (3) If left1 + width1 > left2 + width2, top1 < top2, it means the search box is in the top-right area outside the current viewport, such as Figure 10 Search box C in Fig. 1;

[0116] (4) If left1 < left2, top1 > top2, and top1 + height1 < top2 + height2, it means the search box is in the left area outside the current viewport, such as Figure 10 Search box D in Fig. 1;

[0117] (5) If left1 + width1 > left2 + width2, top1 > top2, and top1 + height1 < top2 + height2, it means the search box is in the right area outside the current viewport, such as Figure 10 Search box E in Fig. 1;

[0118] (6) If left1 < left2, top1 + height1 > top2 + height2, it means the search box is in the bottom-left area outside the current viewport, such as Figure 10 Search box F in Fig. 1;

[0119] (7) If left1 > left2, and left1 + width1 < left2 + width2, top1 + height1 > top2 + height2, it means the search box is in the bottom area outside the current viewport, such as Figure 10 Search box G in Fig. 1;

[0120] (8) If left1 + width1 > left2 + width2, top1 + height1 > top2 + height2, it means the search box is in the bottom-right area outside the current viewport, such as Figure 10 Search box H in Fig. 1.

[0121] (9) In other cases, it means the search box is in the current viewport.

[0122] S1103, when the search box is located outside the current viewport, calculating position difference information based on the first position information and the second position information, and adjusting the current viewport based on the position difference information, so that the search box is located in the adjusted viewport.

[0123] The position difference information between the first position information and the second position information can be calculated, and the display device 200 can adjust the position of the scroll bar 400 based on the position difference information, so as to adjust the position of the current viewport, so that the search box is located in the adjusted viewport. The specific adjustment process is as follows (corresponding to the position condition in S1102):

[0124] (1) Call Set Scroll Offset(left1-10, top1-10), and the search box A is displayed in the adjusted viewport as shown in ② of FIG. 11B, and the display position of the search box A is search box A'; Figure 10

[0125] (2) Call Set Scroll Offset(left2, top1-10), and the search box B is displayed in the adjusted viewport as shown in ② of FIG. 11B, and the display position of the search box B is search box B'; Figure 10

[0126] (3) Call Set Scroll Offset(left1+width1-width2+10, top1-10), and the search box C is displayed in the adjusted viewport as shown in ② of FIG. 11B, and the display position of the search box C is search box C'; Figure 10

[0127] (4) Call Set Scroll Offset(left1-10, top2), and the search box D is displayed in the adjusted viewport as shown in ② of FIG. 11B, and the display position of the search box D is search box D'; Figure 10

[0128] (5) Call Set Scroll Offset(left1+width1-width2+10, top2), and the search box E is displayed in the adjusted viewport as shown in ② of FIG. 11B, and the display position of the search box E is search box E'; Figure 10

[0129] (6) Call Set Scroll Offset(left1-10, top1+height1-height2+10), and the search box F is displayed in the adjusted viewport as shown in ② of FIG. 11B, and the display position of the search box F is search box F'; Figure 10

[0130] ​​​​​​(7) Call Set Scroll Offset(left2, top1+height1-height2+10), the search box G is displayed in the adjusted viewport as shown in FIG. 2(2) and the display position of the search box G' is as shown in FIG. 2(2). Figure 10

[0131] (8) Call Set Scroll Offset(left1+width1-width2+10, top1+height1-height2+10), the search box H is displayed in the adjusted viewport as shown in FIG. 2(2) and the display position of the search box H' is as shown in FIG. 2(2). Figure 10

[0132] Therefore, by adjusting the position of the viewport, the search box can be located in the adjusted viewport, so as to display the search box on the user interface, so that the user can see the search box on the webpage in the website and see the keyword filled in the search box by the display device 200, thereby ensuring the experience of the user.

[0133] Further, after the display device 200 fills the keyword into the search box, after waiting for a preset time interval, the display device 200 searches based on the keyword and displays the search result, i.e., the next page. In this way, the display device 200 can provide the user with the operation of correcting the keyword in the search box or withdrawing the voice instruction sent by mistake within the preset time interval, so as to improve the interaction effectiveness between the user and the display device 200.

[0134] From the above technical solutions, it can be seen that when the display device displays the webpage in the website, it can not only respond to the voice search instruction sent by the user, but also obtain the DOM tree of the webpage in the website to obtain the candidate input tag elements with the type attribute value of search and the type attribute value of text in the DOM tree, determine a target input tag element in an active state from the candidate input tag elements, and then determine the search box on the webpage in the website based on the target input tag element to provide an input basis for filling in the keyword. Furthermore, the search box located can be displayed in the viewport by automatically adjusting the viewport, so as to ensure the visibility of the search box and improve the interaction experience of the user.

[0135] The similar parts among the embodiments provided in the present application can be referred to each other, the specific implementation provided above is only several examples under the general concept of the present application, and does not constitute a limitation on the protection scope of the present application. Any other implementation extended according to the present application scheme without creative labor belongs to the protection scope of the present application for those skilled in the art.​​

Claims

1. A display device, characterized by comprising: The method comprises the following steps: displaying an intranet page; receiving a voice search instruction sent by a user when displaying the intranet page, the voice search instruction indicating to display a page corresponding to a keyword; obtaining a document object model (DOM) tree of the intranet page in response to the voice search instruction; obtaining all candidate input tag elements in the DOM tree, the candidate input tag elements including input tag elements with a type attribute value of search or including input tag elements with a type attribute value of text and deleting input tag elements with a read-only attribute value or a disabled attribute value; distinguishing the candidate input tag elements into first-class input tag elements and second-class input tag elements according to type attribute values of the input tag elements, constructing a first array with the first-class input tag elements, and constructing a second array with the second-class input tag elements; screening specified attribute values of the second-class input tag elements to determine whether the specified attribute values represent a search intent, and transferring input tag elements representing the search intent from the second array to the first array, the search intent being a search attribute; determining a target input tag element based on the first array, the target input tag element being one of the candidate input tag elements in an activated state; determining a search box of the intranet page, and filling the search box with the keyword to perform a search based on the keyword, the search box corresponding to the target input tag element. The controller obtains all candidate input tag elements in the DOM tree by being configured to:

2. The display device of claim 1, wherein, obtaining all input tag elements in the DOM tree and type attribute values of each input tag element; screening out candidate input tag elements with a type attribute value of search and a type attribute value of text from all the input tag elements; deleting input tag elements with a read-only attribute value or a disabled attribute value from the candidate input tag elements to obtain the candidate input tag elements. The controller determines the target input tag element by being configured to:

3. The display device of claim 1, wherein, constructing a first array and a second array, the first array including first-class input tag elements and the second array including second-class input tag elements, wherein the first-class input tag elements are candidate input tag elements with a type attribute value of search, and the second-class input tag elements are candidate input tag elements with a type attribute value of text; ​ determining a to-be-moved input tag element, the to-be-moved input tag element being a second-type input tag element whose specified attribute value contains a search string, wherein the specified attribute value includes an attribute value specified in the second-type input tag element, and the second-type input tag element has an attribute value of a parent node tag element within 4 levels in the DOM tree; moving the to-be-moved input tag element from the second array to the first array; determining the target input tag element in the moved first array.

4. The display device of claim 3, wherein, The controller determines a to-be-moved input tag element, which is configured to: remove the second-type input tag element with an attribute value login from the second array to obtain a compliant second-type input tag element; identify whether the specified attribute value of the compliant second-type input tag element contains a search string; if the specified attribute value contains a search string, the corresponding compliant second-type input tag element is determined as the to-be-moved input tag element; if the specified attribute value does not contain a search string, the attribute values of parent node tag elements within 4 levels in the DOM tree are sequentially judged, and the compliant second-type input tag element with the search string is determined as the to-be-moved input tag element.

5. The display device of claim 4, wherein, The controller sequentially judges whether the attribute values of parent node tag elements within 4 levels in the DOM tree contain a search string, and is further configured to: obtain the tag name of the currently judged parent node tag element; wherein when the tag name is body or form, the sequential judgment is terminated.

6. The display device of claim 3, wherein, The controller determines a target input tag element, which is configured to: identify whether the number of input tag elements in the moved first array is 0; if the number is not 0, one input tag element in the moved first array in an active state is determined as the target input tag element; if the number is 0, identify whether the number of input tag elements in the moved second array is 0, wherein when the number is not 0, each input tag element in the moved second array is moved to the moved first array to obtain a second-moved first array, and one input tag element in the second-moved first array in an active state is determined as the target input tag element; when the number is 0, it is determined that the target input tag element does not exist.

7. The display device of claim 3, wherein, The controller determines the target input tag element in the moved first array, which is configured to: If there are multiple input label elements in an active state in the first array after the transfer, the last one is added to the first array, and the input label element in the active state is determined as the target input label element; If there is no input label element in an active state in the first array after the transfer, the first input label element added to the first array is determined as the target input label element.

8. The display device of claim 1, wherein, The controller determines a search box of the webpage in the website, which is configured to: obtain first position information corresponding to the search box in the DOM tree and second position information corresponding to the current viewport; determine whether the search box is located in the current viewport based on the first position information and the second position information; wherein, when the search box is located outside the current viewport, a position difference value information is calculated based on the first position information and the second position information, and the current viewport is adjusted based on the position difference value information, so that the search box is located in the adjusted viewport.

9. The display device of claim 1, wherein, The controller fills the keyword in the search box to search based on the keyword, which is configured to: after the keyword is filled in the search box, a preset time interval is waited, and a search is performed based on the keyword.

10. A voice search method of a page within a website, characterized by, The method applied to a display device comprises: when displaying the webpage in the website, a voice search instruction sent by a user is received, the voice search instruction indicating that a page corresponding to a keyword is displayed; in response to the voice search instruction, a file object model (DOM) tree of the webpage in the website is obtained; all candidate input label elements in the DOM tree are obtained, the candidate input label elements being input label elements including a type attribute value of search or including a type attribute value of text and deleting input label elements with a read-only attribute value or a disabled attribute value; the candidate input label elements are distinguished into first-class input label elements and second-class input label elements according to type attribute values of the input label elements, the first-class input label elements are used to construct a first array, and the second-class input label elements are used to construct a second array; whether a specified attribute value of the second-class input label elements represents a search intent is screened, and input label elements representing the search intent are transferred from the second array to the first array, the search intent being a search attribute; based on the first array, a target input label element is determined, the target input label element being one of the candidate input label elements in an active state; a search box of the webpage in the website is determined, and the keyword is filled in the search box to search based on the keyword, the search box corresponding to the target input label element.

Images