Terminal display method and device based on voice recognition, storage medium and equipment

Abstract

The terminal display method and device based on voice recognition provided in the application can determine the instruction type of a voice instruction sent by a user when the voice instruction is received. When the instruction type is number viewing, the target application to which the current display content in the display screen belongs can be determined, and each display content in the target application and the number corresponding to each display content can be sent to the display screen for display. When the instruction type is number confirmation, the target number in the number confirmation instruction and the target display content corresponding to the target number can be determined, and the target display content can be sent to the display screen for display. The application can improve the convenience of voice control of a terminal device by identifying a display number instead of identifying the name of display content. After the display content of the display screen is updated according to the voice instruction, the application can continue to receive a new voice instruction sent by the user and perform subsequent steps.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a terminal display method, apparatus, storage medium and device based on speech recognition. Background Technology

[0002] In today's technologically advanced world, various terminal devices are emerging in an endless stream. Electronic devices such as mobile phones and televisions are important parts of life, greatly enriching and facilitating people's daily lives. When using these terminal devices, users usually need to perform precise operation through physical contact. For example, they can select a target by pressing buttons on the corresponding remote control or by directly touching the target position on the terminal's display screen.

[0003] To facilitate use of terminal devices by users with limited physical dexterity, some devices currently employ voice commands to control the display screen. This method identifies keywords in the user's voice commands to retrieve the names of various displayed content. However, its control capabilities are limited, relying on pre-defined names to invoke corresponding functions. For example, if the displayed content's name contains homophones, uncommon characters, artistic fonts, or foreign languages, making it difficult for the user to pronounce accurately, the terminal device cannot successfully recognize the voice command, resulting in poor voice control performance. Summary of the Invention

[0004] The purpose of this application is to at least solve one of the aforementioned technical defects, particularly the technical defect in the prior art where, when the name of the displayed content contains homophones, rare characters, artistic characters, foreign characters, etc., causing the user to be unable to accurately pronounce the name, the terminal device cannot successfully recognize the voice command, resulting in poor voice control performance.

[0005] This application provides a terminal display method based on speech recognition, the method comprising:

[0006] Receive voice commands sent by the user and determine the command type of the voice commands;

[0007] If the instruction type is a number viewing instruction, then the target application to which the currently displayed content belongs is determined, and after sending each display content in the target application and the number corresponding to each display content to the display screen for display, the process returns to execute the voice instruction sent by the user and its subsequent steps.

[0008] If the instruction type is a number confirmation instruction, then the target number in the number confirmation instruction and the target display content corresponding to the target number are determined, and the target display content is sent to the display screen for display.

[0009] Optionally, sending each display content in the target application and the corresponding number of each display content to the display screen for display includes:

[0010] Determine the hyperlink language page corresponding to the target application, and parse the hyperlink language page into a node tree, wherein the nodes in the node tree include attributes of any displayed content in the target application;

[0011] Assign numbers to multiple nodes in the node tree and establish a mapping relationship between each node and its number;

[0012] Based on the mapping relationships, the display content and number corresponding to each node are sent to the display screen for display.

[0013] Optionally, determining the hyperlink language page of the target application includes:

[0014] Send a transport protocol request to the target application and receive a response packet returned by the target application in accordance with the transport protocol request;

[0015] The response packet is parsed to obtain the hyperlink language page of the target application.

[0016] Optionally, parsing the hyperlink language page into a node tree includes:

[0017] Read the text content of the hyperlink language page, and parse each tag in the text content as well as the attributes and hierarchical structure corresponding to each tag;

[0018] Convert each tag into a corresponding node, and add the attributes of each tag to the corresponding node;

[0019] The nodes with added attributes are connected according to the hierarchical structure of each tag to construct the node tree of the hyperlink language page.

[0020] Optionally, assigning numbers to multiple nodes in the node tree includes:

[0021] Traverse the attributes of each node in the node tree, filter out nodes whose attributes meet the preset rules, and assign numbers to each filtered node in sequence.

[0022] Optionally, determining the target display content corresponding to the target number includes:

[0023] Determine the mapping relationship corresponding to the target number in the node tree, and the target node in the mapping relationship of the target number;

[0024] Read the coordinate information from the attributes of the target node, and generate a simulated click command for the target node based on the coordinate information;

[0025] The simulated click command is executed on the hyperlink language page, and the execution result is sent to the target application.

[0026] The target application receives the display content returned by the target application based on the execution result, and uses it as the target display content corresponding to the target number.

[0027] Optionally, the method further includes:

[0028] If the instruction type is a voice control instruction, the voice control instruction is converted into a simulated control command and executed to perform corresponding operations on the currently displayed content.

[0029] This application also provides a terminal display device, including:

[0030] The instruction receiving module is used to receive voice instructions sent by the user and determine the instruction type of the voice instructions;

[0031] The number viewing module is used to determine the target application to which the currently displayed content belongs if the instruction type is a number viewing instruction. After sending each display content in the target application and the number corresponding to each display content to the display screen for display, it returns to execute the voice instruction sent by the user and its subsequent steps.

[0032] The number confirmation module is used to determine the target number in the number confirmation instruction and the target display content corresponding to the target number if the instruction type is a number confirmation instruction, and send the target display content to the display screen for display.

[0033] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the terminal display method as described in any of the above embodiments.

[0034] This application also provides a computer device, including: one or more processors, and memory;

[0035] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the terminal display method as described in any of the above embodiments.

[0036] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0037] The terminal display method, apparatus, storage medium, and device provided in this application, based on voice recognition, can first determine the command type of the voice command upon receiving a voice command sent by a user. This command type can be a number-viewing command or a number-confirming command. When the received voice command is a number-viewing command, the target application to which the currently displayed content belongs can be determined, and each display content in the target application, along with its corresponding number, can be sent to the display screen for display, thereby representing the correspondence between the number and the display content. If the received voice command is a number-confirming command, the target number in the number-confirming command and the target display content corresponding to that number can be determined, and the target display content can be sent to the display screen for display. This application improves the convenience of voice control of the terminal device by recognizing the display number instead of the name of the display content, avoiding situations where the display content name contains homophones, rare characters, artistic fonts, foreign languages, etc., which would prevent users from performing voice control. Furthermore, after updating the display content according to the voice command, the device can continue to receive new voice commands from the user and execute subsequent steps. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A schematic flowchart illustrating a terminal display method based on speech recognition provided in an embodiment of this application;

[0040] Figure 2 A flowchart illustrating a method for determining target display content provided in an embodiment of this application;

[0041] Figure 3 A schematic diagram of the structure of a terminal display device based on voice recognition provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] In today's technologically advanced world, various terminal devices are emerging in an endless stream. Electronic devices such as mobile phones and televisions are important parts of life, greatly enriching and facilitating people's daily lives. When using these terminal devices, users usually need to perform precise operation through physical contact. For example, they can select a target by pressing buttons on the corresponding remote control or by directly touching the target position on the terminal's display screen.

[0045] To facilitate use of terminal devices by users with limited physical dexterity, some devices currently employ voice commands to control the display screen. This method identifies keywords in the user's voice commands to retrieve the names of various displayed content. However, its control capabilities are limited, relying on pre-defined names to invoke corresponding functions. For example, if the displayed content's name contains homophones, uncommon characters, artistic fonts, or foreign languages, making it difficult for the user to pronounce accurately, the terminal device cannot successfully recognize the voice command, resulting in poor voice control performance.

[0046] Based on this, this application proposes the following technical solution, as detailed below:

[0047] In one embodiment, such as Figure 1 As shown, Figure 1 This application provides a schematic flowchart of a terminal display method based on speech recognition, which includes the following:

[0048] S110: Receives voice commands sent by the user and determines the command type of the voice command.

[0049] In this step, when the user is viewing the relevant content displayed on the terminal device through the screen, if there is a need to switch the displayed content, the user can read the corresponding voice command to the terminal device. After receiving the user's voice command, the terminal device can first determine the command type of the voice command so that it can switch the displayed content on the screen according to the command type.

[0050] The term "terminal device" refers to a computer terminal device capable of projecting display content onto a screen. This can be a mobile terminal such as a mobile phone or laptop, or a fixed terminal such as a desktop computer or a set-top box; there are no restrictions on this. Furthermore, the display content on the screen can include, but is not limited to, text, audio, video, and images.

[0051] Specifically, after receiving a user's voice command, the terminal device can recognize the voice command using a preset speech recognition model and convert it into a text command based on the recognition result. The terminal device can then divide the text command into multiple shorter phrases to filter out effective phrases, thereby improving the efficiency of speech recognition. After selecting effective phrases, the terminal device can match them against a predefined corpus, labeling the parts of speech of each effective phrase based on the matching results. This allows the terminal device to classify the voice command based on the parts of speech of each effective phrase, thus determining the command type.

[0052] It should be noted that the command type of voice command represents different user needs, such as viewing the list of displayed content, switching displayed content, and adjusting the display progress on the screen. Therefore, before responding to a user's voice command, the terminal device needs to determine the command type and then perform the corresponding operation based on the command type.

[0053] S120: If the instruction type is a number viewing instruction, then determine the target application to which the currently displayed content belongs, and after sending each display content in the target application and the corresponding number of each display content to the display screen for display, return to execute the voice instruction sent by the user and its subsequent steps.

[0054] In this step, when the voice command type is determined to be a numbered viewing command through step S110, the terminal device can first read the currently displayed content on the screen and determine the target application to which the currently displayed content belongs. Then, it can obtain each display content in the target application and the number corresponding to each display content, and display each display content and the number corresponding to each display content on the screen. After that, the terminal device can continue to listen to the user's voice and determine whether the user has sent a new voice command.

[0055] It is understood that the terminal device in this application interfaces with multiple third-party applications, each storing display data for multiple display contents. Therefore, the terminal device can retrieve relevant display data from the corresponding third-party application according to the user's needs, convert the display data into display content, and send it to the display screen for display. To improve the convenience of voice control, the terminal device can assign numbers to the display content in each third-party application, so that the user can express their needs for the display content based on the number, avoiding the inability to accurately pronounce the name when the display content name contains homophones, uncommon characters, artistic fonts, foreign languages, etc. Therefore, when the user is unclear about the mapping relationship between the number and the display content, they can send a number-based command to the terminal device to view it.

[0056] Specifically, since the same number corresponds to different display content in different third-party applications, when the terminal device receives a number viewing instruction sent by the user, it can first determine the current display content on the screen, then take the third-party application to which the current display content belongs as the target application, then obtain the mapping relationship between each display content and the number from the target application, and send each mapping relationship to the screen for display so that the user can view it.

[0057] It should be noted that the displayed content of the target application will be adjusted appropriately over time, such as adding, deleting, or changing positions. Therefore, each time the terminal device receives a user's number viewing command, it can update the mapping relationship between each displayed content and number in the target application and send the updated mapping relationship to the display screen for display.

[0058] S130: If the instruction type is a number confirmation instruction, then determine the target number in the number confirmation instruction and the target display content corresponding to the target number, and send the target display content to the display screen for display.

[0059] In this step, when the voice command type is determined to be a number confirmation command through step S110, the terminal device can identify the target number in the number confirmation command, then confirm the target display content corresponding to the target number, and then send the target display content to the display screen for display.

[0060] It is understandable that the same ID can exist in different third-party devices. In other words, the same ID corresponds to different display content in different third-party devices. Therefore, the terminal device can first determine the third-party application to which the current display content on the screen belongs, and take the third-party application as the target application. Then, it can view the display content corresponding to the target ID based on the target application, so that the terminal device can send the display content as the target display content to the screen for display.

[0061] For example, when a user is watching a movie through a set-top box and wants to switch to another foreign movie, but the user cannot read the movie title, but knows that its number in the target application is 2, the user can say the number confirmation command "click number 2" to the set-top box. After receiving the number confirmation command, the set-top box can obtain the display content of the movie corresponding to number 2 in the target application and send the obtained display content to the display screen as the target display content.

[0062] In the above embodiments, when a voice command sent by a user is received, the command type can be determined first. This command type can be a number-viewing command or a number-confirming command. When the received voice command is a number-viewing command, the target application to which the currently displayed content belongs can be determined, and each display content in the target application, along with its corresponding number, can be sent to the display screen for display, thus representing the correspondence between the number and the display content. If the received voice command is a number-confirming command, the target number in the number-confirming command and the target display content corresponding to that number can be determined, and the target display content can be sent to the display screen for display. This application improves the convenience of voice control of the terminal device by identifying the display number instead of the name of the display content, avoiding situations where the display content name contains homophones, rare characters, artistic characters, foreign languages, etc., preventing users from performing voice control. Furthermore, after updating the display content according to the voice command, the system can continue to receive new voice commands from the user and execute subsequent steps.

[0063] In one embodiment, sending each display content in the target application and the corresponding number of each display content to the display screen for display may include:

[0064] S121: Determine the hyperlink language page of the target application and parse the hyperlink language page into a node tree.

[0065] S122: Assign numbers to multiple nodes in the node tree and establish a mapping relationship between each node and its number.

[0066] S123: Based on each mapping relationship, send the display content and number corresponding to each node to the display screen for display.

[0067] In this embodiment, after determining the target application, the terminal device can determine the hyperlink language page of the target application, and then parse the hyperlink language page into a node tree. The number of nodes here consists of multiple nodes, and each node is labeled with the attribute of any display content in the target application. After obtaining the node tree corresponding to the target application, the terminal device can assign numbers to multiple nodes in the node tree and establish a mapping relationship between each node and the number. Finally, based on each mapping relationship, the display content and number corresponding to each node can be sent to the display screen for display.

[0068] Hyperlink language pages refer to web pages written using markup languages, such as HTML (HyperText Markup Language) pages, used to display various information and provide various functions. These web pages include multiple hyperlinks. In simple terms, a hyperlink is a link within one webpage or document that leads to another webpage or document. In hyperlink languages, hyperlinks are typically defined using the `<hyperlink>` tag, which contains the hyperlink's attributes and link text. The link text is the content displayed when the user clicks on it.

[0069] Furthermore, the DOM (Document Object Model Tree) in this application refers to the tree structure of a hyperlinked language page on a server, used to represent the page's structure and content. A DOM tree is typically composed of multiple connected nodes, where nodes represent tags on the page, and the connecting lines between nodes represent relationships between them, such as parent-child relationships, sibling relationships, etc. Therefore, this application can use a DOM tree to visualize hyperlinked language pages.

[0070] It should be noted that, since the link attributes of tags in hyperlink language pages are diverse, not every tag's corresponding link text needs to be sent to the display screen for display. Therefore, when assigning numbers to nodes in the node tree, the terminal device only needs to assign numbers to the nodes whose link text needs to be displayed, thereby improving the data processing efficiency of the terminal device.

[0071] In one embodiment, determining the hyperlink language page of the target application in step S121 may include:

[0072] S211: Send a transport protocol request to the target application and receive a response packet returned by the target application in accordance with the transport protocol request.

[0073] S212: Parse the response packet to obtain the hyperlink language page of the target application.

[0074] In this embodiment, when determining the hyperlink language page of the target application, the terminal device can first generate a transport protocol request and send the transport protocol request to the target application. When the target application receives the transport protocol request, it can package the hyperlink language page into a response packet after obtaining it, and return the response packet to the terminal device so that the terminal device can parse the response packet to obtain the hyperlink language page of the target application.

[0075] Specifically, after identifying the target application, the terminal device can establish a connection to the target application's server using a transport protocol, such as HTTP or TCP. The terminal device can then send a request to the target application's server, including information about the hyperlinked language page. Upon receiving the request, the target application's server can return a corresponding response packet based on the requested information. Finally, after receiving the response packet from the target application's server, the terminal device can disconnect from the transport protocol connection.

[0076] It is understood that the transport protocol here refers to the protocol used for data transmission in a computer network, which specifies the format, process, and rules of data transmission. In the network hierarchy, transport protocols include application layer protocols, transport layer protocols, and network layer protocols. Application layer protocols refer to protocols for transmitting data between applications, such as HTTP, FTP, and SMTP; transport layer protocols refer to protocols for enabling end-to-end communication in network transmission, such as TCP and UDP; and network layer protocols refer to protocols for enabling communication between hosts in network transmission, such as IP, ICMP, and ARP. Therefore, different transport protocols have different characteristics and applicable scenarios, and this application can select and apply them according to specific needs without imposing any restrictions.

[0077] In one embodiment, parsing the hyperlink language page into a node tree in step S121 may include:

[0078] S213: Read the text content of the hyperlink language page and parse each tag in the text content, as well as the attributes and hierarchical structure corresponding to each tag.

[0079] S214: Convert each tag into a corresponding node and add the attributes of each tag to the corresponding node.

[0080] S215: Connect the nodes after adding attributes according to the hierarchical structure of each tag to construct the node tree of the hyperlink language page.

[0081] In this embodiment, when the terminal device parses the hyperlink language page into a node tree, it can first read the text content of the hyperlink language page, as well as each tag and its position in the text content. Then, it can parse the attributes and hierarchical structure corresponding to each tag based on the position of each tag. Next, the terminal device can convert each tag into a corresponding node and add the attributes of each tag to the corresponding node. Finally, it can connect the nodes after adding attributes based on the hierarchical structure of each tag, thereby completing the construction of the node tree.

[0082] It is understood that in this application, the attributes define the specific characteristics and behaviors of the displayed content corresponding to the tag in the hyperlink language page, such as the format, position and size of the displayed content, while the hierarchy refers to the position of the tag in the hyperlink language page, which is represented in the node tree as the relationship with other nodes, such as sibling relationship, parent-child relationship, etc.

[0083] Specifically, when constructing a node tree, the terminal device can use a network connection or local file system to read the text content of the hyperlink language page. Before converting each tag into a corresponding node, it can first generate a root node of the node tree based on the text content. Then, it can parse the tags in the hyperlink language page and convert each tag into a node in the node tree. For each node, it can parse its attributes and add them to the corresponding node. Finally, the terminal device can connect each node sequentially from the root node according to the hierarchical structure of each tag and the parent-child relationship between each tag, thereby constructing a complete node tree.

[0084] In one embodiment, assigning numbers to multiple nodes in the node tree in step S122 may include:

[0085] S221: Traverse the attributes of each node in the node tree, filter out the nodes whose attributes meet the preset rules, and assign numbers to each filtered node in sequence.

[0086] In this embodiment, when assigning numbers to nodes in the node tree, the terminal device can first traverse the attributes of each node in the node tree to filter out nodes whose attributes meet the preset rules. Then, it can assign numbers to each filtered node in sequence to avoid the situation where the same number is repeatedly assigned to different nodes.

[0087] It is understandable that hyperlinked language pages are usually composed of various display content in different formats, such as text, video, audio and images. However, for some terminal devices, not all display content needs to be shown to the user. Therefore, when assigning numbers, the terminal device can first filter out nodes whose attributes meet the preset rules from each node, and then assign numbers to each of the filtered nodes in sequence.

[0088] For example, in a target application configured on a TV set-top box, the main content displayed on its hyperlink language page is in the format of video and images. The main function of the TV set-top box is to play videos to users. In this regard, the TV set-top box can traverse the attributes of the node tree of the hyperlink language page and filter out the nodes whose attributes are in the format of video. Then, it can assign numbers to each of the filtered nodes in sequence.

[0089] In one embodiment, such as Figure 2 As shown, Figure 2 A flowchart illustrating a method for determining target display content provided in an embodiment of this application; Figure 2 In step S130, determining the target display content corresponding to the target number may include:

[0090] S131: Determine the mapping relationship of the target number in the node tree, and the target node in the mapping relationship of the target number.

[0091] S132: Read the coordinate information from the target node's attributes and generate a simulated click command for the target node based on the coordinate information.

[0092] S133: Execute a simulated click command in the hyperlink language page and send the execution result to the target application.

[0093] S134: Receive the display content returned by the target application based on the execution result, and use it as the target display content corresponding to the target number.

[0094] In this embodiment, after the terminal device recognizes the target number in the number confirmation instruction, it can first determine the mapping relationship of the target number in the node tree, and determine the target node corresponding to the target number according to the mapping relationship. Then, the terminal device can read the coordinate information in the attribute of the target node, and generate a simulated click command for the target node based on the coordinate information. Then, it can use the simulated click command to perform a simulated click on the hyperlink language page, so that the target application can obtain the display data of the display content at the simulated click position and return it to the terminal device. After receiving the display data of the target application, the terminal device can convert it into display content as the target display content corresponding to the target number.

[0095] For example, when the set-top box recognizes "Number 2" in the user's confirmation command, it can determine the attributes of the displayed content corresponding to "Number 2" based on the mapping relationships in the node tree. The coordinate information in these attributes is {'bounds':{'left':50,'top':150,'right':100,'bottom':200}}. In this coordinate information, the 'left' and 'top' attributes represent the coordinates of the top-left corner of the rectangular area of ​​the displayed content within the hyperlink language page, and the 'right' and 'bottom' attributes represent the coordinates of the bottom-right corner of the rectangular area within the hyperlink language page. Through these four attributes, the position and size of the rectangular area can be determined, thereby enabling the positioning and manipulation of the displayed content.

[0096] Therefore, the set-top box can generate a simulated click command based on this coordinate information. The simulated click command generated here can be "input tap(x, y)", where (x, y) is the center coordinate point of the rectangular area of ​​the displayed content. For number 2, the calculation process of its center coordinate point is: x = 50 + (100 - 50) / 2, y = 150 + (200 - 150) / 2, that is, the simulated click command for number 2 is "input tap(75, 175)".

[0097] In one embodiment, the method may further include:

[0098] S140: If the instruction type is a voice control instruction, the voice operation instruction is converted into a simulated control command and executed to perform corresponding operations on the currently displayed content.

[0099] In this embodiment, the voice command sent by the user can also be a voice control command. When the voice command is a voice control command, the terminal device can convert the voice operation command into a simulated control command and execute the simulated operation command to perform corresponding operations on the currently displayed content of the screen. The simulated control command includes, but is not limited to, fast forward, rewind, increase volume, decrease volume, pause, etc.

[0100] For example, if a user needs to leave while watching a movie through a set-top box, they can say "pause" to the set-top box. After receiving the command, the set-top box can perform voice recognition and determine the corresponding function based on the recognition result, thereby generating a simulated operation command. By executing the simulated operation command, the movie can be paused.

[0101] The terminal display device provided in the embodiments of this application is described below. The terminal display device described below and the terminal display method described above can be referred to in correspondence.

[0102] In one embodiment, such as Figure 3 As shown, Figure 3 This application provides a schematic diagram of a terminal display device based on voice recognition, as shown in an embodiment of the present application. The present application also provides a terminal display device, including an instruction receiving module 210, a number viewing module 220, and a number confirmation module 230, specifically comprising the following:

[0103] The instruction receiving module 210 is used to receive voice instructions sent by the user and determine the instruction type of the voice instructions.

[0104] The number viewing module 220 is used to determine the target application to which the currently displayed content belongs if the instruction type is a number viewing instruction. After sending each display content in the target application and the corresponding number of each display content to the display screen for display, it returns to execute the voice instruction sent by the user and its subsequent steps.

[0105] The number confirmation module 230 is used to determine the target number in the number confirmation instruction and the target display content corresponding to the target number if the instruction type is a number confirmation instruction, and then send the target display content to the display screen for display.

[0106] In the above embodiments, when a voice command sent by a user is received, the command type can be determined first. This command type can be a number-viewing command or a number-confirming command. When the received voice command is a number-viewing command, the target application to which the currently displayed content belongs can be determined, and each display content in the target application, along with its corresponding number, can be sent to the display screen for display, thus representing the correspondence between the number and the display content. If the received voice command is a number-confirming command, the target number in the number-confirming command and the target display content corresponding to that number can be determined, and the target display content can be sent to the display screen for display. This application improves the convenience of voice control of the terminal device by identifying the display number instead of the name of the display content, avoiding situations where the display content name contains homophones, rare characters, artistic characters, foreign languages, etc., preventing users from performing voice control. Furthermore, after updating the display content according to the voice command, the system can continue to receive new voice commands from the user and execute subsequent steps.

[0107] In one embodiment, the number viewing module 220 may include:

[0108] The language page parsing submodule is used to determine the hyperlink language page corresponding to the target application and parse the hyperlink language page into a node tree.

[0109] The numbering assignment submodule is used to assign numbers to multiple nodes in the node tree and establish a mapping relationship between each node and its number.

[0110] The number display submodule is used to send the display content and number corresponding to each node to the display screen for display based on the mapping relationship.

[0111] In one embodiment, the language page parsing submodule may include:

[0112] The request sending unit is used to send a transport protocol request to the target application and receive the response packet returned by the target application according to the transport protocol request.

[0113] The language page retrieval unit is used to parse the response packet and obtain the hyperlink language page corresponding to the target application.

[0114] In one embodiment, the language page parsing submodule may further include:

[0115] The tag parsing unit is used to read the text content of the hyperlink language page and parse each tag in the text content, as well as the attributes and hierarchical structure of each tag.

[0116] The node transformation unit is used to convert each tag into a corresponding node and add the attributes of each tag to the corresponding node.

[0117] The node tree building unit is used to connect the nodes with added attributes according to the hierarchical structure of each tag, and build the node tree of the hyperlink language page.

[0118] In one embodiment, the number allocation submodule may include:

[0119] The numbering and filtering unit is used to traverse the attributes of each node in the node tree, filter out nodes whose attributes meet the preset rules, and assign numbers to each filtered node in sequence.

[0120] In one embodiment, the number confirmation module 230 may include:

[0121] The node determination submodule is used to determine the mapping relationship of the target number in the node tree, as well as the target node in the mapping relationship of the target number.

[0122] The command generation submodule is used to read the coordinate information from the attributes of the target node and generate a simulated click command for the target node based on the coordinate information.

[0123] The command execution submodule is used to execute simulated click commands in hyperlink language pages and send the execution results to the target application.

[0124] The content acquisition submodule is used to receive the content returned by the target application based on the execution result, which serves as the target display content corresponding to the target number.

[0125] In one embodiment, the apparatus may further include:

[0126] The voice control module is used to convert voice operation commands into simulated control commands if the command type is a voice control command, and then execute the simulated control commands to perform corresponding operations on the currently displayed content.

[0127] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the speech recognition-based terminal display method as described in any of the above embodiments.

[0128] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the speech recognition-based terminal display method as described in any of the above embodiments.

[0129] Indicatively, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 4 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the terminal display method of any of the above embodiments.

[0130] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0131] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0133] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0134] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A terminal display method based on speech recognition, characterized in that, The method includes: Receive voice commands sent by the user and determine the command type of the voice commands; The process of receiving voice commands sent by the user and determining the command type of the voice commands specifically includes: The voice command is recognized by a preset voice recognition model, and the voice command is converted into a text command based on the recognition result; The text instructions are divided into phrases, and valid phrases are selected from each phrase. The voice commands are classified by matching effective word groups with a predefined corpus and labeling the parts of speech of the effective word groups according to the matching results. The command type corresponding to the voice command is obtained based on the classification result; If the instruction type is a number viewing instruction, then the target application to which the currently displayed content belongs is determined, and after sending each display content in the target application and the number corresponding to each display content to the display screen for display, the process returns to execute the voice instruction sent by the user and its subsequent steps. The step of sending each display content in the target application and the corresponding number of each display content to the display screen for display includes: Determine the hyperlink language page corresponding to the target application, and parse the hyperlink language page into a node tree, wherein the nodes in the node tree include attributes of any displayed content in the target application; Assign numbers to multiple nodes in the node tree and establish a mapping relationship between each node and its number; Based on the mapping relationships, the display content and number corresponding to each node are sent to the display screen for display. The construction of the node tree includes: Read the text content of the hyperlink language page, as well as the tags and their positions within that text content; The attributes and hierarchical structure corresponding to each tag are obtained by parsing the tag position. The attributes define the specific characteristics and behaviors of the displayed content corresponding to the tag on the hyperlink language page. Convert each tag into a corresponding node, and add the attributes of each tag to the corresponding node; Connect the nodes with added attributes according to the hierarchical structure of each tag to construct a node tree; Determining the hyperlink language page of the target application includes: Send a transport protocol request to the target application and receive a response packet returned by the target application in accordance with the transport protocol request; The response packet is parsed to obtain the hyperlink language page of the target application; If the instruction type is a number confirmation instruction, then the target number in the number confirmation instruction and the target display content corresponding to the target number are determined, and the target display content is sent to the display screen for display.

2. The terminal display method according to claim 1, characterized in that, The step of parsing the hyperlink language page into a node tree includes: Read the text content of the hyperlink language page, and parse each tag in the text content as well as the attributes and hierarchical structure corresponding to each tag; Convert each tag into a corresponding node, and add the attributes of each tag to the corresponding node; The nodes with added attributes are connected according to the hierarchical structure of each tag to construct the node tree of the hyperlink language page.

3. The terminal display method according to claim 1, characterized in that, Assigning numbers to multiple nodes in the node tree includes: Traverse the attributes of each node in the node tree, filter out nodes whose attributes meet the preset rules, and assign numbers to each filtered node in sequence.

4. The terminal display method according to claim 1, characterized in that, Determining the target display content corresponding to the target number includes: Determine the mapping relationship corresponding to the target number in the node tree, and the target node in the mapping relationship of the target number; Read the coordinate information from the attributes of the target node, and generate a simulated click command for the target node based on the coordinate information; The simulated click command is executed on the hyperlink language page, and the execution result is sent to the target application. The target application receives the display content returned by the target application based on the execution result, and uses it as the target display content corresponding to the target number.

5. The terminal display method according to claim 1, characterized in that, The method further includes: If the instruction type is a voice control instruction, the voice control instruction is converted into a simulated control command and executed to perform corresponding operations on the currently displayed content.

6. A terminal display device, characterized in that, include: The instruction receiving module is used to receive voice instructions sent by the user and determine the instruction type of the voice instructions; The process of receiving voice commands sent by the user and determining the command type of the voice commands specifically includes: The voice command is recognized by a preset voice recognition model, and the voice command is converted into a text command based on the recognition result; The text instructions are divided into phrases, and valid phrases are selected from each phrase. The voice commands are classified by matching effective word groups with a predefined corpus and labeling the parts of speech of the effective word groups according to the matching results. The command type corresponding to the voice command is obtained based on the classification result; The number viewing module is used to determine the target application to which the currently displayed content belongs if the instruction type is a number viewing instruction. After sending each display content in the target application and the number corresponding to each display content to the display screen for display, it returns to execute the voice instruction sent by the user and its subsequent steps. The step of sending each display content in the target application and the corresponding number of each display content to the display screen for display includes: Determine the hyperlink language page corresponding to the target application, and parse the hyperlink language page into a node tree, wherein the nodes in the node tree include attributes of any displayed content in the target application; Assign numbers to multiple nodes in the node tree and establish a mapping relationship between each node and its number; Based on the mapping relationships, the display content and number corresponding to each node are sent to the display screen for display. The construction of the node tree includes: Read the text content of the hyperlink language page, as well as the tags and their positions within that text content; The attributes and hierarchical structure corresponding to each tag are obtained by parsing the tag position. The attributes define the specific characteristics and behaviors of the displayed content corresponding to the tag on the hyperlink language page. Convert each tag into a corresponding node, and add the attributes of each tag to the corresponding node; Connect the nodes with added attributes according to the hierarchical structure of each tag to construct a node tree; Determining the hyperlink language page of the target application includes: Send a transport protocol request to the target application and receive a response packet returned by the target application in accordance with the transport protocol request; The response packet is parsed to obtain the hyperlink language page of the target application; The number confirmation module is used to determine the target number in the number confirmation instruction and the target display content corresponding to the target number if the instruction type is a number confirmation instruction, and send the target display content to the display screen for display.

7. A storage medium, characterized in that: The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the terminal display method as described in any one of claims 1 to 5.

8. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the terminal display method as described in any one of claims 1 to 5.

Images