Method and apparatus for processing voice instructions

Abstract

The present invention relates to a method and apparatus for processing voice commands. A computer-implemented method performed by one or more computing devices to execute a voice command based on displayed information is provided, the method comprising: converting a user's utterance to text to obtain an input text string, prioritizing a plurality of screen texts displayed on a display screen, and executing a voice command corresponding to a highest priority text among the plurality of screen texts, wherein the priority of the plurality of screen texts indicates a degree of relevance of the screen text to the user's utterance intent.

Description

[0001] Cross-references to related applications This application claims priority to Korean Patent Application No. 10-2025-0043506, filed on April 3, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0002] In some embodiments, the present invention relates to a method and apparatus for processing voice commands. More specifically, the present invention relates to a method and apparatus for processing voice commands that avoids or minimizes conflicts between multiple global commands and multiple local commands. Background Technology

[0003] The statements in this section are provided only as background information in relation to the present invention and do not necessarily constitute prior art.

[0004] In recent years, the application of voice recognition services has gradually expanded to various fields such as vehicles. Voice recognition services enable users to control devices or software using their voice. Voice recognition services are applied to a variety of devices such as vehicles, smartphones, and home appliances, helping users achieve device control without manual operation.

[0005] Vehicles equipped with voice recognition services respond to user speech by generating voice commands to control the vehicle. Users can lock / unlock doors or adjust the interior temperature by pre-activating the air conditioning system.

[0006] To enable speech recognition services, vehicles are equipped with a speech recognition system that pre-stores a list of commands, including voice instructions. The software application recognizes the voice commands, and the system responds to the user's voice commands by selecting the control command most similar to the user's voice command from the command list and executing that control command.

[0007] Speech recognition commands can be divided into two categories: "global commands" and "local commands." Global commands are available to the user at any time when speech recognition is enabled; while local commands are available to the user only when a specific application interface is displayed (i.e., when a specific application is running), even when speech recognition is enabled. Users can perform the required functions by speaking the text displayed on the running application interface.

[0008] When a specific software application runs, it can execute both global and local commands. For example, if both global and local commands correspond to the user's speech, a conflict may occur between them, causing the speech recognition system to fail to issue control commands that match the user's intended speech. Conflicts may also occur between multiple global commands or multiple local commands.

[0009] Therefore, in order to enable users to easily use speech recognition systems, a method is needed to avoid or minimize conflicts between multiple global instructions and multiple local instructions. Summary of the Invention

[0010] The present invention aims to provide a method and apparatus for processing voice commands, which can avoid or minimize conflicts between multiple global commands and multiple local commands.

[0011] The purpose of this invention is not limited to the specific purposes described above, and those skilled in the art will clearly understand, through the following detailed description, that the above and other purposes achievable by this invention are possible.

[0012] According to at least one embodiment, the present invention provides a computer-implemented method for executing voice commands by a computing device. The method includes: the computing device acquiring an input text string by converting a user's speech into text; the computing device determining whether the input text string is classified as a highest-priority command; based on the input text string not being classified as a highest-priority command, the computing device determining the priority of a plurality of texts displayed on a screen; and the computing device executing a voice command corresponding to the highest-priority text among the plurality of texts. Here, the highest-priority command is a global voice command not displayed on the screen; the priority is determined based on the relevance of the texts displayed on the screen to the user's speech intent.

[0013] According to another embodiment of the present invention, an apparatus for executing voice commands based on display information is provided, comprising a memory configured to store instructions and at least one processor. The at least one processor is configured to execute instructions to: determine an input text string by converting the user's speech into text; determine whether the input text string is classified as a highest priority instruction; determine the priority of a plurality of texts displayed on a display screen based on the fact that the input text string is not classified as a highest priority instruction; and execute a voice command corresponding to the highest priority text among the plurality of screen texts; wherein, the highest priority instruction is a global voice command not displayed on the display screen; the priority indicates the degree of relevance between the text among the plurality of texts displayed on the display screen and the user's speech intent.

[0014] According to one embodiment of the present invention, conflicts between multiple global instructions and multiple local instructions can be avoided or minimized.

[0015] According to one embodiment of the present invention, by avoiding or minimizing conflicts between multiple global instructions and multiple local instructions, the execution of the user's speech can be achieved. Figure 1 This allows for more precise control commands, thereby enhancing the convenience of the voice recognition system.

[0016] The effects of the present invention are not limited to those described above, and those skilled in the art can clearly understand other effects not explicitly stated from the following description. Attached Figure Description

[0017] Figure 1 This is a schematic block diagram of an overall system according to at least one embodiment of the present invention.

[0018] Figure 2 This is a block diagram of a speech recognition system according to at least one embodiment of the present invention.

[0019] Figure 3 The display screen is shown as consisting of multiple layers.

[0020] Figure 4 Showing Figure 3 An example priority chart of scores for the text on each screen.

[0021] Figure 5 This is a flowchart of a process for prioritizing screen text according to at least one embodiment of the present invention.

[0022] Figure 6 This is a flowchart of a process performed by a speech recognition engine to classify a user's speech intent from their speech, according to at least one embodiment of the present invention.

[0023] Figure 7 It is a schematic block diagram of a computing device that can be used to implement the method or apparatus according to the present invention. Detailed Implementation

[0024] Some embodiments of the invention will now be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals preferably denote the same elements, even if these elements are shown in different figures. Furthermore, for clarity and brevity, detailed descriptions of related known components and functions will be omitted from the following description of some embodiments when they are considered to obscure the subject matter of the invention.

[0025] Various ordinal numbers or letter codes, such as first, second, i), ii), a), b), etc., are used as prefixes to describe components of embodiments of the present invention. They are used only to distinguish one component from other components and do not imply or indicate the nature, order, or sequence of the components. Throughout this specification, when a component "comprises" or "contains" a component, that component means that other components are also included, unless otherwise expressly stated, other components are not excluded.

[0026] The following description of the invention, presented in conjunction with the accompanying drawings, is intended to illustrate various aspects of the invention and is not intended to represent the only possible implementation of the technical concepts of the invention.

[0027] Figure 1 This is a schematic block diagram of an overall system according to at least one embodiment of the present invention.

[0028] refer to Figure 1 The overall system includes: vehicle 10, server 20 and terminal 30.

[0029] Figure 1 The vehicle 10 also includes: a voice input interface 110, a voice recognition engine 120, an infotainment device 130, an output interface 140, a controller 150, and a communication module 160. Figure 1 The modules shown are not all essential components; in other implementations, some modules included in vehicle 10 may be added, modified, or removed. Furthermore, Figure 1 The components shown represent functionally distinct elements, at least one of which can be implemented in a form integrated into an actual physical environment.

[0030] The voice input interface 110 is configured to receive voice input from the user.

[0031] The voice input interface 110 includes one or more microphones for receiving user speech. The microphones can be located, for example, on the vehicle's steering wheel, center dashboard, headliner, or rearview mirror to receive user voice input.

[0032] The voice recognition engine 120 is configured to analyze the user's voice input and send control commands to the infotainment device 130.

[0033] The voice recognition engine 120 can utilize a pre-stored instruction table to provide control commands corresponding to the user's voice input. The instruction table may include a global instruction table and local instruction tables, the global instruction table recording information about global instructions, and the local instruction tables recording information about local instructions for various applications run by the infotainment device 130.

[0034] First, the speech recognition engine 120 determines the voice command corresponding to the user's voice input. Specifically, the speech recognition engine 120 converts the user's speech into text and determines the user's speech intent based on the converted text.

[0035] The speech recognition engine 120 can use utterance intent as a voice command. In other words, utterance intent and voice command can be considered equivalent. For example, if the user's utterance is "call Hong Gil-dong", the speech recognition engine 120 can determine the user's voice intent as "call <name>", and can also determine the voice command corresponding to the user's utterance as "call <name>".

[0036] The voice recognition engine 120 then refers to an instruction table to identify the control command corresponding to the voice command. The voice recognition engine 120 sends the identified control command to the infotainment device 130. The infotainment device 130 executes the task according to the control command.

[0037] In another embodiment, the speech recognition engine 120 utilizes the server 20 to analyze the user's speech input. The speech recognition engine 120 sends speech data to the server 20, receives the analysis results of the speech data from the server 20, and generates control commands based on the analysis results. Here, the analysis results may include utterance intent and slots.

[0038] refer to Figure 2 A more detailed description of the speech recognition engine 120.

[0039] The infotainment device 130 is configured to provide various types of information to the user in response to control commands, either in the form of voice, visual means, or a combination of both.

[0040] The infotainment device 130 can receive control commands from the voice recognition engine 120 and perform tasks according to the received control commands.

[0041] The infotainment device 130 can run at least one application to provide users with the information or functions they require. For example, the infotainment device 130 can execute a telephone application, a navigation application, a web portal application, a media application, etc., to provide telephone functions, destination information, weather information, music playback functions, etc.

[0042] Output interface 140 is configured to output information provided by infotainment device 130.

[0043] Output interface 140 may include an AVN display, instrument panel display, or head-up display (HUD) disposed on the central instrument panel of vehicle 10. Alternatively, output interface 140 may include a rear-seat display located behind the headrests of the front seats for rear passengers to view, or output interface 140 may include a headliner-mounted display used in multi-passenger vehicles.

[0044] The output interface 140 may include at least one speaker for outputting information provided by the infotainment device 130 in the form of sound.

[0045] In addition, the output interface 140 may include other types of devices, such as haptic feedback devices that generate vibrations.

[0046] The controller 150 performs various controls related to the vehicle 10 based on input signals received via the voice input interface 110 or the infotainment device 130. For example, the controller 150 can control at least one of the following: an air conditioning unit, a window control unit, a seat heating unit, a seat adjustment unit, or a lighting unit.

[0047] The communication module 160 communicates with the terminal 30 based on control signals received via the voice input interface 110 or the infotainment device 130. For example, the communication module 160 can receive audio signals for playing music, signals about the content of a call, etc., from the terminal 30, and can send signals to the terminal 30 for playing the next song, signals for making a phone call, etc.

[0048] The communication module 160 can employ at least one of various wireless communication schemes (such as Bluetooth, 4G communication, 5G communication, or Wi-Fi) for signal transmission and reception. Alternatively, the communication module 160 can utilize a cable connected to a Universal Serial Bus (USB) port, an auxiliary (AUX) port, etc., to transmit and receive information with other devices.

[0049] Figure 2 This is a block diagram of a speech recognition system according to at least one embodiment of the present invention.

[0050] refer to Figure 2 The voice recognition engine 120 recognizes and understands the user's speech and provides control commands and / or responses corresponding to the user's speech. In this invention, the user represents a driver or passenger.

[0051] To this end, the speech recognition engine 120 includes: a speech recognition module 210, a natural language understanding (NLU) module 220, and a priority ranking module 230. The speech recognition module 210 converts the user's utterances and voice commands into text; the natural language understanding module (NLU module) 220 determines the user's utterance intent; and the priority ranking module 230 calculates the priority score of the voice commands. The speech recognition engine 120 may further include: a response generation module or a dialogue manager, the response generation module performing processing to provide a response corresponding to the user's utterance intent; and the dialogue manager managing the overall dialogue between the speech recognition engine 120 and the user.

[0052] The speech recognition module 210 acquires the user's speech received from the vehicle-mounted microphone and converts the user's speech into an input text string by utilizing at least one speech-to-text (STT) engine. The STT engine can apply a speech recognition algorithm or a deep learning model to the speech signal representing the user's speech, thereby converting the speech signal into text.

[0053] For example, the speech recognition module 210 can extract feature vectors from the user's speech using feature vector extraction techniques such as cepstral mapping, linear predictive coding (LPC), Mel-frequency cepstral coefficients (MFCC), or filter bank energy. The speech recognition module 210 can then obtain recognition results by comparing the extracted feature vectors with a trained benchmark pattern. For this purpose, an acoustic model (which models and compares the signal features of speech) or a language model (which models the linguistic order relationships of words, syllables, etc., in the recognition vocabulary) can be used.

[0054] The speech recognition module 210 can also convert the user's speech into an input text string in text form based on a model trained using machine learning or deep learning.

[0055] The NLU module 220 classifies the user's verbal intent based on the input text string.

[0056] In one example, NLU module 220 can compare an input text string with a predefined grammar to determine the user's utterance intent. For instance, when the preset grammar is "call <name>" and the input text string is "call Hong Jitong", NLU module 220 can determine the utterance intent as "call <name>".

[0057] As another example, NLU module 220 can utilize word segmentation techniques, deep learning models, etc., to determine the user's utterance intent for the input text string. Specifically, NLU module 220 can segment the input text string into morphological units. Furthermore, NLU module 220 can label each unit with a part-of-speech tag. NLU module 220 projects the units into a vector space. Each unit or combination of units is converted into an embedding vector. To improve performance, sequence embedding, positional embedding, etc., can be performed simultaneously. By aggregating the embedding vectors, or by applying a first deep learning model and a second deep learning model to the embedding vectors respectively, NLU module 220 determines the utterance intent of the input text string. Here, the first deep learning model can be a recurrent neural network that is pre-trained to classify utterance intent in response to the input embedding vector. The second deep learning model can be a recurrent neural network that is pre-trained to determine slots in response to the input embedding vector.

[0058] In at least one implementation, the NLU module 220 may not be able to classify the user's speech intent by comparing the input text string with the prescribed syntax and / or by using a deep learning model. In this case, the NLU module 220 may further classify the user's speech intent based on the input text string and the priority score calculated by the priority sorting module 230.

[0059] In another implementation, NLU module 220 can classify the user’s utterance intent based on priority scores and the input text string without comparing the input text string with a prescribed grammar and / or using a deep learning model.

[0060] The priority sorting module 230 can calculate a priority score for each piece of screen text and symbol presented on the display screen. This priority score indicates the degree of relevance of each piece of text and symbol to the user's utterance intent. For ease of description, the term "screen text" can be used to refer to all text and symbols displayed on the display screen.

[0061] The priority sorting module 230 receives the input text string from the speech recognition module 210 and receives display information from the infotainment device 130 or the output interface 140.

[0062] The priority ranking module 230 calculates a priority score for the screen text based on the input text string and the displayed information. The NLU module 220 can classify the user's speech intent based on the priority score calculated by the priority ranking module 230.

[0063] Since users of a speech recognition system often refer to the content displayed on the screen when speaking, if the user's speech intent is unclear, the speech recognition engine 120 can more accurately determine the user's speech intent by considering the current screen content.

[0064] The speech recognition engine 120 determines the content of the specific screen text that the user references and speaks. The speech recognition engine 120 then determines the utterance intent based on the text referenced by the user.

[0065] The priority sorting module 230 can calculate priority scores for text on each screen based on the following factors: (i) the structure of the display screen, (ii) the form and content of the text, and (iii) the functionality of the graphical user interface (GUI) components, thereby determining the text on the display screen that the user is referencing.

[0066] The priority sorting module 230 can prioritize the text on the display screen based on the screen's structure. The screen's structure may include: the Z-index of each of the multiple layers that make up the screen and / or the hierarchical / tree structure of the text.

[0067] The display screen of output interface 140 can be composed of multiple layers to provide additional information or interact with the user on the default screen. Typically, each layer displays different content (such as lists, selection pop-ups, etc.) and has corresponding instructions.

[0068] The Z-index determines the stacking order of multiple layers on the screen. The higher the Z-index value of a layer, the higher its display priority on the screen. The lower the Z-index value of a layer, the more obscured its view.

[0069] In at least one embodiment, the priority sorting module 230 can prioritize screen text based on the Z-index value of the layer. When the display screen consists of multiple layers, the layer with the higher Z-index value has a higher priority than the layer with the lower Z-index value. That is, the text displayed at the top layer has a higher priority than the text displayed at the bottom layer.

[0070] In at least one implementation, the priority of text based on Z-index values ​​can be represented as Z-index scores. That is, the Z-index score of the text displayed at the top level is determined to be higher than the Z-index score of the text displayed at the bottom level.

[0071] Figure 3 The display screen is shown as consisting of multiple layers.

[0072] Figure 3The display screen consists of three layers: L0, L1, and L2. Layer L0 serves as the main screen and is obscured by the other layers. The first layer, L1, is an example screen displayed during navigation application operation, showing Points of Interest (POIs) on the map and information about the first POI. The second layer, L2, displays a list of POIs, which includes those from the first layer, L1.

[0073] Figure 4 A priority chart is shown, which illustrates Figure 3 Example scores for the text on each screen. Figure 4 In the middle, the first chart is Figure 3 The first layer, L1, displays a priority chart of the text, while the second chart is... Figure 3 The second layer, L2, displays a priority chart of the text.

[0074] Figure 4 The priority chart shows Figure 3 Example Z-index score S of screen text Z . refer to Figure 4 , Figure 3 In the example, the second layer (L2) is the top layer, so text belonging to the second layer (L2) has the highest Z-index score of 0.9. Text belonging to the first layer (L1) has a relatively high Z-index score of 0.5.

[0075] At the same time, the information or functions provided by the application are often presented to the user in a hierarchical / tree structure to make navigation easier. For example, the settings screen may display a settings menu that includes sub-items, each of which can be selected to display a sub-settings menu that includes even smaller sub-items.

[0076] In at least one implementation, the priority sorting module 230 can calculate the priority of screen text based on a text tree structure. For example, text belonging to a lower-level item can be assigned a higher priority than text belonging to a higher-level item.

[0077] The priority sorting module 230 can prioritize screen text based on text similarity and / or font style.

[0078] Users may not repeat the screen text word for word, but rather omit or paraphrase some parts. Even if the user repeats the screen text word for word, when the speech recognition module 210 recognizes the user's speech and converts it into an input text string, there may still be spaces or spelling differences between the user's speech and the input text string. Similarly, there may be differences between the input text string and the screen text. In addition, multiple texts similar to the user's speech may be displayed on the screen.

[0079] Accordingly, the priority ranking module 230 can prioritize the text based on the similarity between the input text string and the screen text. The priority ranking module 230 can calculate the similarity between the input text string and the screen text using an approximate string matching algorithm, which compares the similarity between the two texts. For example, a fuzzy matching algorithm can be used as an approximate string matching algorithm to compare the similarity between two text strings, but the invention is not limited thereto.

[0080] In at least one implementation, the similarity between the input text string and the screen text can be represented as a similarity score.

[0081] Figure 4 The priority chart shows that when the input text string is, for example, "Starbucks Pohang Steel Branch", Figure 3 An example similarity score S of the screen text S .

[0082] refer to Figure 4 In the first layer L1 text, "Starbucks Samsung Store" has a similarity score of 0.5 because it is partially similar to the input text string; while "Business Hours" and "Phone Number" have a similarity score of 0.1 because they are not similar to the input text string.

[0083] refer to Figure 4 Among the text belonging to the second level (L2), "Starbucks Samsung Store", "Starbucks Pohang Steelhouse Store", "Starbucks Daecheon Crossroads Store", and "Starbucks Pohang Steelhouse Crossroads Store" are highly similar to the input text string, thus having similarity scores of 0.5, 0.9, 0.4, and 0.8 respectively. The other texts are not similar to the input text string, and therefore have a similarity score of 0.1.

[0084] In this way, the priority sorting module 230 can prioritize multiple similar texts, such as "Starbucks Pohang Steel House" and "Starbucks Pohang Steel House Crossroads", by comparing the text with the input text string.

[0085] At the same time, when users may refer to highlighted text while speaking, the creators of the application screens can emphasize important or frequently used features by changing the font style of the corresponding text displayed on the screen.

[0086] Accordingly, the priority sorting module 230 can prioritize control instructions based on font style. For example, when text is displayed in a large font size and highlighted (e.g., bold, italic, etc.), the text is given a higher priority.

[0087] In at least one implementation, the prioritization of text based on font style can be reflected by a font style score. That is, the font style score of large-font and highlighted text is determined to be higher than that of ordinary text.

[0088] Figure 4 The priority chart shows Figure 3 Exemplary font style score S for screen text F . refer to Figure 4 Within the first level (L1) of text, phrases like "business hours" and "phone number" have a font style score of 0.2 because they are plain text, while "Starbucks Samsung Store" has a font style score of 0.5 because it is bold. The numbers "1", "2", "3", and "4" have a font style score of 0.7 because their font size is larger than the other text and they are bold. All text in the second level (L2) has the same font size and font style, therefore they have the same font style score.

[0089] The priority sorting module 230 can prioritize screen text based on the functionality of GUI components. In at least one embodiment, the priority of text on operable GUI components is determined to be higher than the priority of text on inoperable GUI components.

[0090] An interactive GUI component is a graphical element that a user can interact with and is designed to perform a specific task or function. Examples of interactive GUI components include buttons, checkboxes, drop-down menus, toggle switches, etc.

[0091] Since interactive GUI components are the components that users use to interact with the system, users are likely to refer to interactive GUI components when speaking. Therefore, text on interactive GUI components should be given higher priority.

[0092] Non-interactive or non-operable GUI components, as opposed to operable GUI components, are graphical elements that do not provide direct interaction with the user. Examples of non-operable GUI components include labels.

[0093] Since non-operable GUI components are primarily designed for displaying information or page layout, users typically do not refer to them when making comments. Therefore, text on non-operable GUI components should be given lower priority.

[0094] On the other hand, the priority of text can be determined differently depending on the type of operable GUI component. For example, the text on a button can be given a higher priority than the text on a checkbox. However, the invention is not limited to this example.

[0095] Furthermore, priorities can be determined differently depending on whether the GUI component displays text or symbols. For example, a button that displays text may have a higher priority than a button that displays symbols. However, the invention is not limited to this example.

[0096] In at least one implementation, the priority of the text based on the functionality of the GUI component can be represented as a GUI component score.

[0097] Figure 4 The priority chart shows Figure 3 Example GUI component score S for screen text G .

[0098] refer to Figure 4 In the text belonging to the first layer L1, the numbers "1", "2", "3", "4" and "Reservation" have a GUI component score of 0.8 because they are text on buttons; while "Starbucks Samsung Store", "Business Hours", "Phone Number", etc. have a GUI component score of 0.1 because they are text on labels.

[0099] refer to Figure 4 The "telephone symbol," belonging to the second level (L2), has a GUI component score of 0.9 because it is a symbol on a button. The remaining text is button text, therefore they have a GUI component score of 0.8.

[0100] The priority sorting module 230 can prioritize text based on the Z-index score, similarity score, font style score, GUI component score, etc. of the text on each screen.

[0101] In at least one implementation, the priority sorting module 230 can prioritize the text in the following order: Z-index score, similarity score, font style score, and GUI component score.

[0102] Figure 5 This is a flowchart of a process for prioritizing screen text according to at least one embodiment of the present invention.

[0103] First, prioritize the texts based on their Z-index scores (step S510). Figure 4 In the example, the Z-index score of the text belonging to the second level (L2) is 0.9, which is higher than the Z-index score of the text belonging to the first level (L1) is 0.5. Therefore, the text belonging to the second level (L2) has a higher priority than the text belonging to the first level (L1).

[0104] Second, prioritize the texts based on their similarity scores (step S520). Figure 4In the example, if the user's utterance is "Starbucks Pohang Steel Branch", then "Starbucks Pohang Steel Branch" has the highest similarity score in the L2 text, and therefore is given the highest priority. Therefore, priority is assigned to "Starbucks Pohang Steel Branch" in the order of appearance: "Starbucks Pohang Steel Branch" with the highest similarity score, "Starbucks Pohang Steel Crossroads Branch" with the second highest similarity score, "Starbucks Samsung Branch" with the third highest similarity score, and "Starbucks Daecheon Crossroads Branch" with the lowest similarity score.

[0105] Third, prioritize the text based on font style scores (step S530). Figure 4 In the example, apart from the text that is prioritized in step S520, the remaining text belonging to the second level L2 has the same font style score, so the remaining text is not prioritized in step S530.

[0106] Finally, the text is prioritized based on the GUI component scores (step S540).

[0107] exist Figure 4 In the example, among the text belonging to the second level L2 that was not prioritized in steps S510 to S530, "telephone symbol" has the highest GUI component score of 0.9, therefore "telephone symbol" is given a higher priority among the texts that were not prioritized. The remaining texts that were not prioritized are all given the same priority.

[0108] In another embodiment, the priority sorting module 230 can determine the priority of a text by calculating a priority score based on Z-index score, similarity score, font style score, GUI component score, etc.

[0109] Formula 1 is used to calculate the priority P(t) of the screen text t.

[0110] P(t)=w1·Z(t)+w2·S(t)+w3·F(t)+w4·G(t) Formula 1 Z(t) represents the Z-index score of text t. For example, the top layer can have (but is not limited to) a Z-index score of 1, while the bottom layer can have (but is not limited to) a Z-index score of 0.1. S(t) represents the similarity score of text t. F(t) represents the font style score of text t. G(t) represents the GUI component score of text t. In one example, the text in a button can have (but is not limited to) a GUI component score of 1, while the text in a label can have (but is not limited to) a GUI component score of 0.1.

[0111] Furthermore, w1 is the weight of the Z-index score Z(t), w2 is the weight of the similarity score S(t), w3 is the weight of the font style score F(t), and w4 is the weight of the GUI component score G(t). By adjusting the relative values ​​of w1, w2, w3, and w4, the degree of influence of each score on the priority can be set. For example, w1=0.3, w2=0.4, w3=0.2, w4=0.1. However, the invention is not limited to this example.

[0112] In one example Figure 4 The priority chart shows the priority score (P) of the text, which is calculated using Formula 1, where w1=0.3, w2=0.4, w3=0.2, and w4=0.1. (Reference) Figure 4 If the user's utterance is "Starbucks Pohang Steel Branch", then "Starbucks Pohang Steel Branch" becomes the text with the highest priority score of 0.75. Therefore, the speech recognition engine 120 classifies "Select the second option"—corresponding to the utterance intent of "Starbucks Pohang Steel Branch"—as the user's utterance intent (voice command).

[0113] In other implementations, the priority of the text can be calculated as a compromise between the method of one implementation and the method of another implementation.

[0114] For example, the priority sorting module 230 can prioritize text by first sorting it based on the Z index score, and then calculating the priority score P(t) based on similarity score, font style score, GUI component score, etc.

[0115] As another example, the priority sorting module 230 can first calculate the priority score P(t) of the text based on the similarity score, Z index score and font style score, and finally sort the text with higher priority scores P(t) based on the functionality of the GUI component.

[0116] NLU module 220 can determine the user's speech intent based on the priority calculated by priority sorting module 230. For example, NLU module 220 can determine the user's speech intent from the voice command corresponding to the highest priority text in the screen text.

[0117] The speech recognition engine 120 may include at least one processor and a memory including at least one instruction. The speech recognition engine 120 may execute the instruction through the at least one processor to perform the functions of the speech recognition module 210, the NLU module 220, and the priority sorting module 230. The speech recognition engine 120 may further include a communication unit for communicating with external devices.

[0118] Figure 6 This is a flowchart of a process performed by a speech recognition engine 120 to classify a user's speech intent from a user's speech, according to at least one embodiment of the present invention.

[0119] The speech recognition module 210 acquires the user's speech received from the vehicle microphone and converts the user's speech into an input text string in text form (step S610).

[0120] NLU module 220 classifies the user's utterance intent based on the input text string (step S620). For example, NLU module 220 can determine the user's utterance intent by comparing the input text string with a prescribed grammar, or by using a deep learning model or the like.

[0121] If the NLU module 220 successfully classifies the speech intent (step S620 is yes), then the control command corresponding to the classified speech intent is sent to the infotainment device 130 and / or the controller 150.

[0122] If the NLU module 220 is unable to classify the speech intent due to conflicts between multiple voice commands (step S620 is no), then the NLU module 220 determines whether there is a highest priority command among the multiple voice commands (step S630). Here, the highest priority command is a global command that does not appear on the display of the output interface 140. In the event of a conflict with a local command, the highest priority command is determined as the final command regardless of the priority of the local command.

[0123] If a highest priority command exists among multiple voice commands (yes in step S630), the NLU module 220 determines the highest priority command corresponding to the input text string as the user's speech intent (step S640). For example, the command "open" belonging to the vehicle control domain is the highest priority command because it is a global command that does not appear on the display. Therefore, if the user says "open the window," the NLU module 220 will send a control command corresponding to the command "open" to the infotainment device 130 and / or controller 150 without performing a priority comparison.

[0124] If no highest priority command exists among the multiple voice commands (no in step 630), then the priority sorting module 230 will prioritize the screen text (step 650). Here, priority indicates the relevance of the screen text to the user's speech intent. Higher priority indicates that the text is more likely to be relevant to the user's speech intent, while lower priority indicates that the text is less likely to be relevant to the user's speech intent. For details on the method of prioritizing screen text, see [link to documentation]. Figure 5 This will not be elaborated upon here.

[0125] The NLU module 220 selects the highest priority text based on the priority of the text calculated by the priority sorting module 230, and determines the voice command corresponding to the selected text as the user's speech intent (step S660). The NLU module 220 sends the control command corresponding to the user's speech intent to the infotainment device 130 and / or the controller 150.

[0126] Figure 7 It is a schematic block diagram of a computing device that can be used to implement the method or apparatus according to the present invention.

[0127] The computing device 700 may include some or all of the following: a memory 710, a processor 720, a storage device 730, an input / output interface 740, and a communication interface 750. The computing device 700 may structurally and / or functionally include at least a portion of the apparatus according to the invention. The computing device 700 may be a fixed computing device (such as a desktop computer, server, etc.) or a mobile computing device (such as a laptop computer, smartphone, in-vehicle electronics, etc.). The computing device 700 may include any dedicated hardware accelerator capable of efficiently processing computations related to artificial intelligence models. For example, the computing device 700 may include a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0128] Memory 710 may store programs that cause processor 720 to perform methods or operations under various embodiments of the present invention. For example, the program may include a plurality of instructions that are executed by processor 720, and processor 720 may execute the plurality of instructions to perform the methods or operations described above. Memory 710 may be a single memory or multiple memories. For example, information required to perform methods or operations according to various embodiments of the present invention may be stored in a single memory or distributed among multiple memories. When memory 710 is composed of multiple memories, they may be physically separated. Memory 710 may include at least one of volatile memory and non-volatile memory. Volatile memory may include, for example, Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), and non-volatile memory may include, for example, flash memory.

[0129] Processor 720 may include at least one core capable of executing at least one set of instructions. Processor 720 may execute instructions stored in memory 710. Processor 720 may be a single processor or multiple processors.

[0130] Storage device 730 retains stored data even when computing device 700 is powered off. For example, storage device 730 may include non-volatile memory, or may include storage media such as magnetic tape, optical disc, or magnetic disk. Programs stored in storage device 730 can be loaded into memory 710 before execution by processor 720. Storage device 730 may store files written in a programming language, and programs generated by compilers, etc., can be loaded into memory 710 from these files. Storage device 730 may store data to be processed by processor 720 and / or data already processed by processor 720.

[0131] The input / output interface 740 can provide an interface with input devices (such as a keyboard, mouse, etc.) and / or output devices (such as a monitor, printer, etc.). Users can trigger the processor 720 to execute programs through the input devices and / or view the processing results of the processor 720 through the output devices.

[0132] The communication interface 750 can provide access to external networks. The computing device 700 can communicate with other devices through the communication interface 750.

[0133] The components of the apparatus and method can be implemented in hardware, software, or a combination of both. Furthermore, the functionality of each component can be implemented in software, and the software functionality of each component can be implemented and executed by a microprocessor.

[0134] Various exemplary implementations of the systems and methods described herein can be implemented using digital electronic circuits, integrated circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations within one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device, wherein the programmable processor may be a dedicated processor or a general-purpose processor. The computer program (also referred to as a program, software, software application, or code) includes instructions from the programmable processor and is stored in a computer-readable recording medium.

[0135] Computer-readable recording media include any type of recording device in which data readable by a computer system can be recorded. Examples of computer-readable recording media include non-volatile or non-transient media such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, optical / disk, storage device, etc. Computer-readable recording media may further include transient media such as data transmission media. Furthermore, computer-readable recording media can be distributed across computer systems connected via a network, where computer-readable code can be stored and executed in a distributed manner.

[0136] Although the steps in the flowcharts / sequence diagrams in this specification are described as being executed sequentially, they only embody the technical concepts of some embodiments of the present invention. Therefore, those skilled in the art regarding the relevant embodiments can perform the steps by changing the order described in the flowcharts / sequence diagrams, or by executing two or more steps in parallel, without departing from the concept and scope of the embodiments. Thus, the steps in the flowcharts / sequence diagrams are not limited to the temporal order shown.

[0137] While various aspects of the invention have been described for illustrative purposes, those skilled in the art will understand that various modifications, additions, and substitutions can be made without departing from the spirit and scope of the invention. Therefore, the various aspects of the invention have been described for the sake of brevity only. The scope of the technical concept of the embodiments of the invention is not limited by the illustrated descriptions. Accordingly, those skilled in the art should understand that the scope of the invention should not be limited by the embodiments explicitly described above, but should be defined by the claims and their equivalents.

Claims

1. A computer-implemented method for executing voice commands by a computing device, comprising: The input text string is obtained by a computing device by converting the user's speech into text; The computing device determines whether the input text string is classified as a highest priority instruction; Based on the fact that the input text string is not classified as the highest priority instruction, the computing device determines the priority of multiple texts displayed on the screen. and The computing device executes the voice command corresponding to the highest priority text among multiple texts. Among them, the highest priority instruction is the global voice instruction that is not displayed on the screen; The priority is determined based on the relevance of the text among the multiple texts displayed on the screen to the user's verbal intent.

2. The computer implementation method according to claim 1, wherein, The input text string is classified as a highest priority instruction, and the computer implementation method includes executing the classified highest priority instructions by a computing device.

3. The computer implementation method according to claim 1, wherein, The priority is determined based on the Z index value assigned to each of the multiple screen layers on the display screen.

4. The computer implementation method according to claim 1, wherein, The determination of the priority includes: An approximate string matching algorithm is used to calculate the similarity between each of the multiple texts and the input text string; Prioritize multiple texts based on the similarity of each text among them.

5. The computer implementation method according to claim 4, wherein, The approximate string matching algorithm includes a fuzzy matching algorithm.

6. The computer implementation method according to claim 1, wherein, The determination of the priority includes: The priority of multiple texts is determined based on their font styles.

7. The computer implementation method according to claim 1, wherein, The determination of the priority includes: The priority of multiple texts is determined based on whether each graphical user interface component displaying multiple texts is an operable graphical user interface component.

8. The computer implementation method according to claim 1, wherein, The determination of the priority includes: A priority score is calculated for each of the multiple texts based on the following scores: i) Z-index score, ii) similarity score, iii) font style score, iv) graphical user interface component score, or any combination of i) to iv); The Z-index score is calculated based on the Z-index value assigned to each of the multiple layers that make up the display screen. The similarity score is calculated by comparing the input text string with the text displayed on the screen using an approximate string matching algorithm; The font style score is calculated based on the font style of the text displayed on the screen; The graphical user interface component score is calculated based on whether the graphical user interface component can be operated.

9. The computer implementation method according to claim 8, wherein, The priority score is a weighted sum of the Z-index score, similarity score, font style score, and graphical user interface component score.

10. The computer implementation method according to claim 1, wherein, The multiple texts include: Display the symbols shown on the screen.

11. An apparatus for executing voice commands based on displayed information, comprising: At least one memory configured to store instructions; and At least one processor; Wherein, the at least one processor is configured to execute instructions to: Determine the input text string by converting the user's utterance into text; Determine whether the input text string is classified as a highest priority instruction; Determine the priority of multiple texts displayed on the screen based on the fact that the input text string was not classified as a highest priority instruction; and Execute the voice command corresponding to the highest priority text among multiple texts; Among them, the highest priority instruction is the global voice instruction that is not displayed on the screen; The priority indicates the degree of relevance between the text displayed on the screen and the user's verbal intent.

12. The apparatus for executing voice commands based on display information according to claim 11, wherein, The input text string is classified as a highest priority instruction, and the processor is configured to execute the classified highest priority instruction.

13. The apparatus for executing voice commands based on display information according to claim 11, wherein, The priority of the multiple texts is determined based on the Z index value assigned to each of the multiple screen layers displayed on the display screen.

14. The apparatus for executing voice commands based on display information according to claim 11, wherein, The priority of the multiple texts is determined based on the similarity between each of the multiple texts and the input text string; The similarity is calculated by comparing the input text string with the text displayed on the screen using an approximate string matching algorithm.

15. The apparatus for executing voice commands based on display information according to claim 14, wherein, The approximate string matching algorithm includes a fuzzy matching algorithm.

16. The apparatus for executing voice commands based on display information according to claim 11, wherein, The priority of the multiple texts is determined based on their font styles.

17. The apparatus for executing voice commands based on display information according to claim 11, wherein, The priority of the multiple texts is determined based on whether each graphical user interface component displaying the multiple texts is an operable graphical user interface component.

18. The apparatus for executing voice commands based on display information according to claim 11, wherein: The priority of the multiple texts is determined based on a priority score, which is calculated based on the following scores: i) Z-index score, ii) similarity score, iii) font style score, iv) graphical user interface component score, or any combination of i) to iv); The Z-index score is calculated based on the Z-index value of each of the multiple layers that make up the display screen. The similarity score is calculated by comparing the input text string with the text displayed on the screen using an approximate string matching algorithm; The font style score is calculated based on the font style of the text displayed on the screen; The graphical user interface component score is calculated based on whether the graphical user interface component can be operated.

19. The apparatus for executing voice commands based on display information according to claim 18, wherein, The priority score is a weighted sum of the Z-index score, similarity score, font style score, and graphical user interface component score.

20. The apparatus for executing voice commands based on display information according to claim 11, wherein, The multiple texts include: Display the symbols shown on the screen.

Images