Methods, devices, terminals, and storage media for voice broadcasting notification messages

Abstract

This application discloses a method, apparatus, terminal, and storage medium for voice broadcasting of notification messages, belonging to the field of human-computer interaction. The method includes: extracting features from the message content and message source of the notification message to obtain a first notification semantic feature; if the notification message includes a structured graphical user interface, extracting features from the element attribute information of the interface elements in the structured graphical user interface to obtain a second notification semantic feature; generating a message prompt text corresponding to the notification message based on the first and second notification semantic features; and broadcasting the message prompt text via voice. Using the solution provided by this application, when it is inconvenient for the user to view the message, notification messages displayed in the form of a structured graphical user interface can be conveyed to the user through voice broadcasting.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a method, apparatus, terminal and storage medium for voice broadcasting notification messages. Background Technology

[0002] When users are unable to view notification messages, reading them aloud via voice is a suitable interaction method.

[0003] In related technologies, converting notification messages into speech mainly involves obtaining the text description information of the notification message, and then using speech conversion technology to convert the text description information into speech information for broadcast to the user.

[0004] However, in practical applications, voice broadcasts of notification messages using related technologies often fail to accurately convey the content of the notification messages to users. Summary of the Invention

[0005] This application provides a method, apparatus, terminal, and storage medium for voice broadcasting notification messages. The technical solution is as follows:

[0006] On the one hand, this application provides a method for voice broadcasting notification messages, the method comprising:

[0007] Feature extraction is performed on the message content and message source of the notification message to obtain the first notification semantic feature of the notification message;

[0008] When the notification message contains a structured graphical user interface, feature extraction is performed on the element attribute information of the interface elements in the structured graphical user interface to obtain the second notification semantic feature of the notification message.

[0009] Based on the first notification semantic features and the second notification semantic features, generate the message prompt text corresponding to the notification message;

[0010] The message prompt text is read aloud via voice.

[0011] On the other hand, this application provides a voice broadcasting device for notification messages, the device comprising:

[0012] The first feature extraction module is used to extract features from the message content and message source of the notification message to obtain the first notification semantic features of the notification message.

[0013] The second feature extraction module is used to extract the element attribute information of the interface elements in the structured graphical user interface when the notification message contains a structured graphical user interface, so as to obtain the second notification semantic feature of the notification message.

[0014] The text generation module is used to generate message prompt text corresponding to the notification message based on the first notification semantic features and the second notification semantic features;

[0015] The voice broadcast module is used to broadcast the message prompt text via voice.

[0016] On the other hand, embodiments of this application provide a terminal, the terminal including a processor and a memory; the memory stores at least one instruction, the at least one instruction being executed by the processor to implement the voice broadcasting method for notification messages as described above.

[0017] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the voice broadcasting method for notification messages as described above.

[0018] On the other hand, embodiments of this application provide a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the voice broadcasting method for notification messages provided in various optional implementations of the above aspects.

[0019] In this embodiment, the terminal extracts features from the message content, message source, and element attribute information of the structured graphical user interface of the notification message to obtain the first notification semantic features and the second notification semantic features, and then obtains the prompt text of the notification message, which is then broadcast to the user via voice. On the one hand, it can extract key information from the notification message, making the prompt text as concise as possible and able to accurately convey the content of the notification message to the user. On the other hand, it can also integrate the discrete and fragmented information in the notification message in the form of a structured graphical user interface into a coherent and natural text description, which is more conducive to the user's understanding of the notification message. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a voice broadcasting method for notification messages provided in an exemplary embodiment of this application is shown;

[0021] Figure 2This illustration shows a schematic diagram of a notification message displayed using a structured graphical user interface provided in an exemplary embodiment of this application;

[0022] Figure 3 A flowchart illustrating the process of generating descriptive text provided in an exemplary embodiment of this application is shown;

[0023] Figure 4 A flowchart illustrating the process of obtaining the semantic features of the second notification provided in an exemplary embodiment of this application is shown;

[0024] Figure 5 A schematic diagram of a notification message interface provided in an exemplary embodiment of this application is shown;

[0025] Figure 6 A flowchart illustrating a voice interaction process provided in an exemplary embodiment of this application is shown;

[0026] Figure 7 This paper shows a structural block diagram of a voice broadcasting device for notification messages according to an embodiment of the present application;

[0027] Figure 8 A structural block diagram of a terminal provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0029] It should be noted that all embodiments of this application are executed on the terminal after the voice broadcast program is triggered, and this application does not limit the method of triggering the voice broadcast program. Furthermore, all structured graphical user interface data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with relevant national and regional laws, regulations, and standards.

[0030] Figure 1 A flowchart of a voice broadcasting method for notification messages provided in an exemplary embodiment of this application is shown. The method includes:

[0031] Step 101: Extract features from the message content and message source of the notification message to obtain the first notification semantic features of the notification message.

[0032] Before displaying a notification message to a user, the terminal first obtains information such as the notification source and message content. This mainly includes system messages, user messages, and third-party messages. The content of the notification message refers to the text content of the notification message, while the message source refers to the attribute information of the notification source, including the application type and the user's historical application records. For example, if the notification source is a news application, the terminal can obtain the application type of the notification message as news, the user's historical application frequency in the past month as 30 times, and the user's historical applicable time period as 8:00-9:00 before constructing the notification message.

[0033] After the terminal obtains the message content and source of the notification message, it encodes the different content using different methods. Optionally, the message content can be encoded using text encoding, while the message source can be encoded using one-hot encoding. After encoding both separately, the encoding results are concatenated to obtain the message encoding vector corresponding to the notification message.

[0034] After obtaining the message encoding vector corresponding to the notification message, the terminal can extract features by means of Transformer model, constructing evaluation function, skip-gram model, etc., to obtain the first semantic features of the notification message.

[0035] Optionally, the terminal performs feature extraction using a Transformer model. The message encoding vector is input into the Transformer model, passing sequentially through an encoder framework and a decoder framework. Each encoder and decoder framework consists of multiple independent feature extractors stacked together. The initial input message encoding vector, after passing through the encoder framework, yields a matrix or vector, which is analogous to an encoding of the message encoding vector. The decoder structure is flexible and can decode the matrix or vector according to different application scenarios. The output result is the first semantic feature of the notification message.

[0036] Step 102: If the notification message contains a structured graphical user interface, feature extraction is performed on the element attribute information of the interface elements in the structured graphical user interface to obtain the second notification semantic feature of the notification message.

[0037] In practical applications, the terminal may be displayed in the form of a structured graphical user interface. Therefore, the message prompt text generated solely through the first semantic feature may not accurately and clearly express the new content of the notification message. Thus, it is necessary to extract the element attribute information of the entire structured graphical user interface, extract features from them, obtain the element semantic information corresponding to each element, and then fuse the semantic information of multiple elements to obtain the second notification semantic feature of the notification message.

[0038] Step 103: Based on the semantic features of the first notification and the semantic features of the second notification, generate the message prompt text corresponding to the notification message.

[0039] The message prompt text is used to describe the key information in the notification message.

[0040] After the terminal obtains the semantic features of the first notification and the semantic features of the second notification, it first fuses the two features, and then uses text generation and text summarization to generate the message prompt text of the notification message.

[0041] Optionally, text generation techniques from numerical values ​​to text can be used to process the semantic features of the structured graphical user interface, and then simplified using text summarization algorithms to finally generate message prompt text.

[0042] Step 104: Read the message prompt text aloud.

[0043] After generating a message prompt text, the terminal will read the content of the message prompt text aloud. In one possible implementation, the terminal uses TTS (Text-to-Speech) technology to convert the message prompt text into a speech signal. TTS mainly consists of two parts: speech analysis and an acoustic system. Speech analysis mainly analyzes the text information of the input message prompt text, determines the text structure and language, standardizes the text, converts the text into phonemes, and finally predicts sentence structure and prosody. The acoustic system part generates the corresponding audio based on the results of the speech analysis, thus realizing the function of converting the message prompt text into a speech signal.

[0044] like Figure 2 As shown, for the first graphical user interface 201, after generating the message prompt text, the terminal broadcasts the message prompt text via voice as follows: "Received a flight assistant message. Your China Eastern Airlines flight MU5332 is scheduled to depart from Shenzhen Bao'an T3 at 07:15 on July 17th and is scheduled to arrive at Shanghai Pudong T1 at 09:35. Check-in counters are E01-E18, and the boarding gate is F28." After the user's aircraft lands, the notification message will change to the second notification message 202. Figure 2 The second notification message 202 generated by the terminal reads: "Received a flight assistant message. Your China Eastern Airlines flight MU5332 arrived at Shanghai Pudong Terminal 1 at 09:35. The flight time was 2 hours and 20 minutes, the flight distance was 1343 km, and the baggage carousel number is 26."

[0045] In summary, in this embodiment, the terminal extracts features from the message content, message source, and element attribute information of the structured graphical user interface of the notification message to obtain the first notification semantic features and the second notification semantic features, and then obtains the notification message prompt text, which is then broadcast to the user via voice. On the one hand, it can extract key information from the notification message, making the prompt text as concise as possible and able to accurately convey the content of the notification message to the user. On the other hand, it can also integrate the discrete and fragmented information in the notification message in the form of a structured graphical user interface into a coherent and natural text description, which is more conducive to the user's understanding of the notification message.

[0046] After the terminal obtains the semantic features of the first notification and the semantic features of the second notification, it will fuse the two features to obtain the semantic fusion features of the notification message.

[0047] The fusion can be direct fusion or weighted fusion. The fusion methods that can be used include maximum fusion, MFB (Multimodal Factorized Bilinear Pooling) and artificial neural network methods.

[0048] After obtaining the semantic fusion features, the terminal inputs the semantic fusion features into the text generation model to obtain the descriptive text corresponding to the notification message.

[0049] The text generation model has the functions of text generation and text summarization. The first text generation model is composed of the text generation architecture and the text summarization architecture. After the semantic fusion features are input into the first text generation model, they are first processed by the text generation architecture to generate a preliminary description text corresponding to the content of the notification message. Then, it is input into the text summarization architecture to simplify the content of the preliminary description text and finally obtain the first description text corresponding to the notification message.

[0050] In practical applications, using a single text generation model to process semantic fusion features for both text generation and summarization places high demands on the model's processing capabilities and may not achieve optimal results. Therefore, the terminal can use two text generation models sequentially to process semantic fusion features for both text generation and summarization, achieving better text processing performance.

[0051] After receiving the description text, the terminal will generate a corresponding message prompt text based on the description text. This includes the following two methods:

[0052] Method 1: By combining the description text with the first message prompt template, you can obtain the message prompt text corresponding to the notification message.

[0053] The first message template is a simple text template used to present descriptive text in a way that is more in line with natural language conversation. For example, the first message template is: "Received a message from XXX..." where "XXX" is used to fill in the name of the message source, and "..." is used to fill in the descriptive text corresponding to the notification message.

[0054] Generating the message prompt text corresponding to the notification message based on the first description text involves combining the first description text with the first message template to obtain the message prompt text corresponding to the notification message.

[0055] Method 2: Identify the interactive elements in the structured graphical user interface, extract the element description text of the interactive elements from their element attribute information, and then concatenate the description text and the element description text using a second message prompt template to obtain the message prompt text.

[0056] Typically, a structured graphical user interface for a notification message includes some interactive elements. When the message is read aloud to the user, these interactive elements can be conveyed to the user through message prompts to facilitate the user's next action.

[0057] First, identify the interactive elements in the structured graphical user interface. This can be done by obtaining the view tree of the structured graphical user interface, reading the element attribute information in the view tree, and determining whether the interface element is interactive by reading whether the element attribute information has interactive attributes. Interactive attributes include clickable attributes and input attributes, etc.

[0058] Next, extract the element description text of the interactive element from its element attribute information. This description text includes both the visible text attribute (Text attribute) and the invisible text attribute (ContentDescription attribute) of the interface element. The Text attribute is visible to the user and displayed as text in the structured graphical user interface. The ContentDescription attribute is an attribute written by the developer inside the interface element, used to describe the functionality of the interface element. It is invisible in the structured graphical user interface; developers hide some text in the ContentDescription attribute for aesthetic purposes.

[0059] Finally, the description text and element description text are concatenated using the second message prompt template to obtain the message prompt text. The second message prompt template is a simple connecting text used to concatenate the description text and element description text. For example, a second message prompt template might be: "Received a message from XXX... For further action, you can say ***", where "XXX" is used to fill in the name of the message source, "..." is used to fill in the description text corresponding to the notification message, and "***" is used to fill in the description text of the interactive element. For example, for a terminal targeting... Figure 2 The message generated by the first graphical user interface 201 is as follows: "Received a flight assistant message. Your China Eastern Airlines flight MU5332 is scheduled to depart from Shenzhen Bao'an T3 at 07:15 on July 17 and arrive at Shanghai Pudong T1 at 09:35. Check-in counters are E01-E18, and the boarding gate is F28. For further operations, you can say 'flight assistant,' 'view your order,' or 'book a hotel.'"

[0060] The following exemplary embodiment will illustrate the process by which a terminal obtains descriptive text through two text generation models.

[0061] Figure 3 A flowchart of a process for generating descriptive text provided in an exemplary embodiment of this application is shown, the process including:

[0062] Step 301: Input the semantic fusion features into the second text generation model to obtain the second descriptive text corresponding to the notification message. The second text generation model has text generation function.

[0063] After the terminal obtains the semantic fusion features, it inputs the semantic fusion features into the second text description model for text generation processing to obtain the second description text corresponding to the notification message.

[0064] In one possible implementation, the framework of the first text generation model includes a signal analysis module, a data interpretation module, a document planning module, and a microplanning and implementation module. The signal analysis module takes semantic fusion features as input, uses various data analysis methods to detect basic patterns in the semantic fusion features, and outputs discrete data patterns. The data interpretation module takes basic patterns and events from the semantic fusion features as input, analyzes these patterns and events to infer more complex and abstract messages, and infers the relationships between messages, finally outputting high-level messages and the relationships between messages, such as causal relationships and temporal relationships. The document planning module takes messages and the relationships between messages as input, analyzes and determines the messages to be mentioned in the first descriptive text, determines the structure of the first descriptive text, and finally outputs the messages to be mentioned and the document structure. The microplanning and implementation module takes simplified messages to be mentioned and the document structure as input, and outputs a second descriptive text using natural language generation techniques.

[0065] The first text generation model enables the final generated second descriptive text to have correct syntax and voice, as well as information with a structured graphical user interface.

[0066] Step 302: Extract features from the second descriptive text and the message source to obtain the third notification semantic features of the notification message.

[0067] After the terminal obtains the second description text, it encodes it and the message source separately to obtain their respective encoding vectors, and then fuses the two encoding vectors to obtain the third notification semantic feature of the notification message.

[0068] Step 303: Input the semantic features of the third notification into the third text generation module to obtain the third descriptive text of the notification message. The third text generation model has a text summarization function.

[0069] The terminal uses a third text generation model to generate and summarize the semantic features of the input third notification. Text summarization is the main function of the third text generation model.

[0070] The third text generation model can use various algorithm models from either extraction-based or summary-based text summarization to process text summarization, which will not be elaborated here.

[0071] Step 304: Based on the third description text, generate the message prompt text corresponding to the notification message.

[0072] There are two ways to generate message prompt text based on the third description text. One is to directly fill the third description text into the first message prompt template. The other is to combine the description text of the interactive element and fill the second message prompt template together. The implementation process of these two methods has been described in the above-mentioned Method 1 and Method 2, and will not be repeated here in the embodiments of this application.

[0073] In this embodiment, the terminal first generates text from the semantically fused text using a second text generation model. Then, the generated second descriptive text and the message source are re-encoded, and finally, the text is processed by a third text generation model for text summarization to obtain the message prompt text. Separating the text generation and text summarization processes reduces the performance requirements of the text generation model, and the generated message prompt information is more representative of the notification message's content, allowing users to receive the message more efficiently.

[0074] When generating the message prompt text corresponding to the notification message, if the notification message is presented in the form of a structured graphical user interface, the terminal needs to extract the element attribute information of the interface elements to obtain the second notification semantic features, which are then fused with the first notification semantic features to generate the descriptive text. However, some elements in the structured graphical user interface are invisible, meaning that even during visual interaction, users cannot see the content of these invisible controls. Therefore, when extracting the second notification semantic features, it is only necessary to extract the features from the element attribute information of the visual interface elements.

[0075] The process of obtaining the semantic features of the second notification will be described below through an illustrative embodiment.

[0076] Figure 4 A flowchart illustrating a process for obtaining a second notification semantic feature according to an exemplary embodiment of this application is shown, the process including:

[0077] Step 401: Obtain the view tree of the structured graphical user interface.

[0078] The View Tree consists of interface elements in a structured graphical user interface, which is used to display notification messages in a structured manner.

[0079] Typically, structured graphical user interfaces (GUIs) used to display notifications are rendered and organized into a tree structure at the underlying level—the view tree. The view tree contains a root node, whose internal nodes are all of type ViewGroup, and the leaf nodes are all of type View. A ViewGroup node can contain multiple ViewGroup nodes and View nodes. View nodes are the elements that actually display content in the user interface, such as buttons, list elements, and text boxes. The view tree structure of the user interface can be obtained through services provided by the system's underlying layer, and the element attributes of each node in the view tree can also be obtained, such as displayed content, clickability, and position coordinates.

[0080] Step 402: Extract the element attribute information of the interface elements from the view tree.

[0081] While acquiring the view tree, the terminal can also obtain the element attribute information of each node in the view tree of the interface elements. After obtaining the element attribute information, the terminal extracts the element attribute information from the view tree for subsequent processing.

[0082] After the terminal extracts the element attribute information, it performs feature extraction on the element attribute information to obtain the element semantic features of the interface elements. When only targeting visual interface elements in a structured graphical user interface, the terminal needs to first determine the visual interface elements in the structured graphical user interface, and then perform feature extraction on the element attribute information of the visual interface elements to obtain the element semantic features.

[0083] Step 403: Based on element attribute information, determine the visual interface elements in the structured graphical user interface.

[0084] The terminal can determine whether an interface element is a visual interface element based on the obtained element attribute information, and can filter visual interface elements by traversing leaf nodes or querying attributes.

[0085] Optionally, visual elements in a structured graphical user interface can be determined by checking whether the Visibility property of the element is in the View.VISIBLE state. This is the most basic way to check whether an element is visible. Alternatively, this method can be used to recursively check whether the Visibility property of the View and its parentView is in the View.VISIBLE state. If the element is in the View.VISIBLE state, it is a visible element.

[0086] Step 404: Extract features from the element attribute information of the visual interface elements to obtain the element semantic features of the visual interface elements.

[0087] When the terminal extracts features from element attribute information, it first encodes the attribute information corresponding to each attribute of the interface element using different encoding methods. For example, it encodes text attributes using a text encoder, encodes the position coordinates of the element using normalized encoding, and encodes the clickability of the element using one-hot encoding, etc.

[0088] After the terminal obtains the encoding results of each attribute of the interface element, it concatenates the encoding results of each attribute to obtain the element semantic features.

[0089] Step 405: Extract and fuse the semantic features of the elements to obtain the second notification semantic features of the notification message.

[0090] After obtaining the semantic features of the elements, the terminal needs to extract and fuse the element attribute information of each visual interface element to obtain the semantic features of the interface elements in the entire structured graphical user interface, which is the second notification semantic feature of the notification message.

[0091] The semantic features of elements can be extracted and fused using neural network models, such as RNN (Recurrent Neural Network) or Transformer models.

[0092] In one possible implementation, the terminal employs a Transformer model to extract and fuse element semantic features. The Transformer model can be divided into an encoder and a decoder. The core idea of ​​using the Transformer model for element semantic feature extraction and fusion is that, given the known mapping relationship between the input vector K and the output vector V, a set of linear combination coefficients can be obtained by calculating the relationship between the unknown encoded output vector Q and K. The encoding result of the Q vector is obtained from the V vector and the corresponding linear combination coefficients. During pre-training of the encoder, the element semantic features are input into the computational network. The self-attention mechanism simultaneously uses the input element semantic features as three sets of vectors—Q, K, and V—for training. After training by the encoder, the corresponding K and V vectors are obtained. These two vectors are then used as hidden layers in the decoder to generate the second notification semantic features.

[0093] In this embodiment, the terminal obtains the view tree of the structured graphical user interface, then obtains the element attribute information of the interface elements, and determines the visual interface elements through the element attribute information. Then, it extracts features from the element attribute information of the visual interface elements, so that the second notification semantic features have more visible content features of the notification message. This makes the generated message prompt text more consistent with the information that the user can obtain under visual interaction, which is more conducive to avoiding redundancy in message prompt text content. Furthermore, by obtaining the view tree, the terminal can avoid application adaptation and has stronger applicability.

[0094] In practical applications, some notification messages are unimportant, such as item recommendations from shopping apps, video push notifications from short video apps, or daily news push notifications from news apps. Therefore, the terminal can first determine the importance of the notification message, and if the notification message is important, the terminal will issue a notification reminder. In addition, some notification messages do not have strong real-time requirements or are from sources the user does not frequently use. Therefore, the terminal can first determine whether the notification message needs to be read aloud, and if so, read it aloud. Some notification messages are short or have specific content, and feature extraction can be skipped; the message content can be read directly. For example, if the notification message originates from a friend's message on a social application, the terminal can directly read the message aloud.

[0095] In summary, under the three circumstances described above, if the terminal notification message is an important message, requires voice prompts, and requires content extraction, a message prompt text will be generated.

[0096] The determination of the importance of a notification message, whether it requires voice prompts, and whether it requires content extraction can be mainly done through the following two methods:

[0097] Method 1: Make judgments based on predefined logical rules.

[0098] Predefined logical rules can be user-defined rules or rules set by the terminal based on the type of message source and historical usage.

[0099] The logical rules are set as a rule table, which is determined based on the message source, message content, and device status. If the notification message meets the criteria of an important message in the rule table and has a voice prompt requirement, the notification message will be read aloud. If the notification message meets the criteria of an important message in the rule table but does not have a voice prompt requirement, the notification message will be reminded with sound effects or ringing. If the notification message does not meet the criteria of an important message and does not have a voice prompt requirement, it will not be reminded or will be displayed as a silent reminder.

[0100] If it is determined that the notification message needs to be read aloud, the content extraction requirement of the notification message is judged according to the defined logical rules. These logical rules are determined based on the message source and message content. After determining that the notification message has a content extraction requirement, the terminal performs feature extraction on the structured graphical user interface and the message content to obtain the first notification semantic feature and the second notification semantic feature of the notification message. Then, based on the first notification semantic feature and the second notification semantic feature, the corresponding message prompt text is generated and read aloud.

[0101] Method 2: Detection is performed using a neural network model.

[0102] When determining whether a notification message requires a voice prompt, a neural network model can also be used. The terminal uses the semantic fusion feature obtained by fusing the first and second semantic features of the notification message as input to the neural network model. Two neural network models are then used to determine the importance of the notification message and whether a voice prompt is required, respectively. This neural network can be a convolutional neural network, a recurrent neural network, etc.

[0103] If it is determined that the notification message needs to be read aloud, the terminal classifies the notification message based on the message source and message content, and obtains the classification result. The classification result is used to indicate whether the notification message has a content extraction requirement.

[0104] In one possible implementation, the terminal uses a notification message classification model to determine whether a notification message requires content extraction. The notification message classification model is pre-trained to calculate the probability that a notification message has a content extraction requirement. The terminal inputs semantic fusion features into the notification message classification model to obtain an output result, which represents the probability that a notification message has a content extraction requirement. Notification messages with a probability higher than a threshold are identified as having a content extraction requirement. This notification message classification model is trained based on the semantic fusion features of a large number of sample notification messages.

[0105] In fact, the determination of whether a notification message is important and whether it requires voice reminders can also be achieved through a classification model. The determination process is similar to the method described above for determining whether a notification message requires content extraction, and will not be elaborated on in this embodiment.

[0106] Figure 5 A schematic diagram of a notification message interface provided in an exemplary embodiment of this application is shown. Figure 5 This includes multiple notification messages: notification message 501, notification message 502, notification message 503, and notification message 504. Upon receiving these notification messages, the terminal first determines whether each message requires voice broadcasting and content extraction. First, the terminal determines the current device status as driving mode and, based on the message content and source, identifies notification messages 501, 502, and 504 as important messages. Further evaluation is then performed. Since the event triggered by notification message 504 is not urgent, the terminal determines that notification message 504 does not require voice notification. If the terminal determines that notification messages 501 and 502 require voice notification, based on their message content and source, it can be determined that these notification messages do not require content extraction. Therefore, the terminal broadcasts the following messages to the user: "Received a text message from Zhang X, we'll meet at the store entrance in a bit," and "Received a WeChat message from Li X, I'll bring the materials tomorrow."

[0107] After the terminal broadcasts a voice message, it can respond to the user's voice command, trigger the element indicated by the voice command, and further complete the voice interaction process.

[0108] Figure 6 A flowchart of a voice interaction process provided in an exemplary embodiment of this application is shown, the process including:

[0109] Step 601: Upon receiving a voice command, generate the command text corresponding to the voice command.

[0110] After receiving a voice command, the terminal sequentially executes a beamforming algorithm, front-end signal processing, and an ASR (Automatic Speech Recognition) algorithm to generate the corresponding command text.

[0111] Optionally, the front-end signal processing uses the ANC (Active Noise Cancellation) algorithm to eliminate ambient noise; the AEC (Acoutic Echo Cancellation) algorithm to eliminate voice echo in the terminal's voice broadcast; and the AGC (Automatic Gain Control) algorithm to adjust the amplitude range of the voice signal so that the amplitude of the processed output signal is stable.

[0112] Step 602: Determine the matching text in the notification message that matches the instruction.

[0113] After the terminal generates the instruction text, it matches it against the bounding elements in the structured graphical user interface (GUI) to determine the matching text within the GUI that corresponds to the instruction text. This matching operation can be implemented using a text matching algorithm, which will not be elaborated upon in this embodiment.

[0114] Once the terminal identifies the matching text, it can execute the operation indicated by the voice command based on the position of the element to which the matching text belongs. However, in practical applications, there may be situations where the matching text is not interactive. Therefore, to avoid program execution errors, it is necessary to determine whether the matching text is an interactive element before executing the operation indicated by the voice command.

[0115] Step 603: If the element to which the matched text belongs is an interactive element, execute the operation indicated by the voice command based on the position of the element to which the matched text belongs.

[0116] Interactive elements are determined by querying the element's attribute information to see if it possesses an interactive attribute. Once the terminal determines that the interface element to which the matching text belongs is an interactive element, it performs the corresponding operation on that interface element according to the instruction text. For example, for... Figure 2 In the first graphical user interface 201, the user issues a voice command: "View Orders". The terminal can determine that the matching text is the text "View Orders" in the first graphical user interface 201, which corresponds to the interface element 203. The terminal determines that the element is an interactive element based on the element attribute information of the element. Therefore, the interface element 203 is triggered, and the user jumps to the user interface corresponding to "View Orders".

[0117] Step 604: If the element to which the matched text belongs is not an interactive element, provide an interactive prompt.

[0118] If it is determined that the element to which the matched text belongs is not an interactive element, the terminal provides a non-interactive prompt to the user based on the content of the instruction text, and announces the text attributes of the interactive elements in the structured graphical user interface to the user.

[0119] For example, targeting Figure 2 In the first graphical user interface 201, when the user issues a voice command such as "Click on the check-in counter", the terminal determines that the "check-in counter" in the first graphical user interface 201 is a non-interactive element based on the element attribute information of the interface elements. At this time, the terminal's voice broadcast content can be: "The check-in counter you indicated does not support interaction. If you need further operation, you can say Flight Assistant, View Orders, or Book a Hotel".

[0120] In this embodiment, after the terminal obtains the user's voice signal, it determines the text that matches the instruction text, and then judges the interactivity of the elements in the matching text. For interactive elements, the terminal triggers the corresponding element to execute the operation indicated by the voice instruction. For non-interactive elements, the terminal prompts the user and provides interactive elements, so that the terminal can realize the complete interaction between the user and the terminal through voice alone, and can guide the user so that the user can more clearly understand the interactive elements in the notification message and can more smoothly carry out voice interaction.

[0121] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0122] Figure 7 A structural block diagram of a voice broadcasting device for notification messages according to an embodiment of this application is shown. The device may include:

[0123] The first feature extraction module 701 is used to extract features from the message content and message source of the notification message to obtain the first notification semantic features of the notification message.

[0124] The second feature extraction module 702 is used to extract the element attribute information of the interface elements in the structured graphical user interface to obtain the second notification semantic feature of the notification message when the notification message contains a structured graphical user interface.

[0125] The text generation module 703 is used to generate message prompt text corresponding to the notification message based on the first notification semantic features and the second notification semantic features;

[0126] The voice broadcast module 704 is used to broadcast the message prompt text via voice.

[0127] Optionally, the text generation module 703 includes:

[0128] The feature fusion unit is used to fuse the first notification semantic features and the second notification semantic features to obtain the semantic fusion features of the notification message.

[0129] The first text generation unit is used to input the semantic fusion features into the text generation model to obtain the descriptive text corresponding to the notification message;

[0130] The second text generation unit is used to generate the message prompt text corresponding to the notification message based on the description text.

[0131] Optionally, the second book generation unit is used for:

[0132] The description text is filled into the first message prompt template to obtain the message prompt text;

[0133] or,

[0134] Identify the interactive elements in the structured graphical user interface; extract the element description text of the interactive elements from the element attribute information of the interactive elements; and concatenate the description text and the element description text using a second message prompt template to obtain the message prompt text.

[0135] Optionally, the first text generation unit is used for:

[0136] The semantic fusion features are input into the first text generation model to obtain the first descriptive text corresponding to the notification message. The first text generation model has text generation and text summarization functions.

[0137] The second text generation unit is used for:

[0138] Based on the first description text, the message prompt text corresponding to the notification message is generated.

[0139] Optionally, the first text generation unit is used for:

[0140] The semantic fusion features are input into the second text generation model to obtain the second descriptive text corresponding to the notification message. The second text generation model has text generation function.

[0141] Feature extraction is performed on the second description text and the message source to obtain the third notification semantic features of the notification message;

[0142] The third notification semantic features are input into the third text generation module to obtain the third descriptive text of the notification message. The third text generation model has a text summarization function.

[0143] The second text generation unit is used for:

[0144] Based on the third description text, the message prompt text corresponding to the notification message is generated.

[0145] Optionally, the second feature extraction module 702 is used for:

[0146] Obtain the view tree of the structured graphical user interface;

[0147] Extract the element attribute information of the interface element from the view tree;

[0148] Feature extraction is performed on the element attribute information to obtain the element semantic features of the interface element;

[0149] The semantic features of the elements are extracted and fused to obtain the second notification semantic features of the notification message.

[0150] Optionally, the second feature extraction module 702 is used for:

[0151] Based on the element attribute information, the visual interface elements in the structured graphical user interface are determined;

[0152] Feature extraction is performed on the element attribute information of the visual interface element to obtain the element semantic features of the visual interface element.

[0153] Optionally, the device further includes:

[0154] The prompt text generation module is used to generate the message prompt text when the notification message is an important message, the notification message has a voice prompt requirement, and the notification message has a content extraction requirement.

[0155] Optionally, the device further includes:

[0156] The message classification module is used to classify the notification message based on the message source and the message content, and obtain the classification result. The classification result is used to indicate whether the notification message has a content extraction requirement.

[0157] Optionally, the device further includes:

[0158] The instruction text generation module is used to generate the instruction text corresponding to the received voice instruction.

[0159] A matching text determination module is used to determine the matching text in the structured graphical user interface that matches the instruction;

[0160] The instruction execution module is used to execute the operation indicated by the voice instruction based on the position of the element to which the matched text belongs.

[0161] Optionally, the instruction execution module is used for:

[0162] If the element to which the matched text belongs is an interactive element, the operation indicated by the voice command is executed based on the position of the element to which the matched text belongs;

[0163] The device further includes:

[0164] The interactive prompt module is used to provide interactive prompts when the element to which the matched text belongs is not an interactive element.

[0165] Please refer to Figure 8 This diagram illustrates a structural block diagram of a terminal provided in an exemplary embodiment of this application. The terminal 800 can be implemented as the terminal in the various embodiments described above. The terminal 800 may include one or more components such as a processor 810 and a memory 820.

[0166] The processor 810 may include one or more processing cores. The processor 810 connects to various parts within the terminal 800 using various interfaces and lines, and performs various functions and processes data of the terminal 800 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 820, and by calling data stored in the memory 820. Optionally, the processor 810 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 810 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), and modem. Specifically, the CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; the NPU is used to implement Artificial Intelligence (AI) functions; and the modem is used to handle wireless communication. It is understandable that the aforementioned modem may not be integrated into the processor 810, but may be implemented using a separate chip.

[0167] The memory 820 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 820 may include a non-transitory computer-readable storage medium. The memory 820 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc.; the data storage area may store data created according to the use of the terminal 800 (such as audio data, phone book, etc.).

[0168] In addition, those skilled in the art will understand that the structure of the terminal 800 shown in the above figures does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the terminal 800 may also include a display screen, camera assembly, microphone, speaker, radio frequency circuit, input unit, sensors (such as accelerometer, angular velocity sensor, light sensor, etc.), audio circuit, WiFi module, power supply, Bluetooth module, etc., which will not be described in detail here.

[0169] This application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the voice broadcasting method for notification messages as described in the above embodiments.

[0170] This application provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the voice broadcasting method for notification messages provided in various optional implementations of the above aspects.

[0171] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0172] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for voice broadcasting notification messages, characterized in that, The method includes: Feature extraction is performed on the message content and message source of the notification message to obtain the first notification semantic feature of the notification message; When the notification message contains a structured graphical user interface, feature extraction is performed on the element attribute information of the interface elements in the structured graphical user interface to obtain the second notification semantic feature of the notification message. When the notification message is an important message, and the notification message has a voice prompt requirement and a content extraction requirement, a message prompt text corresponding to the notification message is generated based on the first notification semantic feature and the second notification semantic feature; The message prompt text is read aloud via voice.

2. The method according to claim 1, characterized in that, The step of generating the message prompt text corresponding to the notification message based on the first notification semantic features and the second notification semantic features includes: The semantic features of the first notification and the semantic features of the second notification are fused to obtain the semantic fusion features of the notification message; The semantic fusion features are input into the text generation model to obtain the descriptive text corresponding to the notification message; Based on the description text, the message prompt text corresponding to the notification message is generated.

3. The method according to claim 2, characterized in that, The step of generating the message prompt text corresponding to the notification message based on the description text includes: The description text is filled into the first message prompt template to obtain the message prompt text; or, Identify the interactive elements in the structured graphical user interface; extract the element description text of the interactive elements from the element attribute information of the interactive elements; and concatenate the description text and the element description text using a second message prompt template to obtain the message prompt text.

4. The method according to claim 2, characterized in that, The step of inputting the semantic fusion features into the text generation model to obtain the descriptive text corresponding to the notification message includes: The semantic fusion features are input into the first text generation model to obtain the first descriptive text corresponding to the notification message. The first text generation model has text generation and text summarization functions. The step of generating the message prompt text corresponding to the notification message based on the description text includes: Based on the first description text, the message prompt text corresponding to the notification message is generated.

5. The method according to claim 2, characterized in that, The step of inputting the semantic fusion features into the text generation model to obtain the descriptive text corresponding to the notification message includes: The semantic fusion features are input into the second text generation model to obtain the second descriptive text corresponding to the notification message. The second text generation model has text generation function. Feature extraction is performed on the second description text and the message source to obtain the third notification semantic features of the notification message; The third notification semantic features are input into the third text generation model to obtain the third descriptive text of the notification message. The third text generation model has a text summarization function. The step of generating the message prompt text corresponding to the notification message based on the description text includes: Based on the third description text, the message prompt text corresponding to the notification message is generated.

6. The method according to claim 1, characterized in that, When the notification message contains a structured graphical user interface, feature extraction is performed on the element attribute information of the interface elements in the structured graphical user interface to obtain the second notification semantic feature of the notification message, including: Obtain the view tree of the structured graphical user interface; Extract the element attribute information of the interface element from the view tree; Feature extraction is performed on the element attribute information to obtain the element semantic features of the interface element; The semantic features of the elements are extracted and fused to obtain the second notification semantic features of the notification message.

7. The method according to claim 6, characterized in that, The step of extracting features from the element attribute information to obtain the element semantic features of the interface elements includes: Based on the element attribute information, the visual interface elements in the structured graphical user interface are determined; Feature extraction is performed on the element attribute information of the visual interface element to obtain the element semantic features of the visual interface element.

8. The method according to claim 1, characterized in that, The method further includes: Based on the message source and the message content, the notification message is classified to obtain a classification result, which is used to indicate whether the notification message has a content extraction requirement.

9. The method according to claim 1, characterized in that, The method further includes: Upon receiving a voice command, generate the command text corresponding to the voice command; Determine the matching text in the structured graphical user interface that matches the instruction; Based on the position of the element to which the matched text belongs, the operation indicated by the voice command is executed.

10. The method according to claim 9, characterized in that, The operation of executing the voice command based on the position of the element to which the matched text belongs includes: If the element to which the matched text belongs is an interactive element, the operation indicated by the voice command is executed based on the position of the element to which the matched text belongs; The method further includes: If the element to which the matched text belongs is not an interactive element, an interactive prompt will be provided.

11. A voice broadcasting device for notification messages, characterized in that, The device includes: The first feature extraction module is used to extract features from the message content and message source of the notification message to obtain the first notification semantic features of the notification message. The second feature extraction module is used to extract the element attribute information of the interface elements in the structured graphical user interface when the notification message contains a structured graphical user interface, so as to obtain the second notification semantic feature of the notification message. The text generation module is used to generate message prompt text corresponding to the notification message based on the first notification semantic features and the second notification semantic features when the notification message is an important message, the notification message has a voice prompt requirement, and the notification message has a content extraction requirement. The voice broadcast module is used to broadcast the message prompt text via voice.

12. A terminal, characterized in that, The terminal includes a processor and a memory; the memory stores at least one instruction, which is executed by the processor to implement the voice broadcasting method for notification messages as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the voice broadcasting method for notification messages as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the voice broadcasting method for notification messages as described in any one of claims 1 to 10.

Images