Information display method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an information display method and device, electronic equipment and a readable storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining voice audio data from audio data collected by a microphone; under the condition that the voice audio data comprises the wake-up keyword, displaying first information; wherein the first information comprises at least one of the following items: speaking content description information; speaker identity information; environment sound event information; second information; the speaking content description information is used for indicating speaking content corresponding to the human voice audio data or summary information of the speaking content; the speaker identity information is used for indicating the identity of a speaker corresponding to the voice audio data; the environment sound event information is used for indicating a sound event occurring in an environment in which the human voice audio data is collected; the second information is used for prompting the speaker to speak.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to an information display method, device, electronic device, and readable storage medium. Background Technology

[0002] Currently, electronic devices can assist users in perceiving surrounding voice events through voice wake-up functions.

[0003] In related technologies, electronic devices alert users when they detect a preset wake word. For example, when an electronic device detects someone calling a user's name, it displays "Someone is calling you" on the screen or vibrates to help the user perceive the call in their surroundings. However, the existing methods provide limited prompts, making it difficult for users to make accurate judgments based on these prompts. Summary of the Invention

[0004] The purpose of this application is to provide an information display method, device, electronic device, and readable storage medium that enables users to more accurately understand the complete content of voice events and promptly identify the caller's intent and potential environmental risks.

[0005] In a first aspect, embodiments of this application provide an information display method, the method comprising: acquiring human voice audio data from audio data collected by a microphone; and displaying first information when the human voice audio data includes a wake-up keyword; wherein the first information includes at least one of the following: speech content description information; speaker identity information; environmental sound event information; and second information; the speech content description information is used to indicate the speech content or an overview of the speech content corresponding to the human voice audio data; the speaker identity information is used to indicate the identity of the speaker corresponding to the human voice audio data; the environmental sound event information is used to indicate sound events occurring in the environment in which the human voice audio data was collected; and the second information is used to prompt the speaker to speak.

[0006] In this embodiment, the information display device acquires human voice audio data from audio data collected by a microphone; when the human voice audio data includes a wake-up keyword, it displays first information; wherein the first information includes at least one of the following: speech content description information; speaker identity information; environmental sound event information; and second information. The speech content description information indicates the speech content or a summary of the speech content corresponding to the human voice audio data; the speaker identity information indicates the identity of the speaker corresponding to the human voice audio data; the environmental sound event information indicates the sound events occurring in the environment where the human voice audio data was collected; and the second information prompts the speaker to speak. Through this solution, the information display device can display multi-level information such as speech content, speaker identity, and environmental sound events based on a single wake-up event, thereby constructing a more complete and three-dimensional acoustic scene perception for the user, helping the user to more accurately understand the complete content of the voice event, promptly identify the caller's intention and potential environmental risks, and thus improve the information display effect.

[0007] In some embodiments of this application, the method further includes, before displaying the first information:

[0008] Receive the first input to the wake word recognition function control;

[0009] In response to the first input, the wake word recognition function is enabled, and the audio data collected by the microphone is monitored to see if it includes the wake word.

[0010] When the human voice audio data includes wake-up keywords, display the first piece of information, including:

[0011] If human voice audio data including wake-up keywords is detected, display secondary information and a description of the spoken content;

[0012] The first information includes the second information and the description of the spoken content.

[0013] In this embodiment, when the wake-word recognition function is enabled and the audio data collected by the microphone is monitored to see if it includes a wake-word keyword, the information display device displays second information and a description of the spoken content when it detects that the human voice audio data includes a wake-word keyword. Thus, the information display device can flexibly provide different levels of information feedback according to the user's actual needs and the scenario. For example, in scenarios where only basic reminders are needed, the user can quickly know that someone is calling; or, in scenarios requiring clear semantic understanding, the specific sentence content and intent of the call are displayed. This ensures the immediacy and accuracy of the prompts, enabling users, especially hearing-impaired individuals, to promptly perceive the event of being called. Furthermore, by recognizing and converting the speech content, the originally singular acoustic event is transformed into text information with clear semantics, reducing the user's comprehension burden and improving the information efficiency and practicality of human-computer interaction. This achieves reliable and valuable voice event perception and prompting in complex acoustic environments.

[0014] In some embodiments of this application, the method further includes, before displaying the first information:

[0015] Receive a second input to the wake word recognition control and the voiceprint recognition control;

[0016] In response to the second input, the wake-up word recognition function and voiceprint recognition function are enabled, the audio data collected by the microphone is monitored to see if the wake-up keyword is included, and the voiceprint information is monitored.

[0017] When the human voice audio data includes wake-up keywords, display the first piece of information, including:

[0018] If the wake word recognition and voiceprint recognition functions are enabled, and the human voice audio data includes the wake keyword, the first message will be displayed.

[0019] The first piece of information is any one of the following:

[0020] Second information and speaker's identity information;

[0021] Secondary information, description of the content of the speech, and speaker's identity information;

[0022] Information describing the content of the speech and information about the speaker's identity.

[0023] In this embodiment, when the wake-up word recognition and voiceprint recognition functions are enabled, and the audio data collected by the microphone is monitored to see if it includes a wake-up keyword, as well as voiceprint information, the information display device displays second information and speaker identity information, or displays second information, speech content description information, and speaker identity information, or displays speech content description information and speaker identity information, depending on the combination of functions enabled by the user. In this way, the information display device can provide composite information prompts ranging from basic event reminders to those integrating speaker identity and speech content, based on the user's enabled function combinations. On the one hand, this ensures the immediacy and accuracy of the prompts, enabling users to promptly perceive the called event; on the other hand, by integrating voiceprint recognition and speech content recognition, a single reminder is transformed into composite information containing speech information and speaker identity information, allowing users to accurately determine the caller's identity and intent, and make a more appropriate and timely response.

[0024] In some embodiments of this application, when the human voice audio data includes a wake-up keyword, the method further includes, before displaying the first information:

[0025] Receive third input to the wake word recognition control and the ambient sound event recognition control;

[0026] In response to a third input, the wake-up word recognition function and the environmental sound event recognition function are enabled. The system monitors whether the audio data collected by the microphone includes the wake-up keyword and monitors environmental sound events.

[0027] When the human voice audio data includes wake-up keywords, display the first piece of information, including:

[0028] If the wake word recognition function and environmental event recognition function are enabled, and the human voice audio data includes the wake keyword, the first information is displayed;

[0029] The first piece of information is any one of the following:

[0030] Secondary information and environmental sound event information;

[0031] Secondary information includes information describing the content of speech and information about environmental sound events;

[0032] Information describing the spoken content and information about environmental sound events.

[0033] In this embodiment, when the wake-up word recognition and environmental event recognition functions are enabled, and the audio data collected by the microphone is monitored to see if it includes a wake-up keyword, and environmental sound events are monitored, the information display device displays second information and environmental sound event information, or displays second information, speech content description information, and environmental sound event information, or displays speech content description information and environmental sound event information, depending on the combination of functions enabled by the user. In this way, the information display device can provide composite information prompts ranging from basic event reminders to those integrating speaker identity and environmental sound events, based on the user's enabled function combinations. On the one hand, this ensures the immediacy and accuracy of the prompts, enabling users to promptly perceive the called event; on the other hand, through environmental sound event monitoring and speech content recognition, a single reminder is transformed into composite information containing speech information and environmental sound event information, allowing users to accurately judge the surrounding environmental sound events and the speaker's intention, and make a more appropriate and timely response.

[0034] In some embodiments of this application, when the human voice audio data includes wake-up keywords, displaying first information includes:

[0035] When the human voice audio data includes a wake-up keyword, a first audio segment and a second audio segment are obtained; the first audio segment is the audio data collected in the first time period before the wake-up timestamp, and the second audio segment is the audio data collected in the second time period after the wake-up timestamp.

[0036] The first audio segment and the second audio segment are spliced ​​together to obtain the spliced ​​audio segment;

[0037] Based on the spliced ​​audio segments, display the first information;

[0038] The first time period and the second time period satisfy any one of the following:

[0039] The first and second time periods are pre-set time periods;

[0040] The first duration of the first time period is the third duration corresponding to the first event type, and the second duration of the second time period is the fourth duration corresponding to the first event type;

[0041] The wake-up timestamp is determined based on any of the following:

[0042] The end time of the wake-up keyword in the audio data;

[0043] The start time of the wake-up keyword in the audio data;

[0044] The moment when the wake-up keyword is detected;

[0045] The first event type is the event type corresponding to human voice audio data.

[0046] In this embodiment, when the wake-word recognition, voiceprint recognition, and environmental event recognition functions are enabled, the information display device displays a combination of multiple information when it detects human voice audio data including wake-up keywords. Thus, the information display device can provide composite information prompts, ranging from basic event reminders to those integrating speech content descriptions, speaker identity, and environmental sound events, based on the user-activated function combinations. On the one hand, this ensures the immediacy and accuracy of the prompts, enabling users to promptly perceive the called event; on the other hand, through environmental sound event monitoring and speech content recognition, a single reminder is transformed into composite information containing speech information and environmental sound event information, allowing users to accurately judge surrounding environmental sound events and the speaker's intentions, and make more appropriate and timely responses.

[0047] In some embodiments of this application, when both the wake word recognition function and the environmental sound event recognition function are enabled, the fifth duration of the first time period is longer than the sixth duration, and the seventh duration of the second time period is longer than the eighth duration; wherein, the sixth duration is the duration of the first time period when both the wake word recognition function and the voiceprint recognition function are enabled; and the eighth duration is the duration of the second time period when both the wake word recognition function and the voiceprint recognition function are enabled.

[0048] With both wake word recognition and voiceprint recognition functions enabled, the ninth duration of the first time period is longer than the tenth duration, and the eleventh duration of the second time period is longer than the twelfth duration. The tenth duration is the duration of the first time period with only wake word recognition function enabled, and the twelfth duration is the duration of the second time period with only wake word recognition function enabled.

[0049] When the event type corresponding to the human voice audio data is an environmental sound event, the thirteenth duration of the first time period is longer than the fourteenth duration, the fifteenth duration of the second time period is longer than the sixteenth duration, the fourteenth duration is the duration of the first time period when the event type corresponding to the human voice audio data is a speaking event, and the sixteenth duration is the duration of the second time period when the event type corresponding to the human voice audio data is a speaking event.

[0050] In this embodiment, by splicing the first audio segment and the second audio segment, a temporally continuous audio segment centered on the wake word is obtained, providing a more complete audio context for scene analysis. This avoids making a single reminder based solely on the wake keyword, ensuring that the most complete description of the spoken content can be obtained subsequently for context recognition, voiceprint recognition, ambient sound recognition, and other processing, thereby obtaining rich and accurate first information, and thus providing users with richer prompt information.

[0051] In some embodiments of this application, after obtaining human voice audio data from the collected audio data, the method further includes:

[0052] Human voice audio data is input into a wake-up model. The wake-up model then performs wake-up word recognition processing on the human voice audio data to obtain a recognition result. The recognition result is either a first indication or a second indication. The first indication indicates that a wake-up keyword has been recognized. The second indication indicates that a wake-up keyword has not been recognized.

[0053] If the identification result is the first indication information, determine the wake-up timestamp.

[0054] In some embodiments of this application, the wake-up timestamp includes at least one of the following:

[0055] The end time corresponding to the wake-up keyword on the timeline of the audio data;

[0056] The start time of the wake-up keyword on the timeline of the audio data;

[0057] The system time at the time of the first indication is output by waking up the model.

[0058] In this embodiment, a dedicated wake-up keyword recognition model is used. Compared with simple endpoint detection or template matching, it can more accurately detect wake-up keywords in continuous speech and resist some noise interference, thereby improving the accuracy and reliability of wake-up keyword recognition and timestamp localization. Furthermore, accurate timestamp localization provides a reliable basis for accurate segmentation of preceding and following audio segments, ensuring that complete speech segments, including wake-up keywords, can be captured, thus providing data assurance for generating high-quality first information.

[0059] In some embodiments of this application, the wake-up model includes a feature extraction module, an acoustic prediction module, and a decoding module;

[0060] By using a wake-up model, wake-up word recognition is performed on human voice audio data to obtain recognition results, including:

[0061] The acoustic feature sequence is obtained by extracting acoustic features from human voice audio data through the feature extraction module.

[0062] The acoustic prediction module predicts the acoustic feature sequence to obtain the acoustic prediction matrix, which includes the probability distribution prediction information of the pronunciation sequence of human voice audio data.

[0063] The acoustic prediction information is decoded by the decoding module to obtain at least one decoding path sequence.

[0064] The decoding module performs pronunciation sequence matching between at least one decoding path sequence and a reference pronunciation sequence to obtain a matching result, and outputs the recognition result based on the matching result.

[0065] The reference pronunciation sequence is the pronunciation sequence of the wake-up keyword.

[0066] In this embodiment, acoustic features are extracted and decoded from human voice audio data using a wake-up model. Since the wake-up model can learn more stable speech feature representations in noisy environments, it can more accurately and reliably identify wake-up keywords in noisy environments. Based on the matching method of the decoding path, it can handle slight variations in the user's pronunciation, such as accent and speech rate, thereby improving the generalization of wake-up keyword recognition.

[0067] In some embodiments of this application, recognition results are generated based on the matching results, including:

[0068] If at least one decoding path sequence exists that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is greater than the confidence threshold, the wake-up model outputs a first recognition result; the first recognition result is the first indication information.

[0069] Alternatively, if there exists a second decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is less than the confidence threshold, the wake-up model outputs a second recognition result, which is a second indication information.

[0070] Alternatively, if there is no third decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, the wake-up model outputs a third recognition result, which includes the second indication information.

[0071] In this embodiment of the application, by calculating the confidence level of the target decoding path sequence that matches the wake word pronunciation sequence and determining whether the confidence level is higher than the confidence level threshold, audio segments with similar pronunciations but not real wake words, such as TV voice and conversations of others, can be effectively filtered out, thereby improving the accuracy of recognition.

[0072] In some embodiments of this application, the prompting information includes descriptive information about the spoken content;

[0073] Based on the concatenated audio segments, the first piece of information is displayed, including:

[0074] The concatenated audio segment is input into the speech recognition model, which then performs speech-to-text processing on the concatenated audio segment to generate a continuous text sequence including wake-up keywords.

[0075] Using a speech recognition model, descriptive information about spoken content is generated based on a continuous text sequence;

[0076] Displays a description of the spoken content.

[0077] In this embodiment of the application, by analyzing the continuous speech including the wake word and subsequent instructions as a whole, it is possible to accurately obtain the complete speech content description information before and after the wake word in the audio segment, thereby deeply understanding the user's true intention and interaction scenario, avoiding semantic fragmentation caused by segmented processing, and ensuring a complete understanding of the user's instructions.

[0078] In some embodiments of this application, the prompt information includes the speaker's identity information corresponding to the human voice audio data;

[0079] Based on the concatenated audio segments, the first piece of information is displayed, including:

[0080] The spliced ​​audio segments are input into the voiceprint recognition model. The voiceprint recognition model extracts voiceprint features from the spliced ​​audio segments to obtain voiceprint feature information.

[0081] By using a voiceprint recognition model, the similarity between voiceprint feature information and at least one preset voiceprint feature information is calculated to obtain similarity information.

[0082] The speaker identity information corresponding to the first voiceprint feature information is determined to be the speaker identity information corresponding to the human voice audio data.

[0083] Display the speaker's identity information;

[0084] The first voiceprint feature information is a voiceprint feature information that has a similarity greater than a first similarity threshold among at least one preset voiceprint feature information.

[0085] In this embodiment, voiceprint recognition is performed by splicing audio segments to extract discriminative voiceprint features. These features are then input into a voiceprint recognition model for processing to identify the speaker's identity information. This enables high-precision identity verification even in noisy environments or short speech scenarios, thereby improving interaction efficiency in complex scenarios.

[0086] In some embodiments of this application, the first information includes environmental sound event information; displaying the first information based on spliced ​​audio segments includes:

[0087] The spliced ​​audio segments are input into the dry sound separation model, and the ambient sound data is separated from the spliced ​​audio segments by the dry sound separation model.

[0088] Ambient sound data is input into an acoustic scene classification model. The acoustic scene classification model is used to classify and identify the ambient sound data to obtain ambient sound category information.

[0089] Based on environmental sound category information, an acoustic scene classification model is used to generate environmental sound event information.

[0090] Displays ambient sound event information.

[0091] In this embodiment of the application, by performing environmental sound recognition and processing on spliced ​​audio segments to obtain environmental sound data, and then performing category recognition on the separated environmental sound data, a two-level processing flow is used to accurately detect and identify potential dangers or important environmental events, such as boiling water, fire alarms, and intrusion alarms, from complex audio mixed with human voices, thus providing users with the most comprehensive and effective scene perception.

[0092] In some embodiments of this application, the method further includes:

[0093] Display the wake word settings interface;

[0094] Receive wake word setting input from the wake word settings interface;

[0095] In response to the wake word setting input, the wake word keyword set in the wake word setting interface is displayed.

[0096] In this embodiment of the application, by displaying at least one alternative pronunciation information of the wake-up keyword for the user to choose from, the user can conveniently select the pronunciation to be used from the alternative pronunciation information. In this way, the user can select the pronunciation of polyphonic characters, rare characters or words with local accents, effectively avoiding the problem of missed wake-up and false wake-up caused by uncertain pronunciation, thereby improving the accuracy and reliability of subsequent wake-up keyword recognition.

[0097] In some embodiments of this application, after receiving wake word setting input in the wake word setting interface, the method further includes:

[0098] When the wake-up keyword includes a polyphonic character, at least one alternative pronunciation option for the polyphonic character in the wake-up keyword will be displayed in the wake-up word setting interface, based on the text content of the wake-up keyword.

[0099] Receive input selecting a target pronunciation option from at least one alternative pronunciation option;

[0100] In response to the input selection, the pronunciation corresponding to the target alternative pronunciation option is determined as the pronunciation of the polyphonic character.

[0101] In some embodiments of this application, the method further includes:

[0102] The wake word management interface is displayed. The wake word management interface includes at least one wake keyword and at least one voiceprint addition option. One wake keyword corresponds to one voiceprint addition option.

[0103] Receive the first input for adding options to the voiceprint corresponding to the target wake-up keyword in at least one wake-up keyword;

[0104] In response to the first input, the voiceprint object settings interface corresponding to the target wake-up keyword is displayed;

[0105] Receive voiceprint object information settings input in the voiceprint object settings interface;

[0106] In response to the input of voiceprint object information settings, the voiceprint object information set in the voiceprint object settings interface is displayed; the voiceprint object information is used to obtain the speaker's identity information when the wake-up keyword is recognized.

[0107] Acquire the wake-up word speech data corresponding to the target wake-up keyword;

[0108] Based on wake-up word speech data, voiceprint registration is performed on the target wake-up keyword, and the association between the voiceprint information of the target wake-up keyword and the voiceprint object information is established.

[0109] In this embodiment, by registering voiceprints for wake-up keywords and strongly associating voiceprints with specific wake-up words and user identities, it is more effective than registering voiceprints for general voice commands in extracting unique acoustic features related to the pronunciation of wake-up words. This helps improve the accuracy and anti-interference capability of voiceprint recognition, thereby enabling the quick and accurate location of the valid command issuer in multi-person, noisy, or emergency scenarios, and executing operations that match the identity and permissions, thus avoiding erroneous responses.

[0110] In some embodiments of this application, the voice data includes N wake word voice samples;

[0111] Obtain the wake-up word speech data corresponding to the target wake-up keyword, including:

[0112] The wake-up word voiceprint recording interface is displayed. The wake-up word voiceprint recording interface includes a first recording prompt message and a first voice input control. The first recording prompt message is used to prompt the user to record a wake-up word voice sample and the number of times the wake-up word voice sample is recorded.

[0113] Receive voice input to the first voice input control;

[0114] In response to voice input, N wake-word voice samples are collected, where N is an integer greater than 1.

[0115] In this embodiment of the application, by recording N wake-up word speech samples, a sufficient and accurate data foundation is provided for subsequent voiceprint feature extraction, which helps to generate more robust and accurate voiceprint feature information, thereby improving the recognition rate of voiceprint objects of subsequent wake-up keywords.

[0116] In some embodiments of this application, voiceprint registration is performed on target wake-up keywords based on voice data, including:

[0117] Voiceprint features are extracted from N wake-up word speech samples to obtain N voiceprint feature information;

[0118] Calculate the similarity between N voiceprint feature information;

[0119] If the similarity between any two voiceprint features in N voiceprint features is greater than the second similarity threshold, then voiceprint registration is performed on the target wake-up keyword based on the N voiceprint features.

[0120] Alternatively, if the similarity between any two voiceprint features among the N voiceprint features is less than the second similarity threshold, obtain M personal voice samples, and perform voiceprint registration for the target wake-up keyword based on the M personal voice samples, where M is greater than N.

[0121] In this embodiment, by recording N wake-up word speech samples and performing consistency verification on the voiceprint feature information corresponding to the N wake-up word speech samples, voiceprint registration is performed if the verification passes, or if the verification fails, the user is prompted to record speech samples again for voiceprint registration. In this way, the reliability of voiceprint registration is significantly improved through the similarity verification mechanism.

[0122] Secondly, embodiments of this application provide an information display device, which includes a processing module and a display module, wherein: the processing module is used to acquire human voice audio data from audio data collected by a microphone; the display module is used to display first information when the human voice audio data includes a wake-up keyword; wherein the first information includes at least one of the following: speech content description information; speaker identity information; environmental sound event information; and second information; the speech content description information is used to indicate the speech content or a summary of the speech content corresponding to the human voice audio data; the speaker identity information is used to indicate the identity of the speaker corresponding to the human voice audio data; the environmental sound event information is used to indicate sound events occurring in the environment in which the human voice audio data was collected; and the second information is used to prompt the speaker to speak.

[0123] In some embodiments of this application, the receiving module is configured to receive a first input to the wake word recognition function control;

[0124] The processing module is also used to respond to the first input received by the receiving module, enable the wake-up word recognition function, and monitor whether the audio data collected by the microphone includes the wake-up keyword;

[0125] The display module is specifically used to display secondary information and speech content description information when human voice audio data, including wake-up keywords, is detected;

[0126] The first information includes the second information and the description of the spoken content.

[0127] In some embodiments of this application, the receiving module is configured to receive a second input to the wake-word recognition function control and the voiceprint recognition control;

[0128] The processing module is also used to respond to the second input received by the receiving module, enable the wake-up word recognition function and the voiceprint recognition function, monitor whether the audio data collected by the microphone includes the wake-up keyword, and monitor the voiceprint information;

[0129] The display module is specifically used to display the first information when the wake-up word recognition function and voiceprint recognition function are enabled and the human voice audio data includes the wake-up keyword.

[0130] The first piece of information is any one of the following:

[0131] Second information and speaker's identity information;

[0132] Secondary information, description of the content of the speech, and speaker's identity information;

[0133] Information describing the content of the speech and information about the speaker's identity.

[0134] In some embodiments of this application, the receiving module is configured to receive third input to the wake-word recognition function control and the environmental sound event recognition control;

[0135] The processing module is also used to respond to the third input received by the receiving module, enable the wake-up word recognition function and the environmental sound event recognition function, monitor whether the audio data collected by the microphone includes the wake-up keyword, and monitor environmental sound events;

[0136] The display module is specifically used to display the first information when the wake word recognition function and the environmental event recognition function are enabled, and when human voice audio data including wake words is detected.

[0137] The first piece of information is any one of the following:

[0138] Secondary information and environmental sound event information;

[0139] Secondary information includes information describing the content of speech and information about environmental sound events;

[0140] Information describing the spoken content and information about environmental sound events.

[0141] In some embodiments of this application, the processing module is configured to acquire a first audio segment and a second audio segment when the human voice audio data includes a wake-up keyword; the first audio segment is audio data collected in a first time period before the wake-up timestamp, and the second audio segment is audio data collected in a second time period after the wake-up timestamp;

[0142] The processing module is also used to splice the first audio segment and the second audio segment to obtain a spliced ​​audio segment;

[0143] The display module is specifically used to display the first information based on the spliced ​​audio segments;

[0144] The first time period and the second time period satisfy any one of the following:

[0145] The first and second time periods are pre-set time periods;

[0146] The first duration of the first time period is the third duration corresponding to the first event type, and the second duration of the second time period is the fourth duration corresponding to the first event type;

[0147] The wake-up timestamp is determined based on any of the following:

[0148] The end time of the wake-up keyword in the audio data;

[0149] The start time of the wake-up keyword in the audio data;

[0150] The moment when the wake-up keyword is detected;

[0151] The first event type is the event type corresponding to human voice audio data.

[0152] In some embodiments of this application, when both the wake word recognition function and the environmental sound event recognition function are enabled, the fifth duration of the first time period is longer than the sixth duration, and the seventh duration of the second time period is longer than the eighth duration; wherein, the sixth duration is the duration of the first time period when both the wake word recognition function and the voiceprint recognition function are enabled; and the eighth duration is the duration of the second time period when both the wake word recognition function and the voiceprint recognition function are enabled.

[0153] With both wake word recognition and voiceprint recognition functions enabled, the ninth duration of the first time period is longer than the tenth duration, and the eleventh duration of the second time period is longer than the twelfth duration. The tenth duration is the duration of the first time period with only wake word recognition function enabled, and the twelfth duration is the duration of the second time period with only wake word recognition function enabled.

[0154] When the event type corresponding to the human voice audio data is an environmental sound event, the thirteenth duration of the first time period is longer than the fourteenth duration, the fifteenth duration of the second time period is longer than the sixteenth duration, the fourteenth duration is the duration of the first time period when the event type corresponding to the human voice audio data is a speaking event, and the sixteenth duration is the duration of the second time period when the event type corresponding to the human voice audio data is a speaking event.

[0155] In some embodiments of this application, the processing module is used for:

[0156] Human voice audio data is input into a wake-up model. The wake-up model then performs wake-up word recognition processing on the human voice audio data to obtain a recognition result. The recognition result is either a first indication or a second indication. The first indication indicates that a wake-up keyword has been recognized. The second indication indicates that a wake-up keyword has not been recognized.

[0157] If the recognition result is the first indication information, determine the wake-up timestamp;

[0158] The wake-up timestamp includes at least one of the following:

[0159] The end time corresponding to the wake-up keyword on the timeline of the audio data;

[0160] The start time of the wake-up keyword on the timeline of the audio data;

[0161] The system time at the time of the first indication is output by waking up the model.

[0162] In some embodiments of this application, the wake-up model includes a feature extraction module, an acoustic prediction module, and a decoding module; the processing module is specifically used for:

[0163] The acoustic feature sequence is obtained by extracting acoustic features from human voice audio data through the feature extraction module.

[0164] The acoustic prediction module predicts the acoustic feature sequence to obtain the acoustic prediction matrix, which includes the probability distribution prediction information of the pronunciation sequence of human voice audio data.

[0165] The acoustic prediction information is decoded by the decoding module to obtain at least one decoding path sequence.

[0166] The decoding module performs pronunciation sequence matching between at least one decoding path sequence and a reference pronunciation sequence to obtain a matching result, and outputs the recognition result based on the matching result.

[0167] The reference pronunciation sequence is the pronunciation sequence of the wake-up keyword.

[0168] In some embodiments of this application, the processing module is specifically used for:

[0169] If at least one decoding path sequence exists that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is greater than the confidence threshold, the wake-up model outputs a first recognition result; the first recognition result is the first indication information.

[0170] Alternatively, if there exists a second decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is less than the confidence threshold, the wake-up model outputs a second recognition result, which is a second indication information.

[0171] Alternatively, if there is no third decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, the wake-up model outputs a third recognition result, which includes the second indication information.

[0172] In some embodiments of this application, the prompting information includes descriptive information about the spoken content;

[0173] The processing module is specifically used to input the spliced ​​audio segment into the speech recognition model, and through the speech recognition model, to perform speech-to-text processing on the spliced ​​audio segment to generate a continuous text sequence including wake-up keywords;

[0174] The processing module is specifically used to generate descriptive information of spoken content based on a continuous text sequence using a speech recognition model;

[0175] The display module is specifically used to display descriptive information about the spoken content.

[0176] In some embodiments of this application, the prompt information includes the speaker's identity information corresponding to the human voice audio data; the processing module is specifically used to input the spliced ​​audio segment into the voiceprint recognition model, and extract the voiceprint features from the spliced ​​audio segment through the voiceprint recognition model to obtain voiceprint feature information;

[0177] The processing module is specifically used to calculate the similarity between voiceprint feature information and at least one preset voiceprint feature information through a voiceprint recognition model to obtain similarity information.

[0178] The processing module is specifically used to determine the speaker identity information corresponding to the first voiceprint feature information as the speaker identity information corresponding to the human voice audio data.

[0179] The display module is specifically used to display the speaker's identity information;

[0180] The first voiceprint feature information is a voiceprint feature information that has a similarity greater than a first similarity threshold among at least one preset voiceprint feature information.

[0181] In some embodiments of this application, the first information includes environmental sound event information; the processing module is specifically used to input the spliced ​​audio segment into the dry sound separation model, and to separate the environmental sound data from the spliced ​​audio segment through the dry sound separation model;

[0182] The processing module is specifically used to input ambient sound data into the acoustic scene classification model, and to perform acoustic scene classification and recognition on the ambient sound data through the acoustic scene classification model to obtain ambient sound category information;

[0183] The processing module is specifically used to generate environmental sound event information based on environmental sound category information using an acoustic scene classification model;

[0184] The display module is specifically used to display environmental sound event information.

[0185] In some embodiments of this application, the receiving module is used to receive wake word setting input in the wake word setting interface;

[0186] The display module is also used to respond to the wake word setting input and display the wake keyword set in the wake word setting interface.

[0187] In some embodiments of this application, the display module is further configured to, in the case that the wake-up keyword includes a polyphonic character, display at least one alternative pronunciation option for the polyphonic character in the wake-up keyword in the wake-up word setting interface according to the text content of the wake-up keyword;

[0188] The receiving module is also configured to receive a selection input for a target alternative pronunciation option among at least one alternative pronunciation options;

[0189] The processing module is also used to determine the pronunciation corresponding to the target alternative pronunciation option as the pronunciation of the polyphonic character in response to the selection input.

[0190] In some embodiments of this application, the display module is further configured to display a wake-up word management interface, which includes at least one wake-up keyword and at least one voiceprint addition option, with one wake-up keyword corresponding to one voiceprint addition option;

[0191] The receiving module is also configured to receive a first input for adding options to the voiceprint corresponding to the target wake-up keyword in at least one wake-up keyword;

[0192] The display module is also used to respond to the first input and display the voiceprint object settings interface corresponding to the target wake-up keyword;

[0193] The receiving module is also used to receive voiceprint object information setting input in the voiceprint object setting interface;

[0194] The display module is also used to respond to the voiceprint object information setting input and display the voiceprint object information set in the voiceprint object setting interface; the voiceprint object information is used to obtain the speaker's identity information when the wake-up keyword is recognized.

[0195] The processing module is also used to acquire the wake-up word voice data corresponding to the target wake-up keyword;

[0196] The processing module is also used to perform voiceprint registration on the target wake-up keyword based on the wake-up word voice data, and to establish the association between the voiceprint information of the target wake-up keyword and the voiceprint object information.

[0197] In some embodiments of this application, the voice data includes N wake word voice samples;

[0198] The display module is also used to display the wake word voiceprint recording interface. The wake word voiceprint recording interface includes a first recording prompt message and a first voice input control. The first recording prompt message is used to prompt the user to record a wake word voice sample and the number of times the wake word voice sample is recorded.

[0199] The receiving module is also used to receive voice input to the first voice input control;

[0200] The processing module is specifically used to respond to voice input by collecting N wake-up word voice samples, where N is an integer greater than 1.

[0201] In some embodiments of this application, the processing module is specifically used for:

[0202] Voiceprint features are extracted from N wake-up word speech samples to obtain N voiceprint feature information;

[0203] Calculate the similarity between N voiceprint feature information;

[0204] If the similarity between any two voiceprint features in N voiceprint features is greater than the second similarity threshold, then voiceprint registration is performed on the target wake-up keyword based on the N voiceprint features.

[0205] Alternatively, if the similarity between any two voiceprint features among the N voiceprint features is less than the second similarity threshold, obtain M personal voice samples, and perform voiceprint registration for the target wake-up keyword based on the M personal voice samples, where M is greater than N.

[0206] It should be noted that the above-mentioned information display device can achieve the same effect as the above-mentioned information display method. Its effect can be found in the effect description of the above-mentioned information display method. To avoid repetition, it will not be described again here.

[0207] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0208] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0209] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0210] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0211] In this embodiment, the information display device acquires human voice audio data from audio data collected by a microphone; when the human voice audio data includes a wake-up keyword, it displays first information; wherein the first information includes at least one of the following: speech content description information; speaker identity information; environmental sound event information; and second information. The speech content description information indicates the speech content or a summary of the speech content corresponding to the human voice audio data; the speaker identity information indicates the identity of the speaker corresponding to the human voice audio data; the environmental sound event information indicates the sound events occurring in the environment where the human voice audio data was collected; and the second information prompts the speaker to speak. Through this solution, the information display device can display multi-level information such as speech content, speaker identity, and environmental sound events based on a single wake-up event, thereby constructing a more complete and three-dimensional acoustic scene perception for the user, helping the user to more accurately understand the complete content of the voice event, promptly identify the caller's intention and potential environmental risks, and thus improve the information display effect.

[0212] Understandably, the information display device provided in the second aspect, the electronic device provided in the third aspect, the readable storage medium provided in the fourth aspect, the chip provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0213] Figure 1 A flowchart illustrating an information display method provided for some embodiments of this application;

[0214] Figure 2 A schematic diagram of a settings interface provided for some embodiments of this application;

[0215] Figure 3A A schematic diagram of an information interface provided for some embodiments of this application;

[0216] Figure 3B A schematic diagram of an information interface provided for some embodiments of this application;

[0217] Figure 4 A schematic diagram of a settings interface provided for some embodiments of this application;

[0218] Figure 5A A schematic diagram of an information interface provided for some embodiments of this application;

[0219] Figure 5B A schematic diagram of an information interface provided for some embodiments of this application;

[0220] Figure 5C A schematic diagram of an information interface provided for some embodiments of this application;

[0221] Figure 6 A schematic diagram of an information interface provided for some embodiments of this application;

[0222] Figure 7A A schematic diagram of an information interface provided for some embodiments of this application;

[0223] Figure 7B A schematic diagram of an information interface provided for some embodiments of this application;

[0224] Figure 7C A schematic diagram of an information interface provided for some embodiments of this application;

[0225] Figure 8 A schematic diagram of a settings interface provided for some embodiments of this application;

[0226] Figure 9 A schematic diagram of an information interface provided for some embodiments of this application;

[0227] Figure 10A A schematic diagram of a wake-up word setting interface provided for some embodiments of this application;

[0228] Figure 10B A schematic diagram of a wake-up word setting interface provided for some embodiments of this application;

[0229] Figure 10C A schematic diagram of a wake-up word setting interface provided for some embodiments of this application;

[0230] Figure 11A A schematic diagram of a wake-up word management interface provided for some embodiments of this application;

[0231] Figure 11B A schematic diagram of a wake-up word management interface provided for some embodiments of this application;

[0232] Figure 11C A schematic diagram of a voiceprint object setting interface provided for some embodiments of this application;

[0233] Figure 12A A schematic diagram of a wake-word voiceprint recording interface provided for some embodiments of this application;

[0234] Figure 12B A schematic diagram of a wake-word voiceprint recording interface provided for some embodiments of this application;

[0235] Figure 13A A schematic diagram of a voiceprint recording interface provided for some embodiments of this application;

[0236] Figure 13B A schematic diagram of a voiceprint recording interface provided for some embodiments of this application;

[0237] Figure 14A A flowchart illustrating an information display method provided for some embodiments of this application;

[0238] Figure 14B A flowchart illustrating an information display method provided for some embodiments of this application;

[0239] Figure 15 Schematic diagrams of the structure of an information display device provided for some embodiments of this application;

[0240] Figure 16 Schematic diagrams of the structure of electronic devices provided for some embodiments of this application;

[0241] Figure 17 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of this application. Detailed Implementation

[0242] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0243] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0244] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included objects. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more, and its meaning is similar to that of "at least one."

[0245] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application is explained below.

[0246] Voiceprint: refers to the highly unique and stable set of acoustic features included in an individual's speech. These acoustic features are determined by the speaker's physiological structure and behavioral habits, and can be used to distinguish and identify different speakers.

[0247] Ambient sound events: These refer to acoustic events that occur in an audio scene and have a specific type or clear semantic meaning. Examples include the sound of boiling water, the beeping of a smoke alarm, the crying of a baby, and the ringing of a doorbell.

[0248] The information display method provided in this application can be executed by an information display device, which can be an electronic device, or a functional module or functional entity within an electronic device. The following description uses an information display device executing the information display method as an example to illustrate the information display method provided in this application.

[0249] The information display method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0250] The information display method provided in this application can be applied to scenarios where users are reminded of voice events. One specific scenario is, for example, in a daily life setting, when a family member calls out to user Xiaoming from another room, "Xiaoming, dinner's ready," Xiaoming not only needs to know that someone is calling him, but also needs to understand the speaker's intention and request in order to respond appropriately. Another specific scenario is when a user is in a group gathering; when someone calls the user, the user not only needs to know that someone is calling him, but also needs to know which of the multiple people is calling him, i.e., he needs to know the caller's identity information. Yet another specific scenario is, in a daily life setting, when someone is calling the user and there are ambient sounds in the background, such as the sound of boiling water or a smoke alarm, the user needs to know that someone is calling him and also needs to be aware of the boiling water or smoke alarm sounds in the environment so that he can respond to the call while being aware of potential safety hazards.

[0251] Figure 1 This is a flowchart illustrating the information display method provided in an embodiment of this application, such as... Figure 1 As shown, the information display method may include the following steps 201 and 202:

[0252] Step 201: The information display device acquires human voice audio data from the audio data collected by the microphone.

[0253] In some embodiments of this application, the electronic device can continuously or periodically collect audio data from the environment using an audio acquisition device. For example, the electronic device can continuously collect audio data from the environment using a microphone array.

[0254] In some embodiments of this application, the electronic device processes the audio data using Voice Activity Detection (VAD) technology to distinguish audio segments that include human voices, silent audio segments, or audio segments that only include ambient noise.

[0255] It should be noted that speech activity detection is a technology used in speech signal processing. Its core function is to automatically identify which parts of an audio signal contain human voices (speech) and which parts are silence or background noise. The working principle of speech activity detection is as follows: it extracts features from the audio signal that can distinguish between speech and noise, such as short-time energy, zero-crossing rate, and spectral characteristics. Then, it compares this information with one or more preset thresholds, and based on the comparison result, classifies the current frame as speech or non-speech.

[0256] It is understandable that the aforementioned human voice audio data refers to continuous audio frames or continuous audio data segments that are determined to contain human speech or voices through VAD technology.

[0257] In some embodiments of this application, when the information display device identifies that multiple consecutive audio frames contain human voices in the voice data, it stores the multiple consecutive audio frames in a voice buffer, and when it is subsequently determined that the duration of the audio frames cached in the voice buffer meets a preset duration or the number of frames meets a preset number of frames, it performs wake-word detection processing on the audio frames in the voice buffer.

[0258] In some examples, the information display device continuously collects audio data through a microphone and stores the collected audio data in an audio buffer. When the audio data in the audio buffer reaches a certain amount, human voice detection is performed. If human voice data is detected in multiple consecutive audio frames in the audio data, these multiple audio frames and some preceding audio frames are sent to a voice buffer. When the duration of the audio frames cached in the voice buffer reaches a preset duration, the cached audio frames are processed for recognition of wake-up keywords.

[0259] Specifically, the microphone starts to continuously pick up sound, and the audio stream is sent to the audio buffer. When the audio data stored in the audio buffer satisfies an audio block, the stored audio data is sent to the VAD module to predict the start and end frames of the speech. If the speech is present, it is judged as 1; if the speech is not present, it is judged as 0. When the speech prediction of 3 consecutive audio blocks is 1, audio data is continuously sent to the speech buffer from the first few frames of the speech to perform subsequent wake-up keyword detection processing.

[0260] It should be noted that an audio chunk refers to dividing the entire audio stream into small, continuous blocks, with each block containing audio data over a period of time.

[0261] The following provides an exemplary description of the information display method provided in the embodiments of this application, combined with specific scenarios.

[0262] For example, in a real-life scenario, Xiaoming's family members call out to him from another room, saying, "Xiaoming, dinner's ready." The information display device continuously collects audio data from the surrounding environment through a microphone array and performs real-time voice activity detection on the collected continuous audio stream. When the detected audio energy exceeds the environmental noise threshold and the spectral characteristics match a human voice pattern, it determines that valid speech exists and accurately extracts the human voice segment: "Xiaoming, dinner's ready." This allows Xiaoming to know that his family is calling him and what they said.

[0263] For example, in a group gathering scenario, when someone calls Xiaoming's name, Xiaoming not only needs to know that someone is calling him, but also needs to know which of the many people is calling him, that is, he needs to know the caller's identity information. The information display device continuously collects audio data from the surrounding environment through a microphone array and performs real-time voice activity detection on the collected continuous audio stream. When the detected audio energy exceeds the environmental noise threshold and the spectral characteristics match the human voice pattern, it determines that there is valid speech and accurately extracts the human voice segment: "Xiaoming". This allows Xiaoming to know that someone is calling him and the identity information of the person calling him.

[0264] For example, in a real-life scenario, when someone calls Xiaoming and there are ambient sounds in the background, such as the sound of boiling water or a smoke alarm, the user needs to know that someone is calling him and also needs to be aware of the boiling water or smoke alarm sounds in the environment so that he can respond to the call while being aware of potential safety hazards. The information display device continuously collects audio data from the surrounding environment through a microphone array and performs real-time voice activity detection on the collected continuous audio stream. When the detected audio energy exceeds the environmental noise threshold and the spectral characteristics match a human voice pattern, it determines that there is valid speech and accurately extracts the human voice segment: "Xiaoming, come here for a moment."

[0265] In some embodiments of this application, the electronic device can collect audio data and extract human voice segments from the collected audio data when the custom wake-up function is enabled, in order to detect wake-up keywords.

[0266] Understandably, the custom wake-up function allows users to define their own wake-up keywords. When the electronic device recognizes the wake-up keyword in the audio stream continuously captured by the microphone, it triggers the corresponding wake-up behavior to provide the user with prompts including descriptive information about the spoken content, speaker identification information, or environmental sound events, enabling the user to accurately perceive voice events in the environment.

[0267] In some embodiments of this application, the electronic device can enable the custom wake-up function based on the user's input of the on / off option of the custom wake-up function in the settings interface.

[0268] Step 202: The information display device displays the first information when the human voice audio data includes the wake-up keyword.

[0269] The first information mentioned above includes at least one of the following: information describing the content of the speech; information about the speaker's identity; information about environmental sound events; and second information.

[0270] In some embodiments of this application, the speech content description information is used to indicate the speech content or a summary of the speech content corresponding to the human voice audio data; the speaker identity information is used to indicate the identity of the speaker corresponding to the human voice audio data; the environmental sound event information is used to indicate the sound events that occurred in the environment in which the human voice audio data was collected; and the second information is used to prompt the speaker to speak.

[0271] In some embodiments of this application, the aforementioned wake-up keyword can be a user-defined word, such as the username "Xiaoming".

[0272] For ease of description, this wake-up keyword can also be called a wake-up word.

[0273] In some embodiments of this application, the information display device can process the human voice audio data based on a deep learning end-to-end wake-up model to determine whether the human voice audio data includes wake-up keywords.

[0274] In some embodiments of this application, the information display device can match the feature sequence of a human voice segment with the feature sequence of a pre-stored wake-up keyword, determine whether they match by calculating the minimum cumulative distance between them, and if they match, determine that the human voice audio data includes the wake-up keyword.

[0275] In some embodiments of this application, the information display device performs audio recognition on human voice audio data to obtain first information, so that the user can more comprehensively understand the context or environmental information when the speaker calls the user.

[0276] In some embodiments of this application, when the information display device recognizes the pronunciation of the wake-up keyword in the human voice audio data, it comprehensively analyzes the current speech content description information, speaker identity, and environmental sound events based on the human voice audio data, thereby providing the user with rich prompt information while notifying the user that someone is calling them.

[0277] In some embodiments of this application, the above-mentioned speech content description information includes at least one of the following: complete speech content corresponding to human voice audio data, speaker's dialogue intention information, and summary information of speech content.

[0278] In some embodiments of this application, the information display device can perform speech recognition and natural language understanding on the aforementioned human voice audio data to obtain information describing the spoken content.

[0279] In some embodiments of this application, the speaker identity information mentioned above includes at least one of the following: the speaker's name, gender, age, etc.

[0280] In some embodiments of this application, the information display device can perform voiceprint recognition on human voice audio data to determine the speaker's identity, gender, age, and other characteristics.

[0281] In some embodiments of this application, the aforementioned environmental sound events may include at least one of the following: quiet, smoke alarm, dog barking, baby crying, water boiling, etc.

[0282] In some embodiments of this application, the information display device can perform environmental sound event detection on human voice audio data to identify ambient sounds present in the background.

[0283] The information display method provided in this application embodiment involves an information display device acquiring human voice audio data from audio data collected by a microphone. If the human voice audio data includes a wake-up keyword, it displays first information. The first information includes at least one of the following: a description of the speech content; speaker identity information; environmental sound event information; and second information. The speech content description information indicates the speech content or a summary of the speech content corresponding to the human voice audio data. The speaker identity information indicates the identity of the speaker corresponding to the human voice audio data. The environmental sound event information indicates sound events occurring in the environment where the human voice audio data was collected. The second information prompts the speaker to speak. Through this solution, the information display device can display multi-level information such as speech content, speaker identity, and environmental sound events based on a single wake-up event, thereby constructing a more complete and three-dimensional acoustic scene perception for the user. This helps the user more accurately understand the complete content of the voice event and promptly identify the caller's intention and potential environmental risks. For example, it helps hearing-impaired individuals accurately understand what others are saying and the risks in the surrounding environment.

[0284] In some embodiments of this application, before displaying the first information in step 202 above, the information display method provided in this application may further include the following steps A1 and A2:

[0285] Step A1: The information display device receives the first input to the wake word recognition function control.

[0286] Step A2: In response to the first input, the information display device activates the wake-up word recognition function and monitors whether the audio data collected by the microphone includes the wake-up keyword.

[0287] For example, combining steps A1 and A2 above, step 202 above may include the following step 202a1:

[0288] Step 202a1: When the information display device detects human voice audio data including wake-up keywords, it displays second information and a description of the spoken content.

[0289] The first information includes the second information and the description of the spoken content.

[0290] In some embodiments of this application, the first input is used to trigger the activation of the wake-up word recognition function and to trigger the monitoring of whether the audio data collected by the microphone includes a wake-up keyword.

[0291] In some embodiments of this application, the first input may include any of the following: user touch input, voice input, gesture input, folding or unfolding operation of the foldable screen, or other feasible inputs, which are not limited in this application embodiment. Further, the touch input may be: user click input, swipe input, press input, etc. Further, the click operation may be any number of clicks. The swipe operation may be a swipe operation in any direction, such as swiping up, swiping down, swiping left, or swiping right, which are not limited in this application embodiment.

[0292] In some embodiments of this application, the wake word recognition function control can be a graphical user interface element such as a switch, button, or desktop shortcut icon, which the user operates to control the on or off state of the wake word recognition function.

[0293] For example, such as Figure 2 As shown, the information display device displays a settings interface 21, which includes a function selection window 22 and a wake-up word recognition control 23. Users can click the wake-up word recognition control 23 to enable the wake-up recognition function and trigger the start of the audio data monitoring process.

[0294] The following provides an exemplary description of the information display method provided in the embodiments of this application, combined with specific scenarios.

[0295] In some examples, in real-life scenarios, when a family member calls out to the user from another room and says, "Xiaoming, dinner's ready," with the wake-word recognition function enabled, the information display device identifies a voice segment through voice activity detection, and detects that the voice segment includes the wake-word "Xiaoming," subsequently displaying the first message. For example... Figure 3A As shown, the first message 32, "Please note, someone is calling Xiaoming," is displayed on the information interface 31, so that the user can know in time that someone is calling him.

[0296] In some examples, in everyday scenarios, when a family member says "Xiaoming, dinner's ready" from another room, with the wake-word recognition function enabled, the information display device first locates the voice segment through voice activity detection, then identifies the wake-word "Xiaoming," and then converts the voice segment into text through automatic speech recognition. Finally, the natural language understanding module performs in-depth analysis of the complete sentence "Xiaoming, dinner's ready," parsing the combined intent of "calling" and "eating," thereby obtaining a complete description of the spoken content. For example... Figure 3B As shown, based on the acquired speech content description information, the information display device displays the first information 34 "Someone is calling: Xiaoming, it's time to eat" on the information interface 33. This clearly includes specific instructions, so that the user not only knows that someone is calling him, but also can accurately understand the speaker's intention and requirements, and thus make an appropriate response.

[0297] In this embodiment, when the wake-word recognition function is enabled and the audio data collected by the microphone is monitored to see if it includes a wake-word keyword, the information display device displays second information and a description of the spoken content when it detects that the human voice audio data includes a wake-word keyword. Thus, the information display device can flexibly provide different levels of information feedback according to the user's actual needs and the scenario. For example, in scenarios where only basic reminders are needed, the user can quickly know that someone is calling; or, in scenarios requiring clear semantic understanding, the specific sentence content and intent of the call are displayed. This ensures the immediacy and accuracy of the prompts, enabling users, especially hearing-impaired individuals, to promptly perceive the event of being called. Furthermore, by recognizing and converting the speech content, the originally singular acoustic event is transformed into text information with clear semantics, reducing the user's comprehension burden and thus lowering the difficulty of understanding and decision-making. Through in-depth security warnings and identity verification, reliable and valuable speech event perception and prompts are achieved in complex acoustic environments.

[0298] In some embodiments of this application, before displaying the first information in step 202 above, the information display method provided in this application may further include the following steps B1 and B2:

[0299] Step B1: The information display device receives a second input to the wake word recognition function control and the voiceprint recognition control.

[0300] Step B2: In response to the second input, the information display device activates the wake-up word recognition function and the voiceprint recognition function, monitors whether the audio data collected by the microphone includes the wake-up keyword, and monitors the voiceprint information;

[0301] For example, combining steps B1 and B2 above, step 202 above may include the following step 202a2:

[0302] Step 202a2: When the wake-up word recognition function and voiceprint recognition function are enabled, and the human voice audio data includes the wake-up keyword, the information display device displays the first information.

[0303] The first piece of information mentioned above is any one of the following:

[0304] Second information and speaker's identity information;

[0305] Secondary information, description of the content of the speech, and speaker's identity information;

[0306] Information describing the content of the speech and information about the speaker's identity.

[0307] In some embodiments of this application, the second input is used to trigger the activation of the wake-up word recognition function and the voiceprint recognition function, and to trigger the monitoring of whether the audio data collected by the microphone includes the wake-up keyword, and to trigger the monitoring of voiceprint information.

[0308] In some embodiments of this application, after the wake word recognition function is enabled, a text prompt can be made to the user that someone is calling him after a pre-set wake word is recognized, and the content of the speaker's speech can also be prompted.

[0309] In some embodiments of this application, after enabling the voiceprint recognition function, upon recognizing a pre-set wake-up word, the voiceprint features in the corresponding audio are extracted and compared with pre-registered voiceprint feature information. When a match is successful, the speaker's identity is determined, and the speaker's identity identifier is included in the generated prompt message. This allows the user to clearly know the caller's specific identity while perceiving the call event, thereby achieving a more accurate and targeted response in multi-person environments or scenarios where speaker differentiation is required.

[0310] In some embodiments of this application, the second input may include any of the following: user touch input, voice input, gesture input, folding or unfolding operation of the foldable screen, or other feasible inputs, which are not limited in this application embodiment. Further, the touch input may be: user click input, swipe input, press input, etc. Further, the click operation may be any number of clicks. The swipe operation may be a swipe operation in any direction, such as swiping up, swiping down, swiping left, or swiping right, which are not limited in this application embodiment.

[0311] In some embodiments of this application, the aforementioned voiceprint recognition control may be a graphical user interface element such as a switch, button, or desktop shortcut icon, which the user operates to control the on or off state of the voiceprint recognition function.

[0312] In combination with the above Figure 2 ,like Figure 4 As shown, the electronic device displays a settings interface 21, which includes a function selection window 22. The function selection window 22 includes a wake-up recognition function control 23 and a voiceprint recognition control 24. Users can click on the voiceprint recognition control 24 to enable the voiceprint recognition function.

[0313] The following provides an exemplary description of the information display method provided in the embodiments of this application, combined with specific scenarios.

[0314] In some examples, when a user is in a group gathering, and someone calls out to the user, the information display device, with the wake-up word recognition and voiceprint recognition functions enabled, extracts the complete audio segment including the calling statement after completing the wake-up word recognition. The voiceprint recognition module then extracts the voiceprint features from the audio and calculates the similarity between the extracted features and the pre-registered voiceprint database, ultimately matching the identity information as "Teacher Wang".

[0315] In some examples, such as Figure 5A As shown, based on the voiceprint recognition results, the information display device displays the identity prompt information 36 "Teacher Wang is calling: Xiaoming" on the information interface 35, which indicates the speaker's identity information. This allows the user to not only know that someone is calling them, but also to accurately know the caller's specific identity information, making it easier to respond in a targeted manner.

[0316] In some examples, such as Figure 5B As shown, the information display device displays the identity prompt information 36a "Teacher Wang calls out: Xiaoming, please speak" on the information interface 35. This indicates the speaker's identity information, that someone is speaking, and the content of the speech. This allows the user to not only know that someone is calling them, but also to accurately know the caller's specific identity information and the speaker's specific intention, making it easier to respond in a targeted manner.

[0317] In some examples, such as Figure 5C As shown, the information display device displays the identity prompt information 36b "Teacher Wang: Xiaoming, please speak" on the information interface 35, which indicates the speaker's identity information and the content of the speech. This allows the user to not only know that someone is calling them, but also to accurately know the specific identity information of the caller and the speaker's specific intention, so as to make a targeted response.

[0318] In this embodiment, when the wake-up word recognition and voiceprint recognition functions are enabled, and the audio data collected by the microphone is monitored to see if it includes a wake-up keyword, as well as voiceprint information, the information display device displays second information and speaker identity information, or displays second information, speech content description information, and speaker identity information, or displays speech content description information and speaker identity information, depending on the combination of functions enabled by the user. In this way, the information display device can provide composite information prompts ranging from basic event reminders to those integrating speaker identity and speech content, based on the user's enabled function combinations. On the one hand, this ensures the immediacy and accuracy of the prompts, enabling users to promptly perceive the called event; on the other hand, by integrating voiceprint recognition and speech content recognition, a single reminder is transformed into composite information containing speech information and speaker identity information, allowing users to accurately determine the caller's identity and intent, and make a more appropriate and timely response.

[0319] In some embodiments of this application, before displaying the first information in step 202 above, the information display method provided in this application may further include the following steps C1 and C2:

[0320] Step C1: The information display device receives a third input to the wake word recognition function control and the ambient sound event recognition control.

[0321] Step C2: In response to the third input, the information display device activates the wake-up word recognition function and the environmental sound event recognition function, monitors whether the audio data collected by the microphone includes the wake-up keyword, and monitors environmental sound events;

[0322] For example, combining steps C1 and C2 above, step 202 above may include the following step 202a3:

[0323] Step 202a3: When the wake-up word recognition function and environmental event recognition function are enabled, and the human voice audio data includes the wake-up keyword, the information display device displays the first information;

[0324] The first piece of information mentioned above is any one of the following:

[0325] Secondary information and environmental sound event information;

[0326] Secondary information includes information describing the content of speech and information about environmental sound events;

[0327] Information describing the spoken content and information about environmental sound events.

[0328] In some embodiments of this application, the third input is used to trigger the activation of the wake-up word recognition function and the environmental sound event recognition function, and to trigger the monitoring of whether the audio data collected by the microphone includes the wake-up keyword, and to trigger the monitoring of environmental sound events.

[0329] In some embodiments of this application, after enabling the environmental event recognition function, when a wake word is detected, the ambient sound component can be separated from the audio containing the wake word, and the separated ambient sound data can be input into an acoustic scene classification model for recognition. Through comparative analysis, the category of a specific sound event occurring in the current environment is determined, and the corresponding ambient sound event information is output in the generated prompt message. In this way, while sensing a calling event, the user can simultaneously learn about important acoustic conditions in the surrounding environment, such as boiling water or smoke alarms, thus knowing that someone is calling them and being able to promptly recognize and respond to potential environmental risks.

[0330] In some embodiments of this application, the aforementioned third input may include any of the following: user touch input, voice input, gesture input, folding or unfolding operation of the foldable screen, or other feasible inputs, which are not limited in this application embodiment. Further, the aforementioned touch input may be: user click input, swipe input, press input, etc. Further, the aforementioned click operation may be any number of clicks. The aforementioned swipe operation may be a swipe operation in any direction, such as swiping up, swiping down, swiping left, or swiping right, etc., which are not limited in this application embodiment.

[0331] In some embodiments of this application, the aforementioned environmental sound event recognition control can be a graphical user interface element such as a switch, button, or desktop shortcut icon, which the user operates to control the on or off state of the environmental sound event recognition function.

[0332] In combination with the above Figure 2 ,like Figure 6 As shown, the electronic device displays a settings interface 21, which includes a function selection window 22. The function selection window 22 includes a wake word recognition function control 23 and an ambient sound event recognition control 24. The user can click the wake word recognition function control 23 to enable the wake word recognition function and click the ambient sound event recognition control 24 to enable the ambient sound event recognition function.

[0333] The following provides an exemplary description of the information display method provided in the embodiments of this application, combined with specific scenarios.

[0334] In some examples, in real-life scenarios, when someone calls out to the user and there is a sound of boiling water in the background, the information display device, with its wake-word recognition and environmental sound event recognition functions enabled, separates the environmental sound component from the audio stream while processing the voice content, and inputs the separated environmental sound into the acoustic scene classification model to accurately identify the specific environmental sound event of "boiling water sound".

[0335] In some examples, such as Figure 7A As shown, the information display device integrates the voice recognition results with environmental sound events and displays the first information 38 on the information interface 37: "Please note, someone is calling Xiaoming nearby" and "Please note, there may be boiling water nearby." This includes both a call reminder and an environmental safety warning, enabling users to be aware of potential safety hazards in the environment while responding to the call and to take appropriate measures.

[0336] In some examples, such as Figure 7B As shown, the information display device displays the first information 38a in the information interface 37: "Please note that someone is calling Xiaoming and saying that the water is boiling. Please note that there may be boiling water sounds around." This information includes the call reminder, the content of the speech, and environmental sound event information, so that the user can be aware of potential safety hazards in the environment while responding to the call and take appropriate measures.

[0337] In some examples, such as Figure 7C As shown, the information display device displays the first information 38b on the information interface 37: "Please note that someone said the water is boiling. Please note that there may be boiling sounds in the surrounding area." This information includes a call reminder, the content of the speech, and environmental sound event information, so that users can be aware of potential safety hazards in the environment while responding to the call and take appropriate measures.

[0338] In this embodiment, when the wake-up word recognition and environmental event recognition functions are enabled, and the audio data collected by the microphone is monitored to see if it includes a wake-up keyword, and environmental sound events are monitored, the information display device displays second information and environmental sound event information, or displays second information, speech content description information, and environmental sound event information, or displays speech content description information and environmental sound event information, depending on the combination of functions enabled by the user. In this way, the information display device can provide composite information prompts ranging from basic event reminders to those integrating speaker identity and environmental sound events, based on the user's enabled function combinations. On the one hand, this ensures the immediacy and accuracy of the prompts, enabling users to promptly perceive the called event; on the other hand, through environmental sound event monitoring and speech content recognition, a single reminder is transformed into composite information containing speech information and environmental sound event information, allowing users to accurately judge the surrounding environmental sound events and the speaker's intention, and make a more appropriate and timely response.

[0339] In some embodiments of this application, before displaying the first information in step 202 above, the information display method provided in this application may further include the following steps D1 and D2:

[0340] Step D1: The information display device receives the fourth input to the wake word recognition function control, the voiceprint recognition control, and the ambient sound event recognition control.

[0341] Step D2: In response to the fourth input, the information display device activates the wake-up word recognition function, voiceprint recognition function, and environmental sound event recognition function, monitors whether the audio data collected by the microphone includes the wake-up keyword, monitors voiceprint information, and monitors environmental sound events.

[0342] For example, combining steps D1 and D2 above, step 202 above may include the following step 202a4:

[0343] Step 202a4: When the wake-up word recognition function, voiceprint recognition function, and environmental event recognition function are enabled, and the human voice audio data includes the wake-up keyword, the information display device displays the first information;

[0344] The first piece of information mentioned above is any one of the following:

[0345] Secondary information, speaker identity information, and environmental sound event information;

[0346] Secondary information includes: description of the spoken content, speaker's identity information, and environmental sound event information;

[0347] Speaker identity information and environmental sound event information;

[0348] Speaker identity information, description of speech content, and environmental sound event information.

[0349] In some embodiments of this application, the fourth input is used to trigger the activation of the wake-up word recognition function, the voiceprint recognition function, and the environmental sound event recognition function, and to trigger the monitoring of whether the audio data collected by the microphone includes the wake-up keyword, the monitoring of voiceprint information, and the monitoring of environmental sound events.

[0350] In some embodiments of this application, the fourth input may include any of the following: user touch input, voice input, gesture input, folding or unfolding operation of the foldable screen, or other feasible inputs, which are not limited in this application embodiment. Further, the touch input may be: user click input, swipe input, press input, etc. Further, the click operation may be any number of clicks. The swipe operation may be a swipe operation in any direction, such as swiping up, swiping down, swiping left, or swiping right, which are not limited in this application embodiment.

[0351] In combination with the above Figure 2 ,like Figure 8 As shown, the electronic device displays a settings interface 21, which includes a function selection window 22. The function selection window 22 includes a wake word recognition function control 25, a voiceprint recognition control 26, and an ambient sound event recognition control 27. Users can click on the wake word recognition function control 25, the voiceprint recognition control 26, and the ambient sound event recognition control 27 to enable the wake word recognition function, the voiceprint recognition function, and the ambient sound event recognition function.

[0352] The following provides an exemplary description of the information display method provided in the embodiments of this application, combined with specific scenarios.

[0353] In some examples, in everyday scenarios, when someone calls out to a user and there is a sound of boiling water in the background, the information display device, while processing the voice content, performs voiceprint recognition on the audio data to obtain the speaker's identity information, and separates the ambient sound components from the audio stream. The separated ambient sound is then input into an acoustic scene classification model to accurately identify the specific environmental sound event of "boiling water sound." For example... Figure 9 As shown, the information display device integrates the voice recognition results with environmental sound events and displays the first information 40 on the information interface 39: "Please note that Xiao Li is calling Xiao Ming. Please note that there may be boiling water sounds around." This includes both a call reminder and an environmental safety warning, enabling users to be aware of potential safety hazards in the environment while responding to the call and to take appropriate measures.

[0354] In this embodiment, when the wake-word recognition, voiceprint recognition, and environmental event recognition functions are enabled, the information display device displays a combination of multiple information when it detects human voice audio data including wake-up keywords. Thus, the information display device can provide composite information prompts, ranging from basic event reminders to those integrating speech content descriptions, speaker identity, and environmental sound events, based on the user-activated function combinations. On the one hand, this ensures the immediacy and accuracy of the prompts, enabling users to promptly perceive the called event; on the other hand, through environmental sound event monitoring and speech content recognition, a single reminder is transformed into composite information containing speech information and environmental sound event information, allowing users to accurately judge surrounding environmental sound events and the speaker's intentions, and make more appropriate and timely responses.

[0355] In some embodiments of this application, step 202a may include steps 202b1 to 202b3:

[0356] Step 202b1: When the human voice audio data includes a wake-up keyword, the information display device acquires the first audio segment and the second audio segment.

[0357] The first audio segment is audio data collected in the first time period before the wake-up timestamp, and the second audio segment is audio data collected in the second time period after the wake-up timestamp.

[0358] Step 202b2: The information display device splices the first audio segment and the second audio segment to obtain a spliced ​​audio segment.

[0359] Step 202b3: The information display device displays the first information based on the spliced ​​audio segments.

[0360] In some embodiments of this application, the first time period and the second time period described above satisfy any one of the following:

[0361] The first and second time periods are pre-set time periods;

[0362] The first duration of the first time period is the third duration corresponding to the first event type, and the second duration of the second time period is the fourth duration corresponding to the first event type.

[0363] In some embodiments of this application, the wake-up timestamp is determined based on any of the following:

[0364] The end time of the wake-up keyword in the audio data;

[0365] The start time of the wake-up keyword in the audio data;

[0366] The moment when the wake-up keyword is detected.

[0367] The first event type is the event type corresponding to human voice audio data.

[0368] In some embodiments of this application, when the information display device determines that the human voice audio data includes a wake-up keyword, it determines the end time or start time of the wake-up keyword in the audio data as the wake-up timestamp, or determines the time when the wake-up keyword is identified as the wake-up timestamp.

[0369] In some embodiments of this application, the information display device acquires audio data within a first time period before the wake-up timestamp as a first audio segment, and audio data within a second time period after the timestamp as a second audio segment, based on the wake-up timestamp.

[0370] In some examples, when a wake-up keyword is included in a human voice segment, the information display device determines the end time point of the wake-up keyword in the audio data, and based on the end time point, retains 1.5 seconds of audio before and 3.5 seconds of audio after, for a total of 5 seconds of audio data. This 5-second audio data is saved as the wake-up success audio, used to obtain the speech content description information corresponding to the wake-up keyword, as well as the input data for subsequent verification of voiceprint information and ambient sound detection.

[0371] In some examples, if less than 3.5 seconds of audio data is retained, all data in the voice buffer is saved and 5 seconds of audio data is added back.

[0372] In some embodiments of this application, when both the wake word recognition function and the environmental sound event recognition function are enabled, the fifth duration of the first time period is longer than the sixth duration, and the seventh duration of the second time period is longer than the eighth duration; wherein, the sixth duration is the duration of the first time period when both the wake word recognition function and the voiceprint recognition function are enabled; and the eighth duration is the duration of the second time period when both the wake word recognition function and the voiceprint recognition function are enabled.

[0373] In some embodiments of this application, when both the wake word recognition function and the voiceprint recognition function are enabled, the ninth duration of the first time period is longer than the tenth duration, and the eleventh duration of the second time period is longer than the twelfth duration. The tenth duration is the duration of the first time period when only the wake word recognition function is enabled, and the twelfth duration is the duration of the second time period when only the wake word recognition function is enabled.

[0374] In some embodiments of this application, when the event type corresponding to the human voice audio data is an environmental sound event, the thirteenth duration of the first time period is longer than the fourteenth duration, the fifteenth duration of the second time period is longer than the sixteenth duration, the fourteenth duration is the duration of the first time period when the event type corresponding to the human voice audio data is a speaking event, and the sixteenth duration is the duration of the second time period when the event type corresponding to the human voice audio data is a speaking event.

[0375] In some examples, different combinations of functions correspond to different audio truncated durations. Specifically, the audio truncated duration required when both wake word recognition and ambient sound event recognition are enabled is greater than the audio truncated duration required when both wake word recognition and voiceprint recognition are enabled, and the audio truncated duration required when both wake word recognition and voiceprint recognition are enabled is greater than the audio truncated duration required when only wake word recognition is enabled.

[0376] It should be noted that environmental sound event recognition typically requires analyzing longer background audio to accurately identify environmental sound events; voiceprint recognition requires longer audio to ensure the stability of feature extraction, thereby ensuring the accuracy of recognition.

[0377] In some examples, different event types in the audio data correspond to different audio truncation durations. For instance, when an event is identified as an ambient sound event, the truncation duration can be extended to ensure accurate identification of the ambient sound event; when an event is identified as a regular speech event, a shorter duration can be used.

[0378] The following provides an exemplary description of the information display method provided in the embodiments of this application, combined with specific scenarios.

[0379] For example, in a real-life scenario, an information display device identifies a human voice segment, "Xiaoming, time to eat," from continuously collected audio data. It confirms that the voice segment contains the wake-up keyword "Xiaoming," and then locates the end point of this wake-up word on the audio timeline. For instance, the precise moment when the pronunciation of the word "ming" ends is at 2.3 seconds. Then, based on this timestamp, the audio collected within one second before this time point is acquired as the first audio segment, and the audio collected within two seconds after this time point is acquired as the second audio segment, thus forming the complete instruction "Time to eat." In this way, through precise time positioning and segmentation, a high-quality audio data foundation can be provided for subsequent voiceprint recognition, semantic understanding, and environmental analysis.

[0380] It should be noted that the first audio segment includes the beginning of the wake-up keyword and the audio information preceding the wake-up keyword, while the second audio segment includes the audio information following the wake-up keyword, such as voice commands or content. By jointly analyzing the audio data before and after the wake-up keyword, a complete temporal context can be constructed, thereby accurately determining the context of the wake-up keyword and supplementing key situational elements, such as speaker identification, complete understanding of the speaker's semantics, and perception of the environmental situation.

[0381] In some embodiments of this application, the information display device uses the wake-up timestamp as a reference time point, splices the first audio segment and the second audio segment in chronological order to obtain a spliced ​​audio segment; then, based on the spliced ​​audio segment, it displays the first information.

[0382] In some embodiments of this application, the information display device uses the determined wake-up timestamp as the alignment reference point, and splices the first audio segment before the timestamp and the second audio segment after the timestamp in their continuous time order to obtain a longer audio segment that is continuous in time, i.e., spliced ​​audio segment, thereby reconstructing the audio context of a complete time window centered on the wake-up keyword.

[0383] For example, in a real-life scenario, a family member calls out, "Xiaoming, dinner's ready." After the information display device recognizes the wake-up keyword "Xiaoming," it determines the end timestamp of the wake-up keyword, and obtains the audio from 500 milliseconds before the end timestamp and the audio from 2 seconds after the end timestamp. These are then concatenated in chronological order to obtain a new audio segment that includes the complete context of the call command, namely, "Xiaoming, dinner's ready."

[0384] It should be noted that the spliced ​​audio segment may include the wake-up keyword itself, the speech features before the wake-up keyword, and the complete instruction after the wake-up word, thus providing a more complete data source for generating comprehensive first information.

[0385] For example, in conjunction with the above example, the information display device processes the spliced ​​audio segment "Xiaoming, time to eat" and identifies the descriptive information of the speech content, such as calling Xiaoming to eat and the speaker being the mother.

[0386] The above information display method will be illustrated with examples in specific scenarios below.

[0387] In some examples, in everyday scenarios, when a family member says "Xiaoming, dinner's ready" from another room, the information display device first locates the human voice audio data using a voice activity detection module. Then, it identifies the wake-up keyword "Xiaoming" using a wake-up model and pinpoints its end time. Next, based on this time point, it acquires the audio from 500 milliseconds before the wake-up keyword as the first audio segment and the audio from 2 seconds after the wake-up keyword as the second audio segment. These two audio segments are then concatenated to obtain the spliced ​​audio segment.

[0388] In this embodiment, by splicing the first audio segment and the second audio segment, a temporally continuous audio segment centered on the wake word is obtained, providing a more complete audio context for scene analysis. This avoids making a single reminder based solely on the wake keyword, ensuring that the most complete description of the spoken content can be obtained subsequently for context recognition, voiceprint recognition, ambient sound recognition, and other processing, thereby obtaining rich and accurate first information, and thus providing users with richer prompt information.

[0389] In some examples, combined with the above Figure 3B In a real-life scenario, when a family member says "Xiaoming, time to eat" from another room, the information display device first locates the voice segment through voice activity detection. Then, it identifies the wake word "Xiaoming" and determines its end time. Based on this time, the device acquires the first audio segment (1 second before the wake word) and the second audio segment (2 seconds after the wake word). After splicing the two audio segments, automatic speech recognition converts the complete audio into text. The natural language understanding module then performs in-depth analysis on the complete sentence "Xiaoming, time to eat," parsing the combined intent of "calling" and "eating," thus obtaining a complete description of the spoken content. Based on this description, the device displays the first information 34, "Someone is calling: Xiaoming, time to eat," on the information interface 33. This clearly includes the specific instruction, allowing the user not only to know that someone is calling them but also to accurately understand the speaker's intent and request, enabling them to respond appropriately.

[0390] In some examples, combined with the above Figure 5AIn a multi-person gathering scenario, when a user is called, the information display device, after recognizing the wake word, first locates the voice segment through voice activity detection, then identifies the wake word "Xiaoming" and determines its start time. Based on this time point, the device acquires the first audio segment (1.5 seconds before the wake word) and the second audio segment (2.5 seconds after the wake word). After splicing the two audio segments, the device uses a voiceprint recognition module to extract voiceprint features. These features are then compared with a pre-registered voiceprint database to calculate similarity, ultimately matching the identity information as "Teacher Wang." Based on the voiceprint recognition result, the device displays the identity prompt "Teacher Wang is calling Xiaoming" on the information interface (35), indicating the speaker's identity. This allows the user to not only know that someone is calling them but also accurately identify the caller, facilitating a targeted response.

[0391] In some examples, combined with the above Figure 7A In a real-life scenario, when someone calls out to the user and there is a boiling water sound in the background, the information display device first locates the voice segment through voice activity detection. Then, it identifies the wake-up word "Xiaoming" and uses the moment of recognition as a wake-up timestamp. The device acquires the first audio segment (3 seconds before the wake-up timestamp) and the second audio segment (3.5 seconds after the wake-up timestamp). After splicing the two audio segments, while processing the voice content, it separates the ambient sound component from the audio stream and inputs the separated ambient sound into an acoustic scene classification model to accurately identify the specific ambient sound category as "boiling water sound." The information display device then fuses the voice recognition result with the ambient sound detection result, displaying the first message 38, "Please note, someone is calling Xiaoming nearby. Please note, there may be a boiling water sound nearby," on the information interface 37. This message includes both a call reminder and an environmental safety warning, allowing the user to be aware of potential safety hazards in the environment while responding to the call and take appropriate measures.

[0392] The information display method provided in this application collects audio data and extracts human voice segments. When the human voice segment includes a wake-up keyword, it uses the end time of the wake-up keyword in the audio data as a reference to obtain audio data for a first time period before and a second time period after. Based on the aforementioned audio data, it generates first information, including at least one of the following: speech content description information, speaker identity information, environmental sound events, and second information. Thus, by acquiring audio data before and after the wake-up keyword, such as the dialogue content before and after the wake-up word and environmental sound events, and obtaining richer information based on this audio data, it can present users with richer acoustic scene information, such as the dialogue content before and after the wake-up word, speaker identity, and surrounding environmental sound characteristics. This helps users (especially hearing-impaired individuals) more accurately understand the complete context of the voice event, promptly identify the caller's intention and potential environmental risks, thereby improving the prompting effect.

[0393] In some embodiments of this application, after step 201 described above, the information display method provided in this application may further include steps 203 and 204:

[0394] Step 203: The information display device inputs the human voice audio data into the wake-up model. The wake-up model then performs wake-up word recognition processing on the human voice audio data to obtain the recognition result.

[0395] The identification result is either a first indication or a second indication; the first indication is used to indicate that a wake-up keyword has been identified; the second indication is used to indicate that a wake-up keyword has not been identified.

[0396] Step 204: If the recognition result is the first indication information, the information display device determines the wake-up timestamp.

[0397] The aforementioned wake-up timestamp includes at least one of the following:

[0398] The end time corresponding to the wake-up keyword on the timeline of the audio data;

[0399] The start time of the wake-up keyword on the timeline of the audio data;

[0400] The system time at the time of the first indication is output by waking up the model.

[0401] In some embodiments of this application, the wake-up model described above is a pre-trained acoustic model specifically designed for wake-up word recognition.

[0402] It's important to note that the core task of the wake-up model is to compare the actual pronunciation sequence of the input audio with the standard pronunciation sequence of the preset wake-up keywords. This processing is streaming, meaning the wake-up model can process continuous audio streams and output intermediate results in real time.

[0403] In some embodiments of this application, the recognition result of the wake-up model may include an indication of successful or unsuccessful recognition, and when a wake-up keyword is recognized, the start or end time of the wake-up keyword on the timeline of the audio data may be output.

[0404] In some embodiments of this application, the information display device may pre-input the pronunciation sequence of the wake-up keyword into the wake-up model for subsequent wake-up word recognition processing.

[0405] For example, in a real-life scenario, a family member calls out, "Xiaoming, dinner's ready!" The voice audio data is input into a wake-up model. The wake-up model performs streaming recognition on this voice audio data. When it processes "Xiaoming," the wake-up model confirms successful recognition and precisely locates the end of the pronunciation of the word "ming" at 2.5 seconds, generating a recognition result that includes a "recognition successful" indication and a timestamp. This timestamp can serve as a reference point for subsequently acquiring the first and second audio segments, providing a time alignment basis for the generation of the first information.

[0406] In some embodiments of this application, the information display device determines the system time of the first indication information output by the wake-up model as the wake-up timestamp.

[0407] For example, in conjunction with the above example, the wake-up model determines the end time of "Xiaoming" in the audio data as the wake-up timestamp. Then, the information display device extracts the audio content "Xiaoming" before the wake-up timestamp as the first audio segment, and obtains the audio content "I'm eating" after the wake-up timestamp as the second audio segment, ensuring the completeness of the analysis of subsequent speech content description information.

[0408] In this embodiment, a dedicated wake-up keyword recognition model is used. Compared with simple endpoint detection or template matching, it can more accurately detect wake-up keywords in continuous speech and resist some noise interference, thereby improving the accuracy and reliability of wake-up keyword recognition and timestamp localization. Furthermore, accurate timestamp localization provides a reliable basis for accurate segmentation of preceding and following audio segments, ensuring that complete speech segments, including wake-up keywords, can be captured, thus providing data assurance for generating high-quality first information.

[0409] In some embodiments of this application, the wake-up model includes a feature extraction module, an acoustic prediction module, and a decoding module; exemplarily, the process of performing wake-up word recognition processing on human voice audio data through the wake-up model in step 203 to obtain the recognition result may include the following steps 203a to 203d:

[0410] Step 203a: The information display device extracts acoustic features from the human voice audio data through the feature extraction module to obtain an acoustic feature sequence.

[0411] Step 203b: The information display device uses the acoustic prediction module to predict the acoustic feature sequence and obtain the acoustic prediction matrix.

[0412] The aforementioned acoustic prediction matrix includes probability distribution prediction information for the pronunciation sequence of human voice audio data.

[0413] Step 203c: The information display device decodes the acoustic prediction information through the decoding module to obtain at least one decoding path sequence.

[0414] Step 203d: The information display device uses the decoding module to match at least one decoding path sequence with a reference pronunciation sequence to obtain a matching result, and outputs the recognition result based on the matching result.

[0415] The above-mentioned reference pronunciation sequence is the pronunciation sequence of the wake-up keyword.

[0416] In some embodiments of this application, the feature extraction module described above is composed of a convolutional neural network (CNN) or a recurrent neural network (RNN) and is used to extract high-level acoustic feature sequences that characterize speech properties.

[0417] In some embodiments of this application, acoustic features are extracted from human voice audio data by the above-mentioned feature extraction module to extract high-level acoustic feature sequences that can characterize speech characteristics, such as Mel-frequency cepstral coefficients (MFCCs) or filter bank features.

[0418] It should be noted that the above acoustic feature sequence retains information in the time dimension.

[0419] For example, for the input human voice audio data "Xiaoming, it's time to eat", the feature extraction module processes it frame by frame or segment by segment, and outputs a corresponding MFCC feature sequence, which quantifies the spectral characteristics of the speech signal.

[0420] In some embodiments of this application, the decoding module described above may be based on a Connectionist Temporal Classification (CTC) or Recurrent Neural Network Transducer (RNN-T) architecture.

[0421] In some embodiments of this application, the information display device performs streaming decoding on the acoustic feature sequence obtained by the feature extraction module to generate one or more possible phoneme or character sequences, i.e., decoding path sequences.

[0422] It should be noted that in streaming processing, the decoding process is synchronized with the audio input, and possible intermediate results can be generated in real time.

[0423] For example, in conjunction with the above example, the information display device decodes the MFCC feature sequence obtained by the feature extraction module from the human voice audio data processing through the decoding module, and generates multiple candidate paths, such as path 1: "xiao mingchi fan le", path 2: "xiao min chi le", etc.

[0424] In some embodiments of this application, the information display device calculates the similarity or alignment score between each path and the target sequence by matching the decoded path sequence with the standard pronunciation sequence of a preset wake-up keyword (such as "xiao ming").

[0425] For example, referring to the above example, for the input human voice audio data "Xiaoming, time to eat," the information display device processes the audio signal through the feature extraction module of the wake-up model, extracting a T×80-dimensional MFCC acoustic feature sequence. Subsequently, the decoding module performs streaming decoding on this acoustic feature sequence, generating multiple candidate decoding path sequences such as "xiao ming" and "xiao min." In the matching stage, the decoding module calculates the matching degree between each candidate path and the preset standard wake-up keyword pronunciation sequence "xiao ming" using a forward algorithm. Based on this matching result, it determines whether the wake-up keyword has been recognized.

[0426] In this embodiment, acoustic features are extracted and decoded from human voice audio data using a wake-up model. Since the wake-up model can learn more stable speech feature representations in noisy environments, it can more accurately and reliably identify wake-up keywords in noisy environments. Based on the matching method of the decoding path, it can handle slight variations in the user's pronunciation, such as accent and speech rate, thereby improving the generalization of wake-up keyword recognition.

[0427] In some embodiments of this application, the process of generating the recognition result based on the matching result in step 203d above may include any one of the following steps 203d1 and 203d3:

[0428] Step 203d1: If the information display device finds a first decoding path sequence that matches the pronunciation sequence of the wake-up keyword in at least one decoding path sequence, and the confidence score of the target decoding path sequence is greater than the confidence threshold, it outputs the first recognition result through the wake-up model.

[0429] The first identification result mentioned above is the first indication information.

[0430] The first recognition result includes an indication of the recognition of the wake-up keyword and the end time point corresponding to the wake-up keyword on the timeline of the audio data.

[0431] Step 203d2: If there is a second decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is less than the confidence threshold, the information display device outputs a second recognition result through the wake-up model.

[0432] The second identification result mentioned above is the second indication information.

[0433] Step 203d3: If there is no third decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, the information display device outputs a third recognition result through the wake-up model.

[0434] The third identification result mentioned above includes the second indication information.

[0435] In some embodiments of this application, the information display device determines the matching result between the decoding path sequence and the wake-up word pronunciation sequence through the decoding module, and confirms successful wake-up when the high confidence condition is met, and outputs the first indication information.

[0436] In some embodiments of this application, the confidence threshold can be in the range of 0 to 1. For example, the confidence threshold can be 0.8, 0.85, 0.9, etc.

[0437] For example, for the input audio data "Xiaoming, it's time to eat," the information display device uses the wake-up model's feature extraction module to extract the MFCC acoustic feature sequence based on a CNN network, and the decoding module uses the CTC algorithm to perform streaming decoding to generate multiple candidate paths. When the decoded path of "xiao ming" matches the standard pronunciation sequence and the confidence score reaches 0.95, the wake-up model generates the first recognition result: recognition successful, and accurately locates the end time of the wake-up word as 2.35 seconds.

[0438] In some embodiments of this application, when the information display device determines that no decoding path matches the wake word sequence through the decoding module, it determines that the wake word has not been recognized and outputs the second indication information.

[0439] For example, for the input human voice audio data "The weather is really nice today", the information display device extracts the acoustic feature sequence through the feature extraction module of the wake-up model. All candidate paths generated by the decoding module, such as "jin tian tianqi zhen hao", do not match the standard wake-up word pronunciation sequence "xiao ming". At this time, the wake-up model generates a second recognition result: no wake-up keyword was recognized.

[0440] In some embodiments of this application, when the information display device determines that a matching path exists but the confidence level is insufficient through the decoding module, it determines that the recognition has failed and outputs a second indication message to avoid false wake-up.

[0441] For example, in a noisy party environment, given the input audio data "Xiao Lin, the weather is nice today," the information display device processes it using a wake-up model and obtains a decoding path that matches the "xiao ming" part. However, due to the high noise level in the environment, the speech may be distorted in timbre and pitch compared to actual speech. The confidence score of this decoding path is 0.65, which is lower than the confidence threshold of 0.85. At this point, the wake-up model outputs a second indication: no wake-up keyword was recognized.

[0442] In conjunction with the above embodiments, the following specific example will be used to illustrate the information display method provided in this application.

[0443] For example, the information display device continuously monitors the input data in the voice buffer through a wake-up model, detecting in real time whether it includes the wake-up keyword registered by the user. When the voice buffer accumulates 80ms of audio data, this 80ms audio data block is sent to the feature extraction module for processing. By continuously processing multiple 80ms audio data blocks, a T×80 acoustic feature matrix is ​​constructed, where T represents the time sequence length required for a single inference by the wake-up model, and 80 corresponds to the dimension of the Mel-frequency cepstral coefficients. Next, the T×80 acoustic feature matrix is ​​input into the acoustic prediction module of the wake-up model to obtain an N×C acoustic prediction matrix, where C represents the vocabulary size of the acoustic modeling unit. This matrix is ​​split into N independent 1×C sub-matrices, each sub-matrice corresponding to a minimum prediction time block. Then, the decoding module processes these 1×C acoustic prediction sub-matrices sequentially using a streaming processing method. During the decoding process, the beam search algorithm is used to match the pronunciation sequence of the wake-up word registered by the user. When a unit matching the target pronunciation sequence is found in the current 1×C dimension acoustic prediction matrix, and its acoustic confidence score is among the top K highest scores, the matching unit is added to the current candidate path.

[0444] For example, B independent alternative paths are maintained during the decoding stage. The sequence probability of each path is determined by the product of the conditional probabilities of its constituent units, as shown in formula (1):

[0445] P(y1:T∣x)=P(y1∣x)⋅P(y2∣x,y1)⋅…⋅P(yT∣x,y1:T−1) (1)

[0446] Where T represents the timestamp of the audio database when it is added to the alternative path, and x represents the score of the acoustic model.

[0447] For example, in order to avoid numerical underflow, a log-likelihood fraction is introduced in conjunction with the above formula (1), as shown in formula (2):

[0448] logP(y1:T|x)=∑ t=1 TlogP(yt∣x,y1:t−1) (2)

[0449] For example, when the sequence in a candidate path completely matches the pronunciation sequence registered by the user, the decoding module determines that the decoding is successful. Then, the overall confidence of the successful path is evaluated. If the confidence exceeds a preset threshold, the wake-up success signal is output through the wake-up model, which includes the identified wake-up keyword text information.

[0450] For example, upon successful wake-up confirmation (i.e., after the wake-up keyword is identified), the information display device extracts 1.5 seconds of audio data before and 3.5 seconds of audio data after the identified timestamp, forming a 5-second wake-up context audio segment. If the extracted time is less than 1.5 seconds, all available data in the voice buffer is used, and the total duration of 5 seconds is supplemented by extending the extraction time afterward. This 5-second audio segment will serve as input data for subsequent processing, used to obtain the semantic context of the wake-up word, perform voiceprint authentication, and execute scene understanding tasks across multiple dimensions, such as environmental sound detection.

[0451] In this way, through this layered processing mechanism, a complete technical chain is realized from low-latency wake-up detection to multi-dimensional scene understanding, which not only ensures the real-time performance of wake-up response, but also provides rich context-aware capabilities.

[0452] In this embodiment of the application, by calculating the confidence level of the target decoding path sequence that matches the wake word pronunciation sequence and determining whether the confidence level is higher than the confidence level threshold, audio segments with similar pronunciations but not real wake words, such as TV voice and conversations of others, can be effectively filtered out, thereby improving the accuracy of recognition.

[0453] In some embodiments of this application, the first information mentioned above includes speech content description information; exemplarily, step 203b3 may include steps E1 to E3:

[0454] Step E1: The information display device inputs the spliced ​​audio segment into the speech recognition model. The speech recognition model then performs speech-to-text processing on the spliced ​​audio segment to generate a continuous text sequence including wake-up keywords.

[0455] Step E2: The information display device generates descriptive information about the spoken content based on a continuous text sequence using a speech recognition model.

[0456] Step E3: The information display device displays a description of the spoken content.

[0457] In some embodiments of this application, the aforementioned speech recognition model is a pre-trained automatic speech recognition model. This speech recognition model can be an end-to-end deep learning model, such as one based on a Transformer or RNN-T architecture.

[0458] In some embodiments of this application, the input spliced ​​audio segments are processed end-to-end using a speech recognition model to transcribe them into corresponding text sequences.

[0459] It should be noted that since the input concatenated audio segment includes the wake-up keyword and its contextual information, the text sequence transcribed by the speech recognition model is continuous and complete, which naturally includes the wake-up keyword itself and the subsequent instruction content, i.e., a continuous text sequence.

[0460] For example, if the concatenated audio segment is "Xiaoming, time to eat", the speech recognition model will transcrib it end-to-end and directly output the continuous text sequence "Xiao Li, time to eat".

[0461] In some embodiments of this application, the information display device performs natural language understanding processing on the obtained continuous text sequence to generate structured speech content description information, such as extracting semantic elements such as the content of instructions, intentions, entities, and actions.

[0462] For example, in conjunction with the above example, the information display device performs NLU analysis on the transcribed text "Xiao Li is eating," which can accurately resolve the intent as "dining invitation" and may be associated with the entity "eating." The resulting speech content description information is: "Eating."

[0463] In some examples, once the wake-up model successfully recognizes the wake-up keyword, the information display device sets the kws_flag flag to 1, triggering the subsequent scene understanding process. At this point, the speech recognition service API is called, taking the acquired concatenated audio segment as input and feeding it into the speech recognition engine for end-to-end transcription processing. Specifically, first, basic prompt information is generated based on the wake-up model's output, displaying the wake-up word recognition result; then, the speech recognition engine completes a deep analysis of the entire audio segment, semantically concatenating the wake-up word text with subsequent instructions to form a complete contextual understanding result.

[0464] For example, in a real-life scenario, the user's registered wake word is "Xiaoming". When the voice input "Xiaoming" is detected, the wake-up model first recognizes the keyword "Xiaoming", triggering the first-stage response and displaying a prompt in the interface pop-up window: "Please note, someone is calling: Li Hua". Next, a 5-second audio clip containing the complete sentence is sent to the speech recognition engine for deep processing. After the speech recognition engine completes the full transcription and semantic analysis of "Xiaoming, it's time to eat", it updates the prompt information, displaying the complete contextual understanding result in the pop-up window: "Someone is calling: Xiaoming, it's time to eat".

[0465] The following provides an exemplary description of the information display method provided in the embodiments of this application through specific scenarios.

[0466] For example, in a real-life scenario, when a family member says "Xiaoming, dinner's ready" from another room, the information display device first locates the human voice audio data using a voice activity detection module. Then, it identifies the wake-up keyword "Xiaoming" using a wake-up model and precisely pinpoints its end time. Next, based on this time point, it acquires the audio from 500 milliseconds before the wake-up keyword as the first audio segment and the audio from 2 seconds after the wake-up keyword as the second audio segment, concatenating the two segments to form a complete context. Then, a deep learning-based speech recognition model converts the complete audio into a text sequence, and a natural language understanding module performs semantic analysis on "Xiaoming, dinner's ready," accurately identifying the combined intent of "calling" and "dining invitation." Based on the deep semantic analysis results, the information display device generates a speech content description: "Someone is calling you to dinner." This description clearly conveys the specific instruction and interaction intent, enabling the user not only to be aware of the calling behavior but also to accurately understand the speaker's request, thus allowing for an appropriate response.

[0467] In this embodiment of the application, by analyzing the continuous speech including the wake word and subsequent instructions as a whole, it is possible to accurately obtain the complete speech content description information before and after the wake word in the audio segment, thereby deeply understanding the user's true intention and interaction scenario, avoiding semantic fragmentation caused by segmented processing, and ensuring a complete understanding of the user's instructions.

[0468] In some embodiments of this application, the first information mentioned above includes speaker identity information; exemplarily, step 203b3 may include steps F1 to F4:

[0469] Step F1: Input the spliced ​​audio segment into the voiceprint recognition model. The voiceprint recognition model extracts voiceprint features from the spliced ​​audio segment to obtain voiceprint feature information.

[0470] Step F2: The information display device uses a voiceprint recognition model to calculate the similarity between the voiceprint feature information and at least one preset voiceprint feature information to obtain similarity information.

[0471] Step F3: The information display device determines the speaker identity information corresponding to the first voiceprint feature information as the speaker identity information corresponding to the human voice audio data;

[0472] Step F4: The information display device displays the speaker's identity information.

[0473] Wherein, the aforementioned first voiceprint feature information is a voiceprint feature information among at least one preset voiceprint feature information whose similarity to the voiceprint feature information is greater than a first similarity threshold.

[0474] In some embodiments of this application, the aforementioned voiceprint recognition model is a pre-trained voiceprint recognition model. This voiceprint recognition model is built based on a deep learning network, such as an x-vector or d-vector architecture, and is used to extract the speaker's identity features from audio.

[0475] In some embodiments of this application, spliced ​​audio segments are processed by a voiceprint recognition model, and feature vectors that can uniquely represent the speaker's identity, i.e., voiceprint feature information, are extracted from them.

[0476] It should be noted that this voiceprint feature information is a high-dimensional vector, such as an embedding vector, which includes the speaker's physiological and behavioral acoustic features.

[0477] For example, in a multi-person gathering scenario, if Xiao Li says "Xiao Ming, come here for a moment", the information display device will input the spliced ​​audio segment including the complete sentence into the voiceprint model to extract voiceprint features and extract Xiao Li's voiceprint feature information.

[0478] In some embodiments of this application, the information display device compares the extracted voiceprint feature information with at least one preset voiceprint feature information that has been pre-registered and stored in a database. Exemplarily, the comparison process obtains quantified similarity information by calculating metrics such as cosine similarity or Euclidean distance.

[0479] In some embodiments of this application, each preset voiceprint feature is associated with an identity identifier, namely preset voiceprint object information, such as "father", "mother", "child", "Xiao Li", etc.

[0480] In some embodiments of this application, if the calculated similarity information is higher than a preset threshold, it is determined that the current speaker is successfully matched with a preset voiceprint object, and then the identity information corresponding to the preset voiceprint object is determined as the speaker identity information of the current audio.

[0481] In some embodiments of this application, the value range of the first similarity threshold can be 0 to 1. For example, the first similarity threshold can be 0.8, 0.85, 0.9, etc.

[0482] For example, in conjunction with the above example, the information display device compares the extracted voiceprint features with preset voiceprint feature information and calculates a similarity of 0.91. If the similarity is greater than the first similarity threshold of 0.85, then the speaker's identity is determined to be the voiceprint object information associated with the preset voiceprint feature information, such as Xiao Li.

[0483] In some examples, when the wake-up model detects a keyword and triggers the wake-up condition, the extracted spliced ​​audio segment is input into the voiceprint recognition model for processing. The voiceprint model first performs deep feature extraction on the input audio data, generating a high-dimensional embedded feature sequence. Then, this real-time extracted embedded feature sequence is compared one by one with all pre-registered voiceprint embedded feature sequences, and the degree of voiceprint matching is evaluated by calculating cosine similarity. Next, the maximum cosine similarity value is selected from all comparison results, and the corresponding registrant index information is recorded. When this maximum value exceeds a preset voiceprint recognition threshold, the voiceprint verification is considered successful, and the registrant's identity information and the wake-up word content are simultaneously displayed in a pop-up window. If the maximum value does not reach the threshold requirement, the voiceprint verification fails, and the pop-up window only displays basic wake-up word information.

[0484] For example, when Teacher Wang calls on students in class and says "Xiaoming," the wake-up word recognition is triggered first, followed by the voiceprint verification process. Features are extracted from the collected audio using a voiceprint model, and similarity is calculated with a pre-stored voiceprint template. When the similarity score reaches 0.92, exceeding the threshold of 0.85, the voiceprint verification is successful, and the complete identity verification result is displayed in a pop-up window: "Teacher Wang calls: Xiaoming."

[0485] In some embodiments of this application, the electronic device can input spliced ​​audio segments into the voiceprint recognition model for processing when the voiceprint recognition function is enabled.

[0486] The following describes the information display method provided in the embodiments of this application by way of example, in conjunction with specific scenarios.

[0487] In some examples, users are in a group gathering scenario. When Xiao Li calls a user, the information display device, after recognizing the wake word, extracts the audio segment including the complete call statement. The voiceprint recognition module then performs deep feature extraction on this audio, obtaining a 256-dimensional voiceprint embedding vector. This vector is then compared with preset voiceprint feature information in a pre-registered voiceprint database using cosine similarity calculation. Assuming the matching degree between the embedding vector and Xiao Li's voiceprint feature information is 0.92, which is greater than the similarity threshold of 0.85, the speaker's identity is confirmed as "Xiao Li." This allows users to accurately identify who is calling them in noisy, multi-person environments. In this way, users not only promptly perceive being called but also accurately pinpoint the specific caller, enabling them to respond directly and accurately to Xiao Li, avoiding misresponses due to unclear voice sources and improving interaction efficiency in complex scenarios.

[0488] In this embodiment, voiceprint recognition is performed by splicing audio segments to extract discriminative voiceprint features. These features are then input into a voiceprint recognition model for processing to identify the speaker's identity information. This enables high-precision identity verification even in noisy environments or short speech scenarios, thereby improving interaction efficiency in complex scenarios.

[0489] In some embodiments of this application, the first information includes environmental sound event information; exemplarily, step 203b3 above may include steps G1 to G4:

[0490] Step G1: The information display device inputs the spliced ​​audio segment into the dry sound separation model, and the ambient sound data is separated from the spliced ​​audio segment through the dry sound separation model.

[0491] Step G2: The information display device inputs the ambient sound data into the acoustic scene classification model. Through the acoustic scene classification model, the ambient sound data is classified and identified to obtain ambient sound category information.

[0492] Step G3: The information display device generates environmental sound event information based on the environmental sound category information through the acoustic scene classification model.

[0493] Step G4: The information display device displays ambient sound event information.

[0494] In some embodiments of this application, the dry audio separation model can be a sound source separation model, which can separate mixed audio into different sound source streams and output a clean ambient sound data stream.

[0495] In some embodiments of this application, the information display device inputs the obtained spliced ​​audio segments into a speech separation model for processing, separates human voice from background ambient sound, and outputs a clean ambient sound data stream.

[0496] For example, in a real-life scenario, Xiaoming's family member says in the kitchen, "Xiaoming, are you home?" The spliced ​​audio segment mixes the family member's voice with the sound of water boiling. The information display device inputs the spliced ​​audio segment into a dry sound separation model to perform environmental sound recognition and separation, and identifies the sound of water boiling in the environment.

[0497] In some embodiments of this application, the aforementioned acoustic scene classification model is a pre-trained classifier that can identify the semantic category of audio segments, such as "smoke alarm", "boiling water", "doorbell", "broken glass", "television sound", "street noise", etc.

[0498] In some embodiments of this application, the information display device inputs the separated ambient sound data into an acoustic scene classification model for processing to obtain specific ambient sound category information.

[0499] For example, in conjunction with the above example, the information display device inputs the separated water boiling sound into the acoustic scene classification model and identifies its category as "water boiling sound".

[0500] In some embodiments of this application, the information display device formats or associates the obtained category information to generate an environmental sound event that is ultimately used for prompting.

[0501] In some embodiments of this application, the electronic device can input spliced ​​audio segments into a dry sound separation model for environmental sound recognition and separation, as well as perform subsequent related processing, when the environmental sound recognition function is enabled.

[0502] In some examples, the information display device initiates an acoustic scene detection process, taking the aforementioned spliced ​​audio segments as input. First, it feeds the data into a deep learning-based dry sound separation model for processing. This model employs spectral analysis and source separation techniques to extract clean ambient sound data from the mixed audio. Subsequently, feature extraction is performed on the ambient sound data, converting it into 500×80-dimensional Mel-frequency spectral features. These acoustic features are then fed into a pre-trained acoustic scene classification model for multi-label classification and recognition, resulting in sound event categories, such as boiling water, fire alarms, smoke alarms, doorbells, and other specific events. When the prediction score for a sound event or acoustic scene exceeds a preset threshold, the category is considered successfully identified. In the case of successful recognition, the index information of the corresponding category is returned, and the ambient sound detection results and wake-up prompts are simultaneously displayed in a pop-up window. If none of the sound categories reach the recognition threshold, only basic wake-up word information is displayed.

[0503] Specifically, when a user enables the ambient sound recognition function, if the ambient sound detection does not find any valid events, the pop-up will maintain the basic wake-up prompt; if the ambient sound detection successfully identifies a specific event, the pop-up will display both the wake-up information and the ambient sound warning.

[0504] For example, in a real-life scenario, when someone calls the user and there is a boiling sound in the background, the information display device, while processing the voice content, separates the pure ambient sound component from the mixed audio stream using a dry sound separation module. The separated ambient sound is then input into an acoustic scene classification model based on a convolutional neural network for multi-class recognition, resulting in the ambient sound being classified as "boiling sound." Subsequently, a composite prompt message is generated: "Please note that someone is calling Xiaoming nearby, and a boiling sound has been detected." This message includes both a reminder of the call and a kitchen safety warning, enabling the user to promptly perceive potential safety hazards in the environment while responding to the call and take appropriate safety measures.

[0505] In this embodiment of the application, by performing environmental sound recognition and processing on spliced ​​audio segments to obtain environmental sound data, and then performing category recognition on the separated environmental sound data, a two-level processing flow is used to accurately detect and identify potential dangers or important environmental events, such as boiling water, fire alarms, and intrusion alarms, from complex audio mixed with human voices, thus providing users with the most comprehensive and effective scene perception.

[0506] In some embodiments of this application, the information display method provided in this application may further include the following steps 207 to 209:

[0507] Step 207: The information display device displays the wake-up word setting interface.

[0508] Step 208: The information display device receives the wake-up word setting input in the wake-up word setting interface;

[0509] Step 209: In response to the wake-up word setting input, the information display device displays the wake-up keyword set in the wake-up word setting interface.

[0510] In some embodiments of this application, the information display device displays a user interface specifically for setting wake-up keywords.

[0511] It should be noted that the user interface includes necessary input controls and visual guidance to assist users in completing the wake word setting process.

[0512] In some embodiments of this application, the above-mentioned wake-up word setting input is used to set wake-up keywords.

[0513] In some embodiments of this application, the wake-up word setting input can include any of the following: user touch input, voice input, gesture input, folding or unfolding operation of the foldable screen, or other feasible inputs, which are not limited in this application embodiment. Further, the touch input can be: user click input, swipe input, press input, etc. Further, the click operation can be any number of clicks. The swipe operation can be a swipe operation in any direction, such as swiping up, swiping down, swiping left, or swiping right, which are not limited in this application embodiment.

[0514] In some embodiments of this application, the wake word setting interface may include a wake word text area. After receiving the user's wake word setting input, the information display device displays the wake word input by the user in the wake word text area, thereby providing the user with clear visual feedback.

[0515] For example, such as Figure 10AAs shown, the wake-up word setting interface 51 includes a wake-up word setting window 52. At the top of the wake-up word setting window 52, a guiding prompt message "Set Wake-up Word" is displayed, and a wake-up word input text box 53 is configured below. The user can activate the wake-up word input text box 53 through touch or click operations and input a custom wake-up keyword, such as "Jie Yuan". When the user finishes input and clicks "Confirm" in the interface, the information display device saves the wake-up keyword to the system configuration library and simultaneously updates the relevant parameter settings of the wake-up model.

[0516] In some embodiments of the present application, the user can add one or more wake-up keywords in the wake-up word setting interface.

[0517] Exemplarily, as Figure 10B shown, the user activates the wake-up word input text box 53 through touch or click operations and inputs custom wake-up keywords, such as "Jie Yuan" and "Xiaoming". After adding these two wake-up keywords, the user continues to input "Xiaowang" in the wake-up word input text box 53 to add the wake-up keyword "Xiaowang".

[0518] In the embodiments of the present application, the user can customize the wake-up keyword in the wake-up word setting interface, making the setting of the wake-up keyword more flexible and meeting the actual needs of the user. And by displaying the set wake-up word in real time on the interface, clear and immediate operation feedback is provided to the user. Thus, the flexibility of setting the wake-up keyword is provided.

[0519] In some embodiments of the present application, after the above step 208, the information display method provided by the embodiments of the present application may further include the following steps 210 to step 212:

[0520] Step 210: When the wake-up keyword includes polyphonic characters, the information display device displays at least one alternative pronunciation option of the polyphonic characters in the wake-up keyword according to the text content of the wake-up keyword in the wake-up word setting interface.

[0521] Step 211: The information display device receives a selection input for a target alternative pronunciation option among the at least one alternative pronunciation options.

[0522] Step 212: The information display device determines the pronunciation corresponding to the target alternative pronunciation option as the pronunciation of the polyphonic characters in response to the selection input.

[0523] In some embodiments of the present application, after the user inputs the text of the wake-up keyword, the information display device generates and displays one or more possible alternative pronunciation options of the wake-up keyword based on the text content through a built-in speech synthesizer or pronunciation database. For example, it is presented in the form of pinyin annotation, audio playback buttons, etc. The above pronunciation information may cover different tones, speech rates, or common variant pronunciations.

[0524] Exemplarily, in combination with the above Figure 10A As shown, such as Figure 10C As shown, the user activates the wake-up word input text box 53 through a touch or click operation, and inputs a custom wake-up keyword, such as "Jie Yuan". Based on the pronunciation of the wake-up keyword, the information display device displays two pronunciations of "Jie", namely "jie" and "xie", and two pronunciations of "Yuan", namely the second tone "yuán" and the fourth tone "yuàn". The user can select the pronunciation of "Jie Yuan" from the above pronunciations.

[0525] In some embodiments of the present application, the information display device binds the pronunciation selected by the user to the text of the wake-up keyword, and uses this as the standard pronunciation for subsequent voice recognition and wake-up model matching.

[0526] In the embodiments of the present application, by displaying at least one alternative pronunciation information of the wake-up keyword for the user to select, the user can conveniently select the pronunciation to be used from the alternative pronunciation information. In this way, the user can select the pronunciation of polyphonic characters, rare characters or words with local accents, effectively avoiding the problems of missed wake-up and false wake-up caused by uncertain pronunciation, thereby improving the accuracy and reliability of subsequent wake-up keyword recognition.

[0527] In some embodiments of the present application, the information display method provided by the embodiments of the present application may further include the following steps 213 to step 219:

[0528] Step 213: The information display device displays a wake-up word management interface.

[0529] Among them, the above wake-up word management interface includes at least one wake-up keyword and at least one voiceprint addition option, and one wake-up keyword corresponds to one voiceprint addition option;

[0530] Step 214: The information display device receives a sixth input for the voiceprint addition option corresponding to the target wake-up keyword among at least one wake-up keyword.

[0531] Step 215: The information display device responds to the sixth input and displays a voiceprint object setting interface corresponding to the target wake-up keyword.

[0532] Step 216: The information display device receives a voiceprint object information setting input on the voiceprint object setting interface.

[0533] Step 217: The information display device responds to the voiceprint object information setting input and displays the voiceprint object information set by the voiceprint object information setting input on the voiceprint object setting interface.

[0534] The aforementioned voiceprint object information is used to obtain the speaker's identity information when the wake-up keyword is recognized.

[0535] Step 218: The information display device acquires the wake-up word voice data corresponding to the target wake-up keyword.

[0536] Step 219: The information display device performs voiceprint registration on the target wake-up keyword based on the wake-up word voice data, and establishes the association between the voiceprint information of the target wake-up keyword and the voiceprint object information.

[0537] In some embodiments of this application, the wake word management interface includes one or more pre-set wake words and provides a function entry point for associating voiceprints for each wake word, namely, a voiceprint addition option.

[0538] In some embodiments of this application, the wake word management interface may also include a switch option corresponding to each wake keyword.

[0539] For example, such as Figure 11A As shown, the wake word management interface 61 includes a wake word setting window 62, which includes three wake words: "Jieyuan", "Xiaoming" and "Xiaowang", corresponding to switch options 63, 64 and 65 respectively, and displays an "Add Voiceprint" option. The wake word "Jieyuan" is in the "on" state of switch option 63, indicating that the word is enabled. Users can add voiceprint information for the wake word through the "Add Voiceprint" option.

[0540] For example, such as Figure 11B As shown, the wake word management interface 61 includes a wake word setting window 62, which includes three wake words: "Jieyuan", "Xiaoming" and "Xiaowang", which correspond to voiceprint addition options 66, 67 and 68 respectively. Users can add voiceprint information for each wake word through the voiceprint addition option corresponding to each wake word.

[0541] In some embodiments of this application, the sixth input may include any of the following: user touch input, voice input, gesture input, folding or unfolding operation of the foldable screen, or other feasible inputs, which are not limited in this application embodiment. Further, the touch input may be: user click input, swipe input, press input, etc. Further, the click operation may be any number of clicks. The swipe operation may be a swipe operation in any direction, such as swiping up, swiping down, swiping left, or swiping right, which are not limited in this application embodiment.

[0542] In some embodiments of this application, the above-mentioned voiceprint object setting interface is used to set and enter object information associated with the voiceprint.

[0543] For example, in conjunction with the above Figure 11B ,like Figure 11C As shown, after the user clicks the voiceprint addition option 67, the information display device displays the voiceprint object setting interface 69 corresponding to the wake-up keyword "Xiaoming". The voiceprint object setting interface 69 includes the prompt information "Custom Voiceprint", "Add Voiceprint Owner Name", and a voiceprint object information text box 70. The user can enter the name of the person corresponding to the voiceprint in the voiceprint object information text box 70, for example, Xiao Li, and click the "Confirm" option to trigger the confirmation of the entered voiceprint owner name.

[0544] Understandably, the name of the person whose voiceprint belongs is used to identify the speaker when the wake-up keyword is detected.

[0545] In this embodiment of the application, by providing a unified wake word management interface for users to manage wake words and voiceprint input processes, users can intuitively and conveniently bind identity information for different users, which greatly simplifies the configuration flexibility in multi-user scenarios.

[0546] In some embodiments of this application, after the voiceprint object information is set, the information display device guides the voiceprint object to say the target wake-up keyword, and collects one or more segments of voice data of the user saying the keyword through the microphone, that is, wake-up word voice data.

[0547] In some embodiments of this application, the information display device uses the collected wake-up word voice data to extract voiceprint features, obtains the corresponding voiceprint feature information, stores it in advance, and establishes the association between the voiceprint feature information and the provider of the voiceprint feature information.

[0548] For example, the information display device collects the wake-up word voice data recorded by Li Hua, extracts the voiceprint features from the wake-up word voice data, generates a feature vector, and associates and stores this feature vector with the identity identifier "Li Hua". Subsequently, when this feature vector is matched through voiceprint recognition, the speaker's identity information can be confirmed as "Li Hua".

[0549] In this embodiment, by registering voiceprints for wake-up keywords and strongly associating voiceprints with specific wake-up words and user identities, it is more effective than registering voiceprints for general voice commands in extracting unique acoustic features related to the pronunciation of wake-up words. This helps improve the accuracy and anti-interference capability of voiceprint recognition, thereby enabling the quick and accurate location of the valid command issuer in multi-person, noisy, or emergency scenarios, and executing operations that match the identity and permissions, thus avoiding erroneous responses.

[0550] In some embodiments of this application, the wake-up word speech data includes N wake-up word speech samples; exemplarily, step 218 may include steps 218a to 218c:

[0551] Step 218a: The information display device displays the wake-up word voiceprint recording interface.

[0552] The wake-up word voiceprint recording interface includes a first recording prompt message and a first voice input control. The first recording prompt message is used to prompt the user to record a wake-up word voice sample and the number of times the wake-up word voice sample is recorded.

[0553] Step 218b: The information display device receives voice input to the first voice input control.

[0554] Step 218c: The information display device responds to the voice input and collects N wake-up word voice samples.

[0555] Where N is an integer greater than 1.

[0556] In some embodiments of this application, the wake word voiceprint recording interface is a voice recording interface used for voiceprint registration.

[0557] In some embodiments of this application, the first voice input control described above is used to trigger the recording of a wake-up word.

[0558] For example, such as Figure 12A As shown, the wake-up word voiceprint recording interface 71 includes a wake-up word voiceprint recording window 72. This window 72 includes the prompt message "Record 3 voice recordings, the content of which is the wake-up keyword," guiding the user to complete the voiceprint recording operation. The wake-up word voiceprint recording window 72 also includes three parallel circular status indicators 73, each corresponding to one of the three voice recording tasks. The first indicator is active, indicating that the first voice recording is currently in progress, while the other two indicators remain inactive, visually displaying the overall recording progress. At the bottom of the wake-up word voiceprint recording window 72 is a "Record" button 74. When the user clicks this button, the information display device activates the audio acquisition module to begin recording the user's voice sample of the wake-up keyword. After each successful recording, the corresponding status indicator updates to active, automatically preparing for the next recording, until all three voice samples are collected. Thus, through clear visual guidance and real-time status feedback, the user can accurately and efficiently complete the voiceprint registration process.

[0559] In some embodiments of this application, the aforementioned voice input may include any of the following: user touch input, voice input, gesture input, folding or unfolding operation of the foldable screen, or other feasible input methods. This application does not limit the types of input methods used. Further, the aforementioned touch input may include: user click input, swipe input, press input, etc. Further, the aforementioned click operation may be any number of clicks. The aforementioned swipe operation may be a swipe operation in any direction, such as swiping up, swiping down, swiping left, or swiping right, etc., and this application does not limit the types of swipe operations used.

[0560] For example, in conjunction with the above Figure 12A Following the interface prompts, the voiceprint subject clicked the record button on the screen to prepare for the first voice recording of the wake-up keyword. For example, the voiceprint subject Xiao Li read "Jie Yuan" to register the voiceprint information for that wake-up keyword.

[0561] In some embodiments of this application, the information display device responds to user input by activating an audio acquisition module to record the user's voice reading of the target wake-up keyword. This process is repeated until a preset number of voice samples, such as three wake-up word voice samples, are collected.

[0562] For example, in conjunction with the above example, after each time Xiao Li clicks the record button, he reads "Jie Yuan" aloud. The information display device collects three segments of Xiao Li's voice data reading "Jie Yuan". At this time, the three parallel circular status indicators 73 in the wake-word voiceprint recording window 72 are all active, indicating that all three segments of voice data have been collected. Figure 12B As shown.

[0563] In this embodiment of the application, by recording N wake-up word speech samples, a sufficient and accurate data foundation is provided for subsequent voiceprint feature extraction, which helps to generate more robust and accurate voiceprint feature information, thereby improving the recognition rate of voiceprint objects of subsequent wake-up keywords.

[0564] In some embodiments of this application, the process of performing voiceprint registration on the target wake-up keyword based on wake-up word speech data in step 219 above may include steps 219a to 219c1, or may include steps 219a, 219b and 219c2:

[0565] Step 219a: The information display device extracts voiceprint features from N wake-up word speech samples to obtain N voiceprint feature information.

[0566] Step 219b: The information display device calculates the similarity between N voiceprint feature information.

[0567] Step 219c1: When the similarity between any two voiceprint feature pieces among the N voiceprint feature pieces is greater than the second similarity threshold, the information display device performs voiceprint registration for the target wake-up keyword based on the N voiceprint feature pieces.

[0568] Step 219c2: If the similarity between any two voiceprint feature information in the N voiceprint feature information is less than the second similarity threshold, the information display device acquires M personal voice samples and performs voiceprint registration for the target wake-up keyword based on the M personal voice samples.

[0569] Where M is greater than N.

[0570] In some embodiments of this application, the aforementioned voiceprint feature information can be a voiceprint feature vector.

[0571] In some embodiments of this application, a wake-up word speech sample corresponds to a voiceprint feature information.

[0572] In some embodiments of this application, the information display device extracts voiceprint features from N collected wake-up word speech samples, generating a corresponding voiceprint feature vector for each sample, thereby obtaining a set of feature vectors.

[0573] For example, in conjunction with the above example, the information display device processes three voice samples recorded by the voiceprint object "Xiao Li" and extracts three high-dimensional voiceprint feature vectors V1, V2 and V3 respectively.

[0574] In some embodiments of this application, when the wake-up word speech sample includes at least three speech samples, that is, when multiple voiceprint feature information includes at least three voiceprint feature information, the information display device calculates the pairwise similarity between all voiceprint feature vectors, for example, by calculating cosine similarity, and calculates an average similarity based on the pairwise similarity to comprehensively evaluate the consistency of this set of vectors.

[0575] In some embodiments of this application, when the wake-up word speech sample includes two speech samples, that is, when multiple voiceprint feature information includes two voiceprint feature information, the information display device calculates the similarity between the two voiceprint feature vectors, for example, by calculating the cosine similarity, and comprehensively evaluates the consistency of the set of vectors based on the similarity.

[0576] It should be noted that a high average similarity means that the user's pronunciation is stable in different readings and the consistency of voiceprint features is good, that is, the voiceprint in the recorded speech data is more accurate.

[0577] For example, in conjunction with the above example, the information display device calculates the similarity between V1 and V2, V1 and V3, and V2 and V3, and obtains an average similarity of 0.88.

[0578] In some embodiments of this application, the first similarity threshold can be in the range of 0 to 1. For example, the first similarity threshold can be 0.8, 0.85, 0.9, etc. For example, the second similarity threshold can be 0.8.

[0579] It should be noted that the thresholds involved in the embodiments of this application, such as the confidence threshold, the first similarity threshold, and the second similarity threshold, can be set according to actual needs, and the embodiments of this application do not limit them.

[0580] In some embodiments of this application, if any two calculated similarities are both higher than a second similarity threshold, the sample quality is considered acceptable, and the voiceprint registration process continues.

[0581] In some embodiments of this application, if the calculated average similarity is higher than a second similarity threshold, the sample quality is considered acceptable, and the voiceprint registration process continues.

[0582] For example, in conjunction with the above example, assuming that the calculated average similarity is 0.88, since 0.88 is greater than 0.8, the information display device determines that the sample is qualified, and generates a fused voiceprint feature information based on V1, V2, and V3 as the voiceprint feature information of the voiceprint object Xiao Li, or determines any one of them as the voiceprint feature information of the voiceprint object Xiao Li, and associates it with the identity of the voiceprint object Xiao Li.

[0583] In some embodiments of this application, the aforementioned human voice sample can be any content recorded by the voiceprint object, such as a voice sample reading a wake-up keyword or a voice sample reading other content.

[0584] In some embodiments of this application, if the calculated average similarity is lower than a threshold, it indicates that the differences in the user's pronunciation across several recordings are too large, and the sample quality is unreliable. In this case, the information display device activates an alternative scheme, guiding the user to re-record multiple audio segments.

[0585] For example, if in a noisy environment, the similarity of three voice samples of the wake-up keyword "Jie Yuan" recorded by the voiceprint subject Xiao Li is only 0.65, then the user is prompted to re-record five custom voice samples.

[0586] In some examples, after successful registration of the voiceprint wake-up word, the information display device determines the voiceprint similarity of the three recorded voice samples. If the similarity meets the condition, the voiceprint registration is confirmed to be successful. Alternatively, if the similarity does not meet the condition, the user is prompted to re-record five 5-second voice samples. At this time, there are no restrictions on the voice content. Then, the same voiceprint feature extraction and similarity analysis are performed on these five new samples. If the voiceprint similarity between all samples reaches the quality threshold, the voiceprint registration is finally completed. Otherwise, the user is prompted to try again.

[0587] Specifically, the user enters a specified wake-up word in three separate recordings as prompted. Each recorded voice is processed by a voiceprint model to generate a corresponding voiceprint embedding feature vector. The information display device calculates the pairwise cosine similarity of these three embedding vectors, obtaining three similarity values, and takes their average as the final quality assessment index. When the average similarity is greater than a preset quality threshold, such as 0.85, the sample quality is deemed acceptable, and the registration process continues. At this point, all the voiceprint embedding sequences recorded in the three recordings are weighted and averaged to generate a fused voiceprint feature information, which is then associated with and stored with the registrant's identity information as benchmark data for subsequent voiceprint verification.

[0588] In this embodiment, by recording N wake-up word speech samples and performing consistency verification on the voiceprint feature information corresponding to the N wake-up word speech samples, voiceprint registration is performed if the verification passes, or if the verification fails, the user is prompted to record speech samples again for voiceprint registration. In this way, the reliability of voiceprint registration is significantly improved through the similarity verification mechanism.

[0589] In some embodiments of this application, the following step H1 may be included before step 219a described above:

[0590] Step H1: The information display device verifies the pronunciation of N wake-up word speech samples based on the pronunciation of the target wake-up keyword.

[0591] For example, in conjunction with step H1 above, step 219a above may include the following step 219a1:

[0592] Step 219a1: If all N wake-up word speech samples pass the verification, the information display device extracts voiceprint features from the N wake-up word speech samples to obtain N voiceprint feature information.

[0593] In some embodiments of this application, before extracting voiceprint features, the information display device matches the pronunciation of the target wake-up keyword with each wake-up word speech sample in the N wake-up word speech samples to verify the pronunciation of the N wake-up word speech samples. If all N wake-up word speech samples can match the pronunciation of the target wake-up word, the verification is confirmed to be successful, and voiceprint features are extracted from the N wake-up word speech samples.

[0594] In some embodiments of this application, before extracting voiceprint features, the information display device may use a speech recognition module to recognize each recorded wake-up word speech sample and transcribe it into text. Then, the transcribed result is compared with the standard pronunciation of the target wake-up keyword to determine whether the user has correctly pronounced the wake-up word.

[0595] In some embodiments of this application, before extracting voiceprint features, the information display device can input N wake-up word speech samples into the wake-up model for processing to identify whether the N wake-up word speech samples match the pronunciation of the target wake-up keyword.

[0596] For example, in conjunction with the above example, in the voiceprint registration process, the user first enters a voiceprint name for identification, which will be displayed as the information when the voiceprint is triggered subsequently. Then, the information display device guides the user to record three 5-second audio samples, and the content of each audio sample must be a registered wake-up word, with pronunciation consistent with the standard pronunciation set during the wake-up word registration stage. Then, pronunciation verification is performed on the three recorded audio samples: each audio sample is input into the wake-up model for streaming recognition, resulting in three sets of wake-up scores. If all three scores exceed a preset pronunciation consistency threshold, the pronunciation is deemed compliant, and the wake-up word registration is confirmed. If any score is below the threshold, the pronunciation is deemed inconsistent, and the user is prompted to re-record the three audio samples.

[0597] In this embodiment of the application, by verifying the pronunciation of the N recorded wake-up word speech samples, a strong correlation between voiceprint features and standard pronunciation is established, thereby ensuring the pronunciation standardization of the voiceprint registration samples and effectively avoiding voiceprint feature contamination caused by pronunciation deviation, thus ensuring the quality of voiceprint registration.

[0598] In some embodiments of this application, the process of obtaining M personal voice samples in step 219c above may include the following steps 219d1 to 219d3:

[0599] Step 219d1: The information display device displays the voiceprint recording interface.

[0600] The aforementioned voiceprint recording interface includes a second recording prompt message and a second voice input control. The second recording prompt message is used to prompt the user to record M personal voice samples.

[0601] Step 219d2: The information display device receives voice input to the second voice input control.

[0602] Step 219d3: The information display device responds to the voice input and collects M personal voice samples.

[0603] In some embodiments of this application, if the N recorded wake-up word voice samples do not meet the requirements, the information display device records M personal voice samples.

[0604] In some embodiments of this application, the voiceprint recording interface is used to record a segment of speech content that is different from the previous one, usually longer or with random content, i.e., a human voice sample.

[0605] For example, such as Figure 13A As shown, the voiceprint recording interface 81 includes a voiceprint recording window 82, which displays the prompt "Record 5 voice recordings, content is unlimited," guiding the user through the voiceprint recording process. The voiceprint recording window 82 also includes five parallel circular status indicators 83, each corresponding to one of the three voice recording tasks. The first indicator is active, indicating that the first voice recording is in progress, while the other four indicators remain inactive, visually displaying the overall recording progress. At the bottom of the voiceprint recording window 82 is a "Record" button. When the user clicks this button, the information display device activates the audio acquisition module to begin recording the user's voice sample. After each successful recording, the corresponding status indicator updates to active, automatically preparing for the next recording, until all five voice samples are collected. Thus, through clear visual guidance and real-time status feedback, the user can accurately and efficiently complete the voiceprint registration process.

[0606] In some embodiments of this application, the aforementioned voice input may include any of the following: user touch input, voice input, gesture input, folding or unfolding operation of the foldable screen, or other feasible input methods. This application does not limit the types of input methods used. Further, the aforementioned touch input may include: user click input, swipe input, press input, etc. Further, the aforementioned click operation may be any number of clicks. The aforementioned swipe operation may be a swipe operation in any direction, such as swiping up, swiping down, swiping left, or swiping right, etc., and this application does not limit the types of swipe operations used.

[0607] For example, in conjunction with the above Figure 13A Following the on-screen prompts, the voiceprint subject clicked the record button to begin the first voice recording. For example, the voiceprint subject, Xiao Li, read "Hello" to record his voiceprint information.

[0608] In some embodiments of this application, the information display device responds to user input by activating an audio acquisition module to record the user's voice reading the target wake-up keyword. This process is repeated until a preset number of voice samples, such as five human voice samples, are collected.

[0609] For example, in conjunction with the above example, after each time Xiao Li clicks the record button, he reads "Hello." The information display device collects five segments of Xiao Li's "Hello" voice data. At this time, all five parallel circular status indicators 83 in the voiceprint recording window 82 are active, indicating that all five segments of voice data have been collected. Figure 13B As shown.

[0610] In this embodiment of the application, a multi-layered voiceprint registration mechanism is constructed by continuing to acquire user-recorded human voice samples when the quality verification of the wake-up word voice sample fails to meet the requirements, thereby ensuring the quality of the registered voiceprint feature information.

[0611] In some embodiments of this application, the process of performing voiceprint registration on the target wake-up keyword based on M personal voice samples in step 219c2 above may include the following steps 219d4 to 219d6:

[0612] Step 219d4: The information display device extracts the voiceprint feature information of M personal voice samples to obtain M voiceprint feature information.

[0613] Step 219d5: The information display device calculates the similarity between the M voiceprint feature information.

[0614] Step 219d6: When the similarity between any two voiceprint feature information in the M voiceprint feature information is greater than the third similarity threshold, the information display device performs voiceprint registration for the target wake-up keyword based on the M personal voice samples.

[0615] In some embodiments of this application, the information display device extracts voiceprint features from the collected M personal voice samples to obtain a new set of voiceprint feature information, such as a voiceprint feature vector.

[0616] For example, the information display device extracts five voiceprint feature vectors V1, V2, V3, V4, and V5 from the user's voice reading the number string "hello" five times.

[0617] In some embodiments of this application, the information display device calculates the similarity between the aforementioned voiceprint feature information and performs a similarity judgment again. If the consistency check is passed, the voiceprint feature information is used to complete the voiceprint registration.

[0618] It should be noted that the third similarity threshold can be the same as the second similarity threshold, or it can be set to different values ​​according to the characteristics of the custom text. For example, the third similarity threshold can be 0.8, 0.85, or 0.9, etc.

[0619] It should be noted that the threshold values ​​involved in the embodiments of this application can be set according to actual needs, and the embodiments of this application do not limit them.

[0620] For example, in conjunction with the above example, the information display device calculates the similarity between five voiceprint feature vectors V1, V2, V3, V4, and V5. For instance, it calculates the similarity between V1 and V2, V1 and V3, ..., and V4 and V5 respectively, and then takes the arithmetic mean of the above similarities as the final similarity. If the calculated average similarity reaches 0.88, exceeding the preset quality threshold of 0.85, it is determined that the voiceprint sample has sufficient consistency and stability, confirming successful voiceprint registration. The five feature vectors are then fused to generate a fused voiceprint feature information and stored in the database. If the average similarity does not reach the threshold, a message "Voiceprint registration failed, please re-enter" is displayed, and a new round of voice acquisition process is initiated.

[0621] In this embodiment of the application, by verifying multiple voice samples entered by the user, the quality and consistency of the registered voiceprint feature information are ensured, thereby improving the accuracy and reliability of voiceprint object identification.

[0622] The registration phase process of the information display method provided in this application embodiment is illustrated below with specific examples.

[0623] For example, such as Figure 14A As shown, the registration process includes the following steps:

[0624] Step 11: Receive wake-up keyword settings input.

[0625] For example, the information display device receives wake-up keywords input by the user through the user interface to define the wake-up keyword content to be registered.

[0626] For example, in a real-life scenario, a user enters "Xiaoming" as a wake-up keyword.

[0627] Step 12: Receive the pronunciation selection input of the wake-up keyword.

[0628] For example, the information display device displays alternative pronunciation information and receives user selection input based on the text content of the wake-up keyword to determine the standard pronunciation of the wake-up keyword.

[0629] Step 13: Determine if voiceprint registration is required.

[0630] For example, the information display device decides whether to continue the voiceprint registration process based on user input; if not, it proceeds to step 14; if yes, it proceeds to step 15.

[0631] Step 14: Confirm that the wake-up keyword registration was successful.

[0632] For example, if the voiceprint is not registered, the wake word registration is confirmed to be successful, and the current process ends.

[0633] For example, if a user only sets the wake word "Xiaoming" without associating it with a voiceprint, the wake word registration is complete.

[0634] Step 15: Collect multiple wake word speech samples with specified pronunciations.

[0635] For example, the information display device guides the user to record multiple segments of wake-up word voice data that conform to a specified pronunciation through a voice input interface.

[0636] For example, the user recorded three voice samples for the wake word "Xiaoming".

[0637] Step 16: Perform wake word recognition processing using a wake-up model.

[0638] For example, the information display device inputs the collected voice samples into the wake-up model, and based on acoustic feature extraction and streaming decoding, matches the wake-up word pronunciation sequence to generate a recognition result.

[0639] For example, the wake-up model performs streaming recognition on the speech sample of "Xiaoming" to verify the accuracy of pronunciation.

[0640] Step 17: Determine if the wake-up was successful.

[0641] For example, the information display device confirms whether the wake word has been successfully recognized based on the output of the wake-up model; if not, it can return to collect samples again; if yes, it executes step 18.

[0642] Step 18: Confirm that the voiceprint wake-up word registration was successful.

[0643] For example, the information display device confirms that the voiceprint wake-up word has been successfully registered and associates the wake-up word "Xiaoming" with the voiceprint object.

[0644] Step 19: Determine whether the voiceprint similarity is greater than the similarity threshold.

[0645] For example, if the similarity between the voiceprint feature information of multiple wake word speech samples is greater than the similarity threshold, then step 20 is executed; if it is less than the threshold, then step 21 is executed.

[0646] Step 20: Confirm successful voiceprint registration.

[0647] Step 21: Collect multiple human voice samples.

[0648] For example, the user is guided to record multiple voice data segments through the voice input interface for voiceprint feature extraction. When the voiceprint similarity meets the threshold requirement, the voiceprint registration is confirmed to be successful, and the voiceprint information is associated with the wake-up keyword and voiceprint object information.

[0649] The following specific examples illustrate the identification phase process of the information display method provided in this application.

[0650] For example, such as Figure 14B As shown, the identification phase process includes the following steps:

[0651] Step 31: Collect audio data.

[0652] For example, the information display device continuously collects raw audio data from the environment through an audio acquisition device.

[0653] Step 32: Determine whether speech is present by using VAD detection.

[0654] For example, the information display device performs voice activity detection on the collected audio data to determine whether there is human voice audio data. If yes, proceed to step 33; otherwise, return to step 32 to continue monitoring.

[0655] Step 33: Extract the acoustic features of the human voice audio data.

[0656] For example, the information display device extracts a T×80-dimensional acoustic feature sequence from the detected human voice audio data.

[0657] Step 34: Perform wake word recognition processing using a wake-up model.

[0658] For example, the acoustic feature sequence is input into the wake-up model, and the confidence score is obtained by streaming decoding through the feature extraction module and the decoding module.

[0659] Step 35: Determine whether the confidence score is greater than the threshold.

[0660] For example, the information display device calculates a confidence score based on the matching result of the decoded path sequence and the wake-up word pronunciation sequence and compares it with a preset confidence threshold.

[0661] For example, if the confidence score is not greater than the threshold, then step 36 is executed; if the confidence score is greater than the threshold, then step 37 is executed.

[0662] Step 36: Determine if no wake-up keyword was identified.

[0663] For example, when the confidence score is less than the threshold, it is determined that no wake-up keyword has been identified, and the monitoring status is returned.

[0664] Step 37: Confirm that the wake-up keyword has been identified.

[0665] For example, when a matching target decoding path sequence exists and the confidence score is greater than a threshold, an indication that a wake-up keyword has been identified is generated, and the end time point corresponding to the wake-up keyword on the timeline of the audio data is determined.

[0666] Step 38: Obtain the first audio segment and the second audio segment, and generate a spliced ​​audio segment.

[0667] For example, the information display device obtains the first audio segment and the second audio segment based on the wake-up timestamp, and splices them together in chronological order to obtain the spliced ​​audio segment.

[0668] Step 39: Process the spliced ​​audio segments using a speech recognition model.

[0669] Step 40: Perform voiceprint recognition on the spliced ​​audio segments using a voiceprint model.

[0670] For example, spliced ​​audio segments are input into a voiceprint recognition model for voiceprint feature extraction and similarity calculation.

[0671] Step 41: Use the dry sound separation model to identify and separate ambient sounds from the spliced ​​audio segments.

[0672] For example, the spliced ​​audio segments are input into the dry sound separation model to separate and obtain ambient sound data.

[0673] Step 42: Classify and identify ambient sound data using an acoustic scene classification model.

[0674] For example, ambient sound data is input into an acoustic scene classification model to obtain ambient sound category information.

[0675] Step 43: Output the first information.

[0676] For example, the information display device generates first information based on the speech content description information, voiceprint object information, and ambient sound category information, and outputs corresponding prompt information.

[0677] The above-described method embodiments, or various possible implementations of the method embodiments, can be executed individually, or, provided there are no contradictions, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.

[0678] The information display method provided in this application can be executed by an information display device. This application uses an information display device executing the information display method as an example to illustrate the information display device provided in this application.

[0679] Figure 15 This is a schematic diagram of the structure of the information display device provided in the embodiments of this application, such as... Figure 15 As shown, the information display device 600 may include a processing module 601 and a display module 602, wherein: the processing module 601 is used to acquire human voice audio data from audio data collected by a microphone; the display module 602 is used to display first information when the human voice audio data includes a wake-up keyword; wherein the first information includes at least one of the following: speech content description information; speaker identity information; environmental sound event information; second information; the speech content description information is used to indicate the speech content or a summary of the speech content corresponding to the human voice audio data; the speaker identity information is used to indicate the identity of the speaker corresponding to the human voice audio data; the environmental sound event information is used to indicate the sound events occurring in the environment where the human voice audio data was collected; the second information is used to prompt the speaker to speak.

[0680] In some embodiments of this application, the receiving module is configured to receive a first input to the wake word recognition function control;

[0681] The processing module is also used to respond to the first input received by the receiving module, enable the wake-up word recognition function, and monitor whether the audio data collected by the microphone includes the wake-up keyword;

[0682] The display module is specifically used to display secondary information and speech content description information when human voice audio data, including wake-up keywords, is detected;

[0683] The first information includes the second information and the description of the spoken content.

[0684] In some embodiments of this application, the receiving module is configured to receive a second input to the wake-word recognition function control and the voiceprint recognition control;

[0685] The processing module is also used to respond to the second input received by the receiving module, enable the wake-up word recognition function and the voiceprint recognition function, monitor whether the audio data collected by the microphone includes the wake-up keyword, and monitor the voiceprint information;

[0686] The display module is specifically used to display the first information when the wake-up word recognition function and voiceprint recognition function are enabled and the human voice audio data includes the wake-up keyword.

[0687] The first piece of information is any one of the following:

[0688] Second information and speaker's identity information;

[0689] Secondary information, description of the content of the speech, and speaker's identity information;

[0690] Information describing the content of the speech and information about the speaker's identity.

[0691] In some embodiments of this application, the receiving module is configured to receive third input to the wake-word recognition function control and the environmental sound event recognition control;

[0692] The processing module is also used to respond to the third input received by the receiving module, enable the wake-up word recognition function and the environmental sound event recognition function, monitor whether the audio data collected by the microphone includes the wake-up keyword, and monitor environmental sound events;

[0693] The display module is specifically used to display the first information when the wake word recognition function and the environmental event recognition function are enabled, and when human voice audio data including wake words is detected.

[0694] The first piece of information is any one of the following:

[0695] Secondary information and environmental sound event information;

[0696] Secondary information includes information describing the content of speech and information about environmental sound events;

[0697] Information describing the spoken content and information about environmental sound events.

[0698] In some embodiments of this application, the processing module is configured to acquire a first audio segment and a second audio segment when the human voice audio data includes a wake-up keyword; the first audio segment is audio data collected in a first time period before the wake-up timestamp, and the second audio segment is audio data collected in a second time period after the wake-up timestamp;

[0699] The processing module is also used to splice the first audio segment and the second audio segment to obtain a spliced ​​audio segment;

[0700] The display module is specifically used to display the first information based on the spliced ​​audio segments;

[0701] The first time period and the second time period satisfy any one of the following:

[0702] The first and second time periods are pre-set time periods;

[0703] The first duration of the first time period is the third duration corresponding to the first event type, and the second duration of the second time period is the fourth duration corresponding to the first event type;

[0704] The wake-up timestamp is determined based on any of the following:

[0705] The end time of the wake-up keyword in the audio data;

[0706] The start time of the wake-up keyword in the audio data;

[0707] The moment when the wake-up keyword is detected;

[0708] The first event type is the event type corresponding to human voice audio data.

[0709] In some embodiments of this application, when both the wake word recognition function and the environmental sound event recognition function are enabled, the fifth duration of the first time period is longer than the sixth duration, and the seventh duration of the second time period is longer than the eighth duration; wherein, the sixth duration is the duration of the first time period when both the wake word recognition function and the voiceprint recognition function are enabled; and the eighth duration is the duration of the second time period when both the wake word recognition function and the voiceprint recognition function are enabled.

[0710] With both wake word recognition and voiceprint recognition functions enabled, the ninth duration of the first time period is longer than the tenth duration, and the eleventh duration of the second time period is longer than the twelfth duration. The tenth duration is the duration of the first time period with only wake word recognition function enabled, and the twelfth duration is the duration of the second time period with only wake word recognition function enabled.

[0711] When the event type corresponding to the human voice audio data is an environmental sound event, the thirteenth duration of the first time period is longer than the fourteenth duration, the fifteenth duration of the second time period is longer than the sixteenth duration, the fourteenth duration is the duration of the first time period when the event type corresponding to the human voice audio data is a speaking event, and the sixteenth duration is the duration of the second time period when the event type corresponding to the human voice audio data is a speaking event.

[0712] In some embodiments of this application, the processing module is used for:

[0713] Human voice audio data is input into a wake-up model. The wake-up model then performs wake-up word recognition processing on the human voice audio data to obtain a recognition result. The recognition result is either a first indication or a second indication. The first indication indicates that a wake-up keyword has been recognized. The second indication indicates that a wake-up keyword has not been recognized.

[0714] If the recognition result is the first indication information, determine the wake-up timestamp;

[0715] The wake-up timestamp includes at least one of the following:

[0716] The end time corresponding to the wake-up keyword on the timeline of the audio data;

[0717] The start time of the wake-up keyword on the timeline of the audio data;

[0718] The system time at the time of the first indication is output by waking up the model.

[0719] In some embodiments of this application, the wake-up model includes a feature extraction module, an acoustic prediction module, and a decoding module; the processing module is specifically used for:

[0720] The acoustic feature sequence is obtained by extracting acoustic features from human voice audio data through the feature extraction module.

[0721] The acoustic prediction module predicts the acoustic feature sequence to obtain the acoustic prediction matrix, which includes the probability distribution prediction information of the pronunciation sequence of human voice audio data.

[0722] The acoustic prediction information is decoded by the decoding module to obtain at least one decoding path sequence.

[0723] The decoding module performs pronunciation sequence matching between at least one decoding path sequence and a reference pronunciation sequence to obtain a matching result, and outputs the recognition result based on the matching result.

[0724] The reference pronunciation sequence is the pronunciation sequence of the wake-up keyword.

[0725] In some embodiments of this application, the processing module is specifically used for:

[0726] If at least one decoding path sequence exists that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is greater than the confidence threshold, the wake-up model outputs a first recognition result; the first recognition result is the first indication information.

[0727] Alternatively, if there exists a second decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is less than the confidence threshold, the wake-up model outputs a second recognition result, which is a second indication information.

[0728] Alternatively, if there is no third decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, the wake-up model outputs a third recognition result, which includes the second indication information.

[0729] In some embodiments of this application, the prompting information includes descriptive information about the spoken content;

[0730] The processing module is specifically used to input the spliced ​​audio segment into the speech recognition model, and through the speech recognition model, to perform speech-to-text processing on the spliced ​​audio segment to generate a continuous text sequence including wake-up keywords;

[0731] The processing module is specifically used to generate descriptive information of spoken content based on a continuous text sequence using a speech recognition model;

[0732] The display module is specifically used to display descriptive information about the spoken content.

[0733] In some embodiments of this application, the prompt information includes the speaker's identity information corresponding to the human voice audio data; the processing module is specifically used to input the spliced ​​audio segment into the voiceprint recognition model, and extract the voiceprint features from the spliced ​​audio segment through the voiceprint recognition model to obtain voiceprint feature information;

[0734] The processing module is specifically used to calculate the similarity between voiceprint feature information and at least one preset voiceprint feature information through a voiceprint recognition model to obtain similarity information.

[0735] The processing module is specifically used to determine the speaker identity information corresponding to the first voiceprint feature information as the speaker identity information corresponding to the human voice audio data.

[0736] The display module is specifically used to display the speaker's identity information;

[0737] The first voiceprint feature information is a voiceprint feature information that has a similarity greater than a first similarity threshold among at least one preset voiceprint feature information.

[0738] In some embodiments of this application, the first information includes environmental sound event information; the processing module is specifically used to input the spliced ​​audio segment into the dry sound separation model, and to separate the environmental sound data from the spliced ​​audio segment through the dry sound separation model;

[0739] The processing module is specifically used to input ambient sound data into the acoustic scene classification model, and to perform acoustic scene classification and recognition on the ambient sound data through the acoustic scene classification model to obtain ambient sound category information;

[0740] The processing module is specifically used to generate environmental sound event information based on environmental sound category information using an acoustic scene classification model;

[0741] The display module is specifically used to display environmental sound event information.

[0742] In some embodiments of this application, the receiving module is used to receive wake word setting input in the wake word setting interface;

[0743] The display module is also used to respond to the wake word setting input and display the wake keyword set in the wake word setting interface.

[0744] In some embodiments of this application, the display module is further configured to, in the case that the wake-up keyword includes a polyphonic character, display at least one alternative pronunciation option for the polyphonic character in the wake-up keyword in the wake-up word setting interface according to the text content of the wake-up keyword;

[0745] The receiving module is also configured to receive a selection input for a target alternative pronunciation option among at least one alternative pronunciation options;

[0746] The processing module is also used to determine the pronunciation corresponding to the target alternative pronunciation option as the pronunciation of the polyphonic character in response to the selection input.

[0747] In some embodiments of this application, the display module is further configured to display a wake-up word management interface, which includes at least one wake-up keyword and at least one voiceprint addition option, with one wake-up keyword corresponding to one voiceprint addition option;

[0748] The receiving module is also configured to receive a first input for adding options to the voiceprint corresponding to the target wake-up keyword in at least one wake-up keyword;

[0749] The display module is also used to respond to the first input and display the voiceprint object settings interface corresponding to the target wake-up keyword;

[0750] The receiving module is also used to receive voiceprint object information setting input in the voiceprint object setting interface;

[0751] The display module is also used to respond to the voiceprint object information setting input and display the voiceprint object information set in the voiceprint object setting interface; the voiceprint object information is used to obtain the speaker's identity information when the wake-up keyword is recognized.

[0752] The processing module is also used to acquire the wake-up word voice data corresponding to the target wake-up keyword;

[0753] The processing module is also used to perform voiceprint registration on the target wake-up keyword based on the wake-up word voice data, and to establish the association between the voiceprint information of the target wake-up keyword and the voiceprint object information.

[0754] In some embodiments of this application, the voice data includes N wake word voice samples;

[0755] The display module is also used to display the wake word voiceprint recording interface. The wake word voiceprint recording interface includes a first recording prompt message and a first voice input control. The first recording prompt message is used to prompt the user to record a wake word voice sample and the number of times the wake word voice sample is recorded.

[0756] The receiving module is also used to receive voice input to the first voice input control;

[0757] The processing module is specifically used to respond to voice input by collecting N wake-up word voice samples, where N is an integer greater than 1.

[0758] In some embodiments of this application, the processing module is specifically used for:

[0759] Voiceprint features are extracted from N wake-up word speech samples to obtain N voiceprint feature information;

[0760] Calculate the similarity between N voiceprint feature information;

[0761] If the similarity between any two voiceprint features in N voiceprint features is greater than the second similarity threshold, then voiceprint registration is performed on the target wake-up keyword based on the N voiceprint features.

[0762] Alternatively, if the similarity between any two voiceprint features among the N voiceprint features is less than the second similarity threshold, obtain M personal voice samples, and perform voiceprint registration for the target wake-up keyword based on the M personal voice samples, where M is greater than N.

[0763] It should be noted that the above-mentioned information display device can achieve the same effect as the above-mentioned information display method. Its effect can be found in the effect description of the above-mentioned information display method. To avoid repetition, it will not be described again here.

[0764] The information display device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0765] The information display device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0766] The information display device provided in this application embodiment can realize the various processes implemented in the information display method embodiment, and will not be described again here to avoid repetition.

[0767] Optionally, such as Figure 16 As shown, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores a program or instructions that can run on the processor 701. When the program or instructions are executed by the processor 701, they implement the various steps of the above-described information display method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0768] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0769] Figure 17 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0770] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0771] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 17 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0772] The processor 110 is used to acquire human voice audio data from audio data collected by the microphone; the display unit 106 is used to display first information when the human voice audio data includes a wake-up keyword; wherein the first information includes at least one of the following: speech content description information; speaker identity information; environmental sound event information; and second information; the speech content description information is used to indicate the speech content or a summary of the speech content corresponding to the human voice audio data; the speaker identity information is used to indicate the identity of the speaker corresponding to the human voice audio data; the environmental sound event information is used to indicate the sound events occurring in the environment in which the human voice audio data is collected; and the second information is used to prompt the speaker to speak.

[0773] In some embodiments of this application, the user input unit 107 is used to receive a first input to the wake word recognition function control;

[0774] The processor 110 is also configured to, in response to the first input received by the user input unit 107, enable the wake-up word recognition function and monitor whether the audio data collected by the microphone includes the wake-up keyword;

[0775] The display unit 106 is specifically used to display second information and speech content description information when human voice audio data including wake-up keywords are detected;

[0776] The first information includes the second information and the description of the spoken content.

[0777] In some embodiments of this application, the user input unit 107 is used to receive a second input to the wake word recognition function control and the voiceprint recognition control;

[0778] The processor 110 is also configured to respond to a second input received by the user input unit 107, enable the wake-up word recognition function and the voiceprint recognition function, monitor whether the audio data collected by the microphone includes a wake-up keyword, and monitor the voiceprint information;

[0779] The display unit 106 is specifically used to display first information when the wake-up word recognition function and the voiceprint recognition function are enabled and the human voice audio data includes the wake-up keyword.

[0780] The first piece of information is any one of the following:

[0781] Second information and speaker's identity information;

[0782] Secondary information, description of the content of the speech, and speaker's identity information;

[0783] Information describing the content of the speech and information about the speaker's identity.

[0784] In some embodiments of this application, the user input unit 107 is used to receive third input to the wake word recognition function control and the environmental sound event recognition control;

[0785] The processor 110 is also configured to respond to a third input received by the user input unit 107, enable the wake-up word recognition function and the environmental sound event recognition function, monitor whether the audio data collected by the microphone includes a wake-up keyword, and monitor environmental sound events.

[0786] The display unit 106 is specifically used to display first information when the wake-up word recognition function and the environmental event recognition function are enabled and the human voice audio data including the wake-up keyword is detected.

[0787] The first piece of information is any one of the following:

[0788] Secondary information and environmental sound event information;

[0789] Secondary information includes information describing the content of speech and information about environmental sound events;

[0790] Information describing the spoken content and information about environmental sound events.

[0791] In some embodiments of this application, the processor 110 is configured to acquire a first audio segment and a second audio segment when the human voice audio data includes a wake-up keyword; the first audio segment is audio data collected in a first time period before the wake-up timestamp, and the second audio segment is audio data collected in a second time period after the wake-up timestamp.

[0792] The processor 110 is also used to splice the first audio segment and the second audio segment to obtain a spliced ​​audio segment;

[0793] Display unit 106 is specifically used to display first information based on spliced ​​audio segments;

[0794] The first time period and the second time period satisfy any one of the following:

[0795] The first and second time periods are pre-set time periods;

[0796] The first duration of the first time period is the third duration corresponding to the first event type, and the second duration of the second time period is the fourth duration corresponding to the first event type;

[0797] The wake-up timestamp is determined based on any of the following:

[0798] The end time of the wake-up keyword in the audio data;

[0799] The start time of the wake-up keyword in the audio data;

[0800] The moment when the wake-up keyword is detected;

[0801] The first event type is the event type corresponding to human voice audio data.

[0802] In some embodiments of this application, when both the wake word recognition function and the environmental sound event recognition function are enabled, the fifth duration of the first time period is longer than the sixth duration, and the seventh duration of the second time period is longer than the eighth duration; wherein, the sixth duration is the duration of the first time period when both the wake word recognition function and the voiceprint recognition function are enabled; and the eighth duration is the duration of the second time period when both the wake word recognition function and the voiceprint recognition function are enabled.

[0803] With both wake word recognition and voiceprint recognition functions enabled, the ninth duration of the first time period is longer than the tenth duration, and the eleventh duration of the second time period is longer than the twelfth duration. The tenth duration is the duration of the first time period with only wake word recognition function enabled, and the twelfth duration is the duration of the second time period with only wake word recognition function enabled.

[0804] When the event type corresponding to the human voice audio data is an environmental sound event, the thirteenth duration of the first time period is longer than the fourteenth duration, the fifteenth duration of the second time period is longer than the sixteenth duration, the fourteenth duration is the duration of the first time period when the event type corresponding to the human voice audio data is a speaking event, and the sixteenth duration is the duration of the second time period when the event type corresponding to the human voice audio data is a speaking event.

[0805] In some embodiments of this application, processor 110 is used for:

[0806] Human voice audio data is input into a wake-up model. The wake-up model then performs wake-up word recognition processing on the human voice audio data to obtain a recognition result. The recognition result is either a first indication or a second indication. The first indication indicates that a wake-up keyword has been recognized. The second indication indicates that a wake-up keyword has not been recognized.

[0807] If the recognition result is the first indication information, determine the wake-up timestamp;

[0808] The wake-up timestamp includes at least one of the following:

[0809] The end time corresponding to the wake-up keyword on the timeline of the audio data;

[0810] The start time of the wake-up keyword on the timeline of the audio data;

[0811] The system time at the time of the first indication is output by waking up the model.

[0812] In some embodiments of this application, the wake-up model includes a feature extraction module, an acoustic prediction module, and a decoding module; the processor 110 is specifically used for:

[0813] The acoustic feature sequence is obtained by extracting acoustic features from human voice audio data through the feature extraction module.

[0814] The acoustic prediction module predicts the acoustic feature sequence to obtain the acoustic prediction matrix, which includes the probability distribution prediction information of the pronunciation sequence of human voice audio data.

[0815] The acoustic prediction information is decoded by the decoding module to obtain at least one decoding path sequence.

[0816] The decoding module performs pronunciation sequence matching between at least one decoding path sequence and a reference pronunciation sequence to obtain a matching result, and outputs the recognition result based on the matching result.

[0817] The reference pronunciation sequence is the pronunciation sequence of the wake-up keyword.

[0818] In some embodiments of this application, the processor 110 is specifically used for:

[0819] If at least one decoding path sequence exists that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is greater than the confidence threshold, the wake-up model outputs a first recognition result; the first recognition result is the first indication information.

[0820] Alternatively, if there exists a second decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is less than the confidence threshold, the wake-up model outputs a second recognition result, which is a second indication information.

[0821] Alternatively, if there is no third decoding path sequence in at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, the wake-up model outputs a third recognition result, which includes the second indication information.

[0822] In some embodiments of this application, the prompting information includes descriptive information about the spoken content;

[0823] The processor 110 is specifically used to input the spliced ​​audio segment into the speech recognition model, and through the speech recognition model, to perform speech transcription processing on the spliced ​​audio segment to generate a continuous text sequence including wake-up keywords;

[0824] The processor 110 is specifically used to generate spoken content description information based on a continuous text sequence through a speech recognition model;

[0825] Display unit 106 is specifically used to display descriptive information about the spoken content.

[0826] In some embodiments of this application, the prompt information includes the speaker's identity information corresponding to the human voice audio data; the processor 110 is specifically used to input the spliced ​​audio segment into the voiceprint recognition model, and extract voiceprint features from the spliced ​​audio segment through the voiceprint recognition model to obtain voiceprint feature information.

[0827] The processor 110 is specifically used to calculate the similarity between voiceprint feature information and at least one preset voiceprint feature information through a voiceprint recognition model to obtain similarity information.

[0828] The processor 110 is specifically used to determine the speaker identity information corresponding to the first voiceprint feature information as the speaker identity information corresponding to the human voice audio data.

[0829] Display unit 106 is specifically used to display speaker identity information;

[0830] The first voiceprint feature information is a voiceprint feature information that has a similarity greater than a first similarity threshold among at least one preset voiceprint feature information.

[0831] In some embodiments of this application, the first information includes ambient sound event information; the processor 110 is specifically used to input the spliced ​​audio segment into the dry sound separation model, and to separate ambient sound data from the spliced ​​audio segment through the dry sound separation model;

[0832] The processor 110 is specifically used to input ambient sound data into the acoustic scene classification model, and to perform acoustic scene classification and recognition on the ambient sound data through the acoustic scene classification model to obtain ambient sound category information;

[0833] The processor 110 is specifically used to generate environmental sound event information based on environmental sound category information through an acoustic scene classification model;

[0834] Display unit 106 is specifically used to display environmental sound event information.

[0835] In some embodiments of this application, the user input unit 107 is used to receive wake word setting input in the wake word setting interface;

[0836] The display unit 106 is also used to display the wake-up keyword set in the wake-up word setting interface in response to the wake-up word setting input.

[0837] In some embodiments of this application, the display unit 106 is also configured to, when the wake-up keyword includes a polyphonic character, display at least one alternative pronunciation option for the polyphonic character in the wake-up keyword in the wake-up word setting interface according to the text content of the wake-up keyword;

[0838] User input unit 107 is also configured to receive selection input for a target alternative pronunciation option among at least one alternative pronunciation options;

[0839] The processor 110 is also configured to, in response to the selection input, determine the pronunciation corresponding to the target alternative pronunciation option as the pronunciation of a polyphonic character.

[0840] In some embodiments of this application, the display unit 106 is also used to display a wake-up word management interface, which includes at least one wake-up keyword and at least one voiceprint addition option, with one wake-up keyword corresponding to one voiceprint addition option;

[0841] User input unit 107 is also configured to receive a first input for adding an option to the voiceprint corresponding to the target wake-up keyword in at least one wake-up keyword;

[0842] The display unit 106 is also used to respond to the first input by displaying the voiceprint object setting interface corresponding to the target wake-up keyword;

[0843] The user input unit 107 is also used to receive voiceprint object information setting input in the voiceprint object setting interface;

[0844] The display unit 106 is also used to respond to the voiceprint object information setting input and display the voiceprint object information set in the voiceprint object setting interface; the voiceprint object information is used to obtain the speaker's identity information when the wake-up keyword is recognized.

[0845] The processor 110 is also used to acquire the wake-up word voice data corresponding to the target wake-up keyword;

[0846] The processor 110 is also used to perform voiceprint registration on the target wake-up keyword based on the wake-up word voice data, and to establish the association between the voiceprint information of the target wake-up keyword and the voiceprint object information.

[0847] In some embodiments of this application, the voice data includes N wake word voice samples;

[0848] The display unit 106 is also used to display the wake word voiceprint recording interface. The wake word voiceprint recording interface includes a first recording prompt message and a first voice input control. The first recording prompt message is used to prompt the user to record a wake word voice sample and the number of times the wake word voice sample is recorded.

[0849] The user input unit 107 is also used to receive voice input to the first voice input control;

[0850] The processor 110 is specifically used to respond to voice input by acquiring N wake-up word voice samples, where N is an integer greater than 1.

[0851] In some embodiments of this application, the processor 110 is specifically used for:

[0852] Voiceprint features are extracted from N wake-up word speech samples to obtain N voiceprint feature information;

[0853] Calculate the similarity between N voiceprint feature information;

[0854] If the similarity between any two voiceprint features in N voiceprint features is greater than the second similarity threshold, then voiceprint registration is performed on the target wake-up keyword based on the N voiceprint features.

[0855] Alternatively, if the similarity between any two voiceprint features among the N voiceprint features is less than the second similarity threshold, obtain M personal voice samples, and perform voiceprint registration for the target wake-up keyword based on the M personal voice samples, where M is greater than N.

[0856] It should be noted that the above-mentioned electronic device can achieve the same effect as the above-mentioned information display method. Its effect can be found in the effect description of the above-mentioned information display method. To avoid repetition, it will not be described again here.

[0857] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0858] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0859] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0860] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described information display method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0861] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0862] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described information display method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0863] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0864] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the information display method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0865] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0866] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0867] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An information display method, characterized in that, The method includes: Human voice audio data is obtained from the audio data collected by the microphone; If the human voice audio data includes a wake-up keyword, display the first information; The first information includes at least one of the following: speech content description information; speaker identity information; environmental sound event information; and second information. The speech content description information is used to indicate the speech content corresponding to the human voice audio data or the summary information of the speech content; the speaker identity information is used to indicate the identity of the speaker corresponding to the human voice audio data; the environmental sound event information is used to indicate the sound events that occurred in the environment in which the human voice audio data was collected; the second information is used to prompt the speaker to speak.

2. The method according to claim 1, characterized in that, Before displaying the first information, the method further includes: Receive the first input to the wake word recognition function control; In response to the first input, the wake-up word recognition function is enabled, and the audio data collected by the microphone is monitored to see if it includes the wake-up keyword; When the human voice audio data includes a wake-up keyword, displaying first information includes: If the detected human voice audio data includes wake-up keywords, the second information and the speech content description information are displayed; The first information includes the second information and the description information of the spoken content.

3. The method according to claim 1, characterized in that, Before displaying the first information, the method further includes: Receive a second input to the wake word recognition control and the voiceprint recognition control; In response to the second input, the wake-up word recognition function and the voiceprint recognition function are enabled, the audio data collected by the microphone is monitored to see if the wake-up keyword is included, and the voiceprint information is monitored. When the human voice audio data includes a wake-up keyword, displaying first information includes: If the wake-up word recognition function and voiceprint recognition function are enabled, and the human voice audio data is detected to include a wake-up keyword, the first information is displayed; The first piece of information is any one of the following: Second information and speaker's identity information; Secondary information, description of the content of the speech, and speaker's identity information; Information describing the content of the speech and information about the speaker's identity.

4. The method according to claim 1, characterized in that, Before displaying the first information when the human voice audio data includes a wake-up keyword, the method further includes: Receive third input to the wake word recognition control and the ambient sound event recognition control; In response to the third input, the wake-up word recognition function and the environmental sound event recognition function are enabled, the audio data collected by the microphone is monitored to see if the wake-up keyword is included, and the environmental sound event is monitored. When the human voice audio data includes a wake-up keyword, displaying first information includes: If the wake-up word recognition function and the environmental event recognition function are enabled, and the human voice audio data is detected to include a wake-up keyword, the first information is displayed; The first piece of information is any one of the following: Secondary information and environmental sound event information; Secondary information includes information describing the content of speech and information about environmental sound events; Information describing the spoken content and information about environmental sound events.

5. The method according to claim 1, characterized in that, When the human voice audio data includes a wake-up keyword, displaying first information includes: If the human voice audio data includes a wake-up keyword, a first audio segment and a second audio segment are obtained; the first audio segment is audio data collected in a first time period before the wake-up timestamp, and the second audio segment is audio data collected in a second time period after the wake-up timestamp. The first audio segment and the second audio segment are spliced ​​together to obtain a spliced ​​audio segment; Based on the spliced ​​audio segments, display the first information; The first time period and the second time period satisfy any one of the following: The first time period and the second time period are preset time periods; The first duration of the first time period is the third duration corresponding to the first event type, and the second duration of the second time period is the fourth duration corresponding to the first event type; The wake-up timestamp is determined based on any of the following: The wake-up keyword is the end time of the audio data; The start time of the wake-up keyword in the audio data; The moment when the wake-up keyword is detected; Wherein, the first event type is the event type corresponding to the human voice audio data.

6. The method according to claim 5, characterized in that, When both the wake word recognition function and the environmental sound event recognition function are enabled, the fifth duration of the first time period is longer than the sixth duration, and the seventh duration of the second time period is longer than the eighth duration; wherein, the sixth duration is the duration of the first time period when the wake word recognition function and the voiceprint recognition function are enabled; and the eighth duration is the duration of the second time period when the wake word recognition function and the voiceprint recognition function are enabled. When both the wake word recognition function and the voiceprint recognition function are enabled, the ninth duration of the first time period is longer than the tenth duration, and the eleventh duration of the second time period is longer than the twelfth duration. The tenth duration is the duration of the first time period when only the wake word recognition function is enabled, and the twelfth duration is the duration of the second time period when only the wake word recognition function is enabled. When the event type corresponding to the human voice audio data is an environmental sound event, the thirteenth duration of the first time period is longer than the fourteenth duration, and the fifteenth duration of the second time period is longer than the sixteenth duration; wherein, the fourteenth duration is the duration of the first time period when the event type corresponding to the human voice audio data is a speaking event; and the sixteenth duration is the duration of the second time period when the event type corresponding to the human voice audio data is a speaking event.

7. The method according to any one of claims 1 to 6, characterized in that, After obtaining human voice audio data from the collected audio data, the method further includes: The human voice audio data is input into a wake-up model, and the wake-up model performs wake-up word recognition processing on the human voice audio data to obtain a recognition result; wherein, the recognition result is a first indication information or a second indication information; the first indication information is used to indicate that a wake-up keyword has been recognized; the second indication information is used to indicate that a wake-up keyword has not been recognized. If the identification result is the first indication information, determine the wake-up timestamp; The wake-up timestamp includes at least one of the following: The wake-up keyword corresponds to the end time on the timeline of the audio data; The wake-up keyword corresponds to the start time on the timeline of the audio data; The system time when the wake-up model outputs the first indication information.

8. The method according to claim 7, characterized in that, The wake-up model includes a feature extraction module, an acoustic prediction module, and a decoding module; The step of performing wake-up word recognition processing on the human voice audio data through the wake-up model to obtain the recognition result includes: The acoustic feature sequence is obtained by extracting acoustic features from the human voice audio data through the feature extraction module. The acoustic prediction module predicts the acoustic feature sequence to obtain an acoustic prediction matrix, which includes the probability distribution prediction information of the pronunciation sequence of the human voice audio data. The acoustic prediction information is decoded by the decoding module to obtain at least one decoding path sequence; The decoding module performs phonetic sequence matching between the at least one decoding path sequence and a reference pronunciation sequence to obtain a matching result, and outputs a recognition result based on the matching result. The reference pronunciation sequence is the pronunciation sequence of the wake-up keyword.

9. The method according to claim 8, characterized in that, The step of generating a recognition result based on the matching result includes: If a first decoding path sequence exists in the at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is greater than a confidence threshold, the wake-up model outputs a first recognition result; the first recognition result is the first indication information. Alternatively, if there exists a second decoding path sequence in the at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, and the confidence score of the target decoding path sequence is less than the confidence threshold, the wake-up model outputs a second recognition result, which is a second indication information. Alternatively, if there is no third decoding path sequence in the at least one decoding path sequence that matches the pronunciation sequence of the wake-up keyword, the wake-up model outputs a third recognition result, which is a second indication information.

10. The method according to claim 5, characterized in that, The prompt information includes a description of the spoken content; The first information displayed based on the spliced ​​audio segments includes: The spliced ​​audio segment is input into a speech recognition model, and the speech recognition model performs speech-to-text processing on the spliced ​​audio segment to generate a continuous text sequence including the wake-up keyword; Based on the continuous text sequence, the speech recognition model generates a description of the spoken content. Display the description information of the spoken content.

11. The method according to claim 5, characterized in that, The prompt information includes the speaker's identity information corresponding to the human voice audio data; The first information displayed based on the spliced ​​audio segments includes: The spliced ​​audio segment is input into the voiceprint recognition model, and the voiceprint recognition model is used to extract voiceprint features from the spliced ​​audio segment to obtain voiceprint feature information. The similarity information is obtained by calculating the similarity between the voiceprint feature information and at least one preset voiceprint feature information using the voiceprint recognition model. The speaker identity information corresponding to the first voiceprint feature information is determined as the speaker identity information corresponding to the human voice audio data; Display the speaker's identity information; Wherein, the first voiceprint feature information is the voiceprint feature information among the at least one preset voiceprint feature information whose similarity to the voiceprint feature information is greater than a first similarity threshold.

12. The method according to claim 5, characterized in that, The first information includes environmental sound event information; the step of displaying the first information based on the spliced ​​audio segments includes: The spliced ​​audio segment is input into the dry sound separation model, and the ambient sound data is separated from the spliced ​​audio segment through the dry sound separation model. The ambient sound data is input into the acoustic scene classification model, and the ambient sound data is classified and identified by the acoustic scene classification model to obtain ambient sound category information. Based on the environmental sound category information, environmental sound event information is generated using the acoustic scene classification model. Display the environmental sound event information.

13. The method according to claim 1, characterized in that, The method further includes: Display the wake word settings interface; Receive wake word setting input in the wake word setting interface; In response to the wake-up word setting input, the wake-up keyword set by the wake-up word setting input is displayed on the wake-up word setting interface.

14. The method according to claim 13, characterized in that, After receiving the wake-up word setting input in the wake-up word setting interface, the method further includes: When the wake-up keyword includes a polyphonic character, at least one alternative pronunciation option for the polyphonic character in the wake-up keyword is displayed in the wake-up word setting interface according to the text content of the wake-up keyword. Receive input for selecting a target alternative pronunciation option from the at least one alternative pronunciation options; In response to the selection input, the pronunciation corresponding to the target alternative pronunciation option is determined as the pronunciation of the polyphonic character.

15. The method according to claim 1 or 13, characterized in that, The method further includes: The wake word management interface is displayed. The wake word management interface includes at least one wake keyword and at least one voiceprint addition option. One wake keyword corresponds to one voiceprint addition option. Receive a sixth input for adding options to the voiceprint of the target wake-up keyword among the at least one wake-up keywords; In response to the sixth input, the voiceprint object setting interface corresponding to the target wake-up keyword is displayed; Receive voiceprint object information setting input in the voiceprint object setting interface; In response to the input of the voiceprint object information setting, the voiceprint object information set by the input of the voiceprint object setting is displayed on the voiceprint object setting interface; the voiceprint object information is used to obtain speaker identity information when the wake-up keyword is recognized; Obtain the wake-up word speech data corresponding to the target wake-up keyword; Based on the wake-up word speech data, voiceprint registration is performed on the target wake-up keyword, and an association is established between the voiceprint information of the target wake-up keyword and the voiceprint object information.

16. The method according to claim 15, characterized in that, The voice data includes N wake-word voice samples; The step of obtaining the wake-up word speech data corresponding to the target wake-up keyword includes: The wake-up word voiceprint recording interface is displayed. The wake-up word voiceprint recording interface includes a first recording prompt message and a first voice input control. The first recording prompt message is used to prompt the user to record a wake-up word voice sample and the number of times the wake-up word voice sample is recorded. Receive voice input to the first voice input control; In response to the voice input, N wake-up word voice samples are collected from the voice input, where N is an integer greater than 1.

17. The method according to claim 15, characterized in that, The step of performing voiceprint registration on the target wake-up keyword based on the voice data includes: Voiceprint features are extracted from the N wake-up word speech samples to obtain N voiceprint feature information; Calculate the similarity among the N voiceprint feature information; If the similarity between any two voiceprint feature pieces among the N voiceprint feature pieces is greater than the second similarity threshold, then voiceprint registration is performed on the target wake-up keyword based on the N voiceprint feature pieces. Alternatively, if the similarity between any two voiceprint features among the N voiceprint features is less than the second similarity threshold, obtain M personal voice samples, and perform voiceprint registration on the target wake-up keyword based on the M personal voice samples, where M is greater than N.

18. An information display device, characterized in that, The device includes: a processing module and a display module, wherein: The processing module is used to obtain human voice audio data from the audio data collected by the microphone; The display module is used to display first information when the human voice audio data includes a wake-up keyword; The first information includes at least one of the following: speech content description information; speaker identity information; environmental sound event information; and second information. The speech content description information is used to indicate the speech content corresponding to the human voice audio data or the summary information of the speech content; the speaker identity information is used to indicate the identity of the speaker corresponding to the human voice audio data; the environmental sound event information is used to indicate the sound events that occurred in the environment in which the human voice audio data was collected; the second information is used to prompt the speaker to speak.

19. The apparatus according to claim 18, characterized in that, The receiving module is used to receive the first input to the wake word recognition function control; The processing module is further configured to respond to the first input received by the receiving module, enable the wake-up word recognition function, and monitor whether the audio data collected by the microphone includes a wake-up keyword; The display module is specifically used to display second information and speech content description information when the human voice audio data includes wake-up keywords; The first information includes the second information and the description information of the spoken content.

20. The apparatus according to claim 18, characterized in that, The receiving module is used to receive second inputs to the wake-word recognition control and the voiceprint recognition control; The processing module is further configured to respond to the second input received by the receiving module, enable the wake-up word recognition function and the voiceprint recognition function, monitor whether the audio data collected by the microphone includes the wake-up keyword, and monitor the voiceprint information; The display module is specifically used to display first information when the wake-up word recognition function and the voiceprint recognition function are enabled, and when the human voice audio data is detected to include a wake-up keyword. The first piece of information is any one of the following: Second information and speaker's identity information; Secondary information, description of the content of the speech, and speaker's identity information; Information describing the content of the speech and information about the speaker's identity.

21. The apparatus according to claim 18, characterized in that, The receiving module is used to receive third inputs from the wake-word recognition control and the ambient sound event recognition control; The processing module is also configured to respond to the third input received by the receiving module, enable the wake-up word recognition function and the environmental sound event recognition function, monitor whether the audio data collected by the microphone includes the wake-up keyword, and monitor environmental sound events; The display module is specifically used to display first information when the wake-up word recognition function and the environmental event recognition function are enabled, and when the human voice audio data is detected to include wake-up keywords. The first piece of information is any one of the following: Secondary information and environmental sound event information; Secondary information includes information describing the content of speech and information about environmental sound events; Information describing the spoken content and information about environmental sound events.

22. The apparatus according to claim 18, characterized in that, The processing module is used to acquire a first audio segment and a second audio segment when the human voice audio data includes a wake-up keyword; the first audio segment is audio data collected in a first time period before the wake-up timestamp, and the second audio segment is audio data collected in a second time period after the wake-up timestamp. The processing module is further configured to splice the first audio segment and the second audio segment to obtain a spliced ​​audio segment; The display module is specifically used to display first information based on the spliced ​​audio segments; The first time period and the second time period satisfy any one of the following: The first time period and the second time period are preset time periods; The first duration of the first time period is the third duration corresponding to the first event type, and the second duration of the second time period is the fourth duration corresponding to the first event type; The wake-up timestamp is determined based on any of the following: The wake-up keyword is the end time of the audio data; The start time of the wake-up keyword in the audio data; The moment when the wake-up keyword is detected; Wherein, the first event type is the event type corresponding to the human voice audio data.

23. The apparatus according to any one of claims 18 to 22, characterized in that, Processing module, used for: The human voice audio data is input into a wake-up model, and the wake-up model performs wake-up word recognition processing on the human voice audio data to obtain a recognition result; wherein, the recognition result is a first indication information or a second indication information; the first indication information is used to indicate that a wake-up keyword has been recognized; the second indication information is used to indicate that a wake-up keyword has not been recognized. If the identification result is the first indication information, determine the wake-up timestamp; The wake-up timestamp includes at least one of the following: The wake-up keyword corresponds to the end time on the timeline of the audio data; The wake-up keyword corresponds to the start time on the timeline of the audio data; The system time when the wake-up model outputs the first indication information.

24. The apparatus according to claim 22, characterized in that, The prompt information includes a description of the spoken content; The processing module is specifically used to input the spliced ​​audio segment into the speech recognition model, and through the speech recognition model, to perform speech-to-text processing on the spliced ​​audio segment to generate a continuous text sequence including the wake-up keyword; The processing module is specifically used to generate spoken content description information based on the continuous text sequence through the speech recognition model; The display module is specifically used to display the descriptive information of the spoken content.

25. The apparatus according to claim 22, characterized in that, The prompt information includes the speaker's identity information corresponding to the human voice audio data; the processing module is specifically used to input the spliced ​​audio segment into the voiceprint recognition model, and extract voiceprint features from the spliced ​​audio segment through the voiceprint recognition model to obtain voiceprint feature information; The processing module is specifically used to calculate the similarity between the voiceprint feature information and at least one preset voiceprint feature information through the voiceprint recognition model to obtain similarity information. The processing module is specifically used to determine the speaker identity information corresponding to the first voiceprint feature information as the speaker identity information corresponding to the human voice audio data; The display module is specifically used to display the speaker's identity information; Wherein, the first voiceprint feature information is the voiceprint feature information among the at least one preset voiceprint feature information whose similarity to the voiceprint feature information is greater than a first similarity threshold.

26. The apparatus according to claim 22, characterized in that, The first information includes environmental sound event information; the processing module is specifically used to input the spliced ​​audio segment into the dry sound separation model, and through the dry sound separation model, separate the environmental sound data from the spliced ​​audio segment; The processing module is specifically used to input the ambient sound data into the acoustic scene classification model, and to perform acoustic scene classification and recognition on the ambient sound data through the acoustic scene classification model to obtain ambient sound category information; The processing module is specifically used to generate environmental sound event information based on the environmental sound category information using the acoustic scene classification model. The display module is specifically used to display the environmental sound event information.

27. The apparatus according to claim 18, characterized in that, The receiving module is used to receive the wake-up word setting input in the wake-up word setting interface; The display module is also configured to, in response to the wake-up word setting input, display the wake-up keyword set by the wake-up word setting input on the wake-up word setting interface.

28. The apparatus according to claim 27, characterized in that, The display module is further configured to, in the case that the wake-up keyword includes a polyphonic character, display at least one alternative pronunciation option for the polyphonic character in the wake-up keyword in the wake-up word setting interface according to the text content of the wake-up keyword; The receiving module is also configured to receive a selection input for a target alternative pronunciation option among the at least one alternative pronunciation options; The processing module is further configured to, in response to the selection input, determine the pronunciation corresponding to the target alternative pronunciation option as the pronunciation of the polyphonic character.

29. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the information display method as described in any one of claims 1-17.

30. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the information display method as described in any one of claims 1-17.

Images