Speech recognition method and device, human-computer interaction device and storage medium

Abstract

The application provides a speech recognition method and device, human-computer interaction equipment and a storage medium, wherein the method comprises: determining an environment video of an environment in which a to-be-recognized speech is located; performing entity extraction on a video description text of the environment video, and updating a keyword library based on an entity extraction result, and / or performing entity relationship extraction on the video description text, and updating a knowledge graph based on an entity relationship extraction result; and performing speech recognition on the to-be-recognized speech based on the updated keyword library and / or the updated knowledge graph. The speech recognition method and device, human-computer interaction equipment and storage medium provided by the application can form personalized knowledge for a user, can expand text corpus that conforms to the habit of the user or matches current environment information, thereby improving the accuracy of speech recognition, improving the success rate of speech command interaction, and improving the user experience.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus, human-computer interaction device, and storage medium. Background Technology

[0002] In voice command interaction scenarios, commands are typically transmitted via voice. This can be achieved by performing speech recognition on the speech and then combining the recognized text to understand the intent; or by directly using an end-to-end voice-to-intent approach to transmit commands.

[0003] Currently, in order to improve the accuracy of speech recognition, personalized solutions are generally used to address users' accent issues; and speech enhancement is usually achieved by combining facial information, especially lip-shape information, to address noise interference issues.

[0004] However, in service-oriented or companion-oriented voice command interaction scenarios, even with the adoption of accent-based customization or multimodal information enhancement for ambient noise speech, the speech recognition results are poor, leading to inaccurate understanding of intent, low success rate of voice command interaction, and negatively impacting user experience. Summary of the Invention

[0005] This invention provides a speech recognition method, apparatus, human-computer interaction device, and storage medium to address the shortcomings of existing technologies, such as poor speech recognition results leading to inaccurate understanding of intent, low success rate of voice command interaction, and negative impact on user experience.

[0006] This invention provides a speech recognition method, comprising:

[0007] Determine the environmental video of the environment in which the speech to be recognized is located;

[0008] Based on the video description text of the environmental video, entity extraction is performed on the video description text, and the keyword library is updated based on the entity extraction results; and / or, entity relation extraction is performed on the video description text, and the knowledge graph is updated based on the entity relation extraction results.

[0009] Speech recognition is performed on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

[0010] According to the speech recognition method provided by the present invention, the step of performing speech recognition on the speech to be recognized based on the updated keyword database and / or the updated knowledge graph includes:

[0011] Based on the updated keyword library and / or the updated knowledge graph, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

[0012] According to the speech recognition method provided by the present invention, the step of decoding the acoustic features of the speech to be recognized based on the updated keyword library and / or the updated knowledge graph to obtain the speech recognition result includes:

[0013] Based on the entity extraction results and / or entity relationship extraction results, new keywords are determined;

[0014] Text retrieval is performed based on the newly added keywords to obtain new text corpus, and the language model is updated based on the new text corpus. The language model is used to decode the acoustic features of the speech to be recognized.

[0015] Based on at least one of the updated keyword database, updated knowledge graph, and updated language model, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

[0016] According to the speech recognition method provided by the present invention, the step of decoding the acoustic features of the speech to be recognized based on the updated keyword library to obtain the speech recognition result includes:

[0017] Based on the time tags of each keyword in the updated keyword library, determine the hot keyword incentive weight for each keyword;

[0018] Based on the hot word incentive weights of each keyword, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

[0019] According to the speech recognition method provided by the present invention, the step of decoding the acoustic features of the speech to be recognized based on the updated knowledge graph to obtain the speech recognition result includes:

[0020] The updated knowledge graph features are fused with the encoded features of the decoding result from the previous decoding time to obtain the fused features for the current decoding time.

[0021] Based on the fusion features at the current decoding moment and the decoding result at the previous decoding moment, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result, which is the decoding result at the final decoding moment.

[0022] According to the speech recognition method provided by the present invention, the step of extracting entities from the video description text based on the environmental video and updating the keyword library based on the entity extraction results includes:

[0023] Based on the video description text, entity extraction is performed on each statement in the video description text to obtain the keywords contained in each statement;

[0024] If the keyword is not included in the keyword library, the keyword is added to the keyword library to obtain an updated keyword library.

[0025] According to the speech recognition method provided by the present invention, the step of extracting entity relations from the video description text and updating the knowledge graph based on the entity relation extraction results includes:

[0026] Entity relations are extracted from each statement in the video description text to obtain knowledge information for each statement, including entities, entity attributes, and attribute values.

[0027] If the knowledge information is not present in the knowledge graph, the knowledge information is added to the knowledge graph to obtain an updated knowledge graph.

[0028] The present invention also provides a voice recognition device, comprising:

[0029] An environmental video determination unit is used to determine the environmental video of the environment in which the speech to be recognized is located.

[0030] The update unit is used to extract entities from the video description text based on the environmental video, update the keyword library based on the entity extraction results, and / or extract entity relations from the video description text, update the knowledge graph based on the entity relation extraction results.

[0031] The speech recognition unit is used to perform speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

[0032] The present invention also provides a human-computer interaction device, comprising a camera, a microphone and a processor connected in sequence;

[0033] The microphone is used to acquire the speech to be recognized;

[0034] The camera is used to acquire environmental video of the environment in which the voice to be recognized is located, and to transmit the environmental video to the processor;

[0035] The processor is configured to perform entity extraction on the video description text based on the environmental video, and update the keyword library based on the entity extraction results, and / or perform entity relation extraction on the video description text, and update the knowledge graph based on the entity relation extraction results; perform speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph, and perform human-computer interaction based on the speech recognition results.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech recognition methods.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described above.

[0039] The speech recognition method, apparatus, human-computer interaction device, and storage medium provided by this invention extract entities from the video description text of environmental videos and update the keyword library based on the entity extraction results, and / or extract entity relationships from the video description text and update the knowledge graph based on the entity relationship extraction results. Based on this, speech recognition is performed on the speech to be recognized using the keyword library and / or knowledge graph, which can form personalized knowledge for the user, expand the text corpus to conform to user habits or match current environmental information, thereby improving the accuracy of speech recognition, increasing the success rate of voice command interaction, and improving the user experience. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts of the speech recognition method provided by the present invention;

[0042] Figure 2 This is the second flowchart of the speech recognition method provided by the present invention;

[0043] Figure 3 This is the third flowchart of the speech recognition method provided by the present invention;

[0044] Figure 4 This is the fourth flowchart of the speech recognition method provided by the present invention;

[0045] Figure 5 This is one of the flowcharts of step 120 in the speech recognition method provided by the present invention;

[0046] Figure 6This is the second flowchart of step 120 in the speech recognition method provided by the present invention;

[0047] Figure 7 This is the fifth flowchart of the speech recognition method provided by the present invention;

[0048] Figure 8 This is a schematic diagram of the structure of the speech recognition device provided by the present invention;

[0049] Figure 9 This is a schematic diagram of the structure of the human-computer interaction device provided by the present invention;

[0050] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] In service-oriented or companion-oriented voice command interaction scenarios, voice commands are usually connected with the surrounding environment, meaning that entities in the surrounding environment appear frequently in the voice commands.

[0053] Existing speech recognition technologies lack descriptions of entities and other relevant information in the surrounding environment. If the voice command interaction scenario is related to the surrounding environment, the effectiveness of voice command recognition depends entirely on the speech recognition model's coverage of those entities, resulting in poor scalability.

[0054] At the same time, because different users have different environmental information, if the user's environmental information cannot be modeled, the ability to personalize the use of voice commands in this scenario will be poor, which will directly affect the user's subjective experience.

[0055] To address the aforementioned issues, this invention provides a speech recognition method. The method involves acquiring environmental video within the application environment, classifying and describing objects within the environment based on the video, and combining this with the user's personalized knowledge storage for speech recognition. This expands the text corpus to match user habits or current environmental information, thereby improving the accuracy of speech recognition, increasing the success rate of voice command interactions, and enhancing the user experience.

[0056] It should be noted that the speech recognition method provided by this invention can be applied to various speech recognition scenarios, especially service-oriented or companion-oriented speech command interaction scenarios such as smart homes and smart vending machines, and can improve the success rate of speech command interaction in such scenarios.

[0057] Figure 1 This is one of the flowcharts illustrating the speech recognition method provided by the present invention. The execution entity for each step in this method can be a speech recognition device, which can be implemented through software and / or hardware. This device can be integrated into an electronic device, which can be a human-computer interaction device itself; or it can be another device, besides the human-computer interaction device, that performs speech recognition and controls the human-computer interaction. Specifically, the human-computer interaction device can be a mobile terminal, smart home device, smart vending machine, smart companion robot, etc.; the other device performing speech recognition and controlling the human-computer interaction can be a server, personal computer, etc. Figure 1 As shown, the speech recognition method provided in this embodiment of the invention may include the following steps:

[0058] Step 110: Determine the environmental video of the environment in which the speech to be recognized is located.

[0059] Specifically, the speech to be recognized refers to the speech that needs to be recognized, which can be acquired by an acoustic acquisition component. The acoustic acquisition component can be a single microphone or a microphone array containing multiple microphones; this embodiment of the invention does not specifically limit this. The speech to be recognized can be a segmented speech or a user speech stream. Here, the user speech stream refers to the speech data stream acquired during voice interaction, which is recorded in real time; specifically, it can be obtained from voice recording or video recording.

[0060] It should be noted that the voice to be recognized here can be a voice data stream recorded by the user for voice interaction, such as a wake-up voice data stream used to wake up the voice interaction, or a voice data stream used to query specific information after wake-up, or a voice data stream recorded when the user interrupts the voice played by the voice interaction system during the voice interaction process. This embodiment of the invention does not make specific limitations on this.

[0061] Considering that the speech content to be recognized often relates to entities existing in the surrounding environment, such as in smart vending service scenarios, users might say: "Where is the apple?" or in companion-type voice interaction scenarios, users might say: "Please get me an apple." Here, the apple is an entity existing in the surrounding environment.

[0062] However, existing speech recognition technologies lack descriptions of entities and other relevant information in the surrounding environment. The effectiveness of speech recognition depends entirely on the speech recognition model's coverage of these entities, resulting in poor scalability.

[0063] Therefore, in this step, in addition to acquiring the speech to be recognized, environmental video of the environment in which the speech is located is also acquired. This environmental video can be a real-time video stream; it can also be the latest video clip stored from monitoring the target environment. This embodiment of the invention does not specifically limit this. For example, an environmental video can be acquired by capturing the scene where the user is present using a camera. Simultaneously, a known target tracking algorithm is used to monitor the scene environment in real time. If a target is added or changed, the video clip is updated again.

[0064] Understandably, environmental videos can represent the environment in which a user is located. Environmental videos contain entities that exist in the environment, as well as the attributes of those entities, such as their size, location, quantity, color, and other attributes.

[0065] Step 120: Based on the video description text of the environmental video, perform entity extraction on the video description text and update the keyword library based on the entity extraction results, and / or, perform entity relation extraction on the video description text and update the knowledge graph based on the entity relation extraction results.

[0066] Specifically, the video description text of an environmental video can represent detailed descriptions of entities existing in the scene, such as "there are three red apples on the table, and a sunflower painting hanging on the wall." The video description text can be extracted using video description algorithms to obtain the environmental video's video description text.

[0067] Building upon this foundation, to establish connections with potential voice commands generated by users during interaction, entity extraction can be performed on the video description text. Entities here can specifically be entity words or keywords. After obtaining the entity extraction results from the video description text, the keyword library can be updated based on these results. The keyword library contains multiple keywords, which can serve as hot words for speech recognition and also be used to expand the corpus coverage during subsequent speech intent understanding, thereby improving the success rate of voice command interaction and enhancing the user experience.

[0068] To obtain keywords, entity extraction can be performed using a pre-trained entity extraction model. Alternatively, each sentence in the video description text can be segmented, and entity extraction can be performed based on the part-of-speech tag of each segment. This embodiment of the invention does not impose specific limitations on this.

[0069] Video description text not only contains entities existing in the environment but also rich knowledge information, specifically attribute information related to each entity. For example, there are three apples, they are red, and they are located on the table. To store this knowledge information more conveniently, it can be described in the form of a knowledge graph. To obtain the knowledge information from the video description text, a pre-trained entity relation extraction model or a known relation extraction framework can be used.

[0070] After obtaining the keywords and / or knowledge information, the keyword library and / or knowledge graph can be updated based on the obtained keywords and / or knowledge information. This update can be to create a new initial keyword library and / or knowledge graph, or to update an existing initial keyword library and / or knowledge graph. For example, adding new keywords and / or knowledge information to the initial keyword library and / or knowledge graph will result in an updated keyword library and / or knowledge graph.

[0071] Step 130: Perform speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

[0072] Specifically, the updated keyword database and / or the updated knowledge graph both contain contextual information about the user's environment, enabling the formation of personalized knowledge tailored to the user. By performing speech recognition based on the updated keyword database and / or the updated knowledge graph, the accuracy of user speech recognition can be improved, thereby increasing the success rate of voice interaction and enhancing the user experience.

[0073] Understandably, speech recognition can be performed based solely on keywords from a keyword database; it can also be performed based on a knowledge graph; and of course, to further improve the accuracy of speech recognition, it can be performed simultaneously based on both a keyword database and a knowledge graph.

[0074] The method provided in this invention extracts entities from the video description text of an environmental video and updates the keyword library based on the entity extraction results, and / or extracts entity relationships from the video description text and updates the knowledge graph based on the entity relationship extraction results. Based on this, speech recognition is performed on the speech to be recognized using the keyword library and / or the knowledge graph. This allows for the formation of personalized knowledge tailored to the user, expanding the text corpus to match user habits or current environmental information, thereby improving the accuracy of speech recognition, increasing the success rate of voice command interaction, and enhancing the user experience.

[0075] Based on the above embodiments, step 130 specifically includes:

[0076] Step 131: Based on the updated keyword library and / or the updated knowledge graph, decode the acoustic features of the speech to be recognized to obtain the speech recognition result.

[0077] Specifically, the speech recognition process generally involves first extracting acoustic features from the speech to be recognized based on an acoustic model, and then decoding the acoustic features to obtain the recognition result.

[0078] In this embodiment, when decoding the acoustic features of the speech to be recognized, the updated keyword library and / or the updated knowledge graph can both serve as auxiliary information, which can help the decoder better understand the user's intent, thereby improving the success rate of decoding.

[0079] The method provided in this invention uses a keyword database and / or knowledge graph to provide auxiliary information for acoustic feature decoding, thereby improving the decoding success rate of speech recognition and further improving the accuracy of speech recognition.

[0080] Based on any of the above embodiments Figure 2 This is the second flowchart of the speech recognition method provided by the present invention, as shown below. Figure 2 As shown, step 131 specifically includes:

[0081] Step 131-1: Based on the entity extraction results and / or entity relationship extraction results, determine the new keywords;

[0082] Step 131-2: Perform text retrieval based on the newly added keywords to obtain new text corpus, and update the language model based on the new text corpus. The language model is used to decode the acoustic features of the speech to be recognized.

[0083] Step 131-3: Based on at least one of the updated keyword library, the updated knowledge graph, and the updated language model, decode the acoustic features of the speech to be recognized to obtain the speech recognition result.

[0084] Specifically, the entity extraction results and / or entity relation extraction results can characterize the entities contained in the video description information, that is, keywords can be obtained based on the entity extraction results and / or entity relation extraction results.

[0085] Since each keyword tag can include both the keyword itself and its time tag, the time tag can be determined based on the video description text, such as the time when the keyword first appeared. Because each keyword includes a time tag, new keywords can be identified based on these time tags; for example, keywords with time tags after the last update time can be considered new keywords.

[0086] After obtaining the new keywords, text searches can be performed based on them. The search sources for text searches can include existing text libraries or internet resources. The resulting text corpus is then obtained through the text search.

[0087] Based on this, the language model is updated according to the newly added text corpus. This language model can be used to decode the acoustic features of the speech to be recognized, and it can be pre-trained. Understandably, the language model can be updated every time new text corpus is acquired. Specifically, the update process can be as follows: interpolate the new text corpus with the language model from the previous time step to obtain the language model at the current time step, and use the language model at the current time step as the updated language model. The updated language model can then be used as the language model in the speech recognition process.

[0088] Therefore, the acoustic features of the speech to be recognized can be decoded based on any one of the updated keyword database, knowledge graph, and language model; it can also be based on any two of them; or it can be based on all the acoustic features of the speech to be recognized to obtain the speech recognition result.

[0089] The method provided in this invention updates the language model by adding new keywords. The updated language model can cover more text corpora that conform to user habits or current environmental information, thereby improving the accuracy of speech recognition and the success rate of voice interaction.

[0090] Based on any of the above embodiments Figure 3 This is the third flowchart of the speech recognition method provided by the present invention, as shown below. Figure 3 As shown, step 131 specifically includes:

[0091] Step 131-4: Determine the hot keyword incentive weight for each keyword based on the time tags of each keyword in the updated keyword library;

[0092] Step 131-5: Based on the hot word incentive weights of each keyword, decode the acoustic features of the speech to be recognized to obtain the speech recognition result.

[0093] Specifically, decoding the acoustic features of the speech to be recognized based on a keyword library can be achieved through a hot word excitation scheme. Keywords in the keyword library can be used as hot word information for speech recognition, and the weight of each keyword as a hot word input in the decoding process can be determined based on the time label of each keyword.

[0094] In other words, the closer the keyword's time tag is to the current moment, the greater the incentive weight of that keyword as a hot word; conversely, the further away the keyword's time tag is from the current moment, i.e., the greater the time interval, the smaller the incentive weight of that keyword as a hot word.

[0095] Subsequently, based on the hot word incentive weight of each keyword, a hot word incentive scheme can be adopted for hot word input during the speech recognition process. It is understandable that the higher the hot word incentive weight, the higher the probability that the keyword will be used as a hot word input; conversely, the lower the hot word incentive weight, the lower the probability that the keyword will be used as a hot word input.

[0096] The method provided in this invention determines the hot word incentive weight of each keyword by using the time tag of each keyword, and performs decoding based on the hot word incentive weight, which can further improve the decoding accuracy in the speech recognition process, thereby improving the success rate of voice interaction.

[0097] Based on any of the above embodiments Figure 4 This is the fourth flowchart of the speech recognition method provided by the present invention, as shown below. Figure 4 As shown, step 131 specifically includes:

[0098] Step 131-6: The graph features of the updated knowledge graph are fused with the encoding features of the decoding result at the previous decoding time to obtain the fused features at the current decoding time.

[0099] Step 131-7: Based on the fusion features at the current decoding moment and the decoding result at the previous decoding moment, decode the acoustic features of the speech to be recognized to obtain the speech recognition result, which is the final decoding result at the decoding moment.

[0100] Specifically, acoustic feature decoding based on knowledge graphs allows the knowledge graph to serve as auxiliary text information in the speech recognition decoding process. In this embodiment, the graph features of the knowledge graph can be obtained through graph convolution, resulting in the graph features of the knowledge spectrograph. The historical decoding result here refers to the historical information generated during the decoding process of the speech to be recognized before the current decoding moment.

[0101] The graph features of the knowledge graph can be fused with the encoded features of historical decoding results using feature concatenation to obtain fused features. However, unlike conventional feature fusion, the feature fusion in this embodiment is dynamic, and its fusion method changes as the feature decoding process progresses.

[0102] During feature fusion, it is possible to analyze and determine which information in the spectral features should be given priority at the current decoding moment, and which information in the coding features of the historical decoding results should be given priority. It is also possible to analyze and determine whether more attention should be paid to the information in the spectral features or the coding features of the historical decoding results at the current decoding moment. Thus, during feature fusion, the information that needs to be given priority is highlighted, while the information that does not need to be given priority is weakened, resulting in fused features that are more suitable for the current decoding moment.

[0103] After obtaining the fusion features at the current decoding moment, the decoding layer can perform decoding based on the fusion features at the current moment and the decoding result of the previous decoding moment, thereby obtaining the decoding result at the current decoding moment and outputting it. The speech recognition result is the final decoding result at the decoding moment.

[0104] Taking a common end-to-end speech recognition framework as an example, knowledge graphs can be used as auxiliary information for decoding. The objective function is:

[0105]

[0106] Where X represents the voice input, Y 1,…i-1 This represents the historical decoding result of speech recognition, i.e., the decoding result at the previous decoding moment; y i This represents the decoding result output corresponding to the current decoding moment, G represents the knowledge graph, and θ represents the training parameters.

[0107] The fusion of knowledge graphs and speech recognition models can be achieved by using graph convolution, which combines the output with the encoded information of historical decoding results.

[0108] The method provided in this invention integrates the knowledge graph with the encoding features of historical decoding results, thereby focusing on important information in the knowledge graph during the speech recognition decoding process and improving the accuracy of speech recognition.

[0109] Based on any of the above embodiments Figure 5 This is one of the flowcharts illustrating step 120 in the speech recognition method provided by the present invention, such as... Figure 5 As shown, step 120 involves extracting entities from the video description text based on the environmental video, and updating the keyword library based on the entity extraction results. Specifically, this includes:

[0110] Step 121: Based on the video description text, extract entities from each statement in the video description text to obtain the keywords contained in each statement;

[0111] Step 122: If the keyword library does not contain the keyword, add the keyword to the keyword library to obtain the updated keyword library.

[0112] Specifically, to obtain keywords from the video description text, entity extraction can be performed on each statement within the text. For example, the video description text... For each of the descriptions Word segmentation and part-of-speech classification are performed using known word segmentation tools. Since the generated keywords are mainly used to expand the corpus coverage in subsequent speech intent understanding processes, entity words or keywords are primarily selected from nouns and added to the keyword library.

[0113] Tags in a keyword database can include keywords and keyword time tags, and can be represented in the following form: {keyword, keyword time tag}. For example, The label for keyword k obtained from the data is {K, t}.

[0114] If keyword K already exists in the keyword library, keyword K will not be added to the keyword library, and the original keyword timestamp will still be retained in the keyword library, meaning the keyword library will not be updated; if keyword K is not included in the keyword library, the keyword and its timestamp will be added to the keyword library in a timely manner to obtain an updated keyword library.

[0115] The method provided in this invention, based on keywords obtained from video description text, updates the keyword database in a timely manner, thereby constructing a dynamic decoding network and further improving the decoding accuracy of speech recognition.

[0116] Based on any of the above embodiments Figure 6 This is the second flowchart of step 120 in the speech recognition method provided by the present invention, as shown below. Figure 6 As shown, step 120 involves extracting entity relations from the video description text and updating the knowledge graph based on the entity relation extraction results, including:

[0117] Step 123: Extract entity relations from each statement in the video description text to obtain knowledge information for each statement. The knowledge information includes entities, entity attributes, and attribute values.

[0118] Step 124: If the knowledge graph does not contain knowledge information, add the knowledge information to the knowledge graph to obtain the updated knowledge graph.

[0119] Specifically, to obtain knowledge information from the video description text, entity relations can be extracted from each statement in the video description text. For the text content of the video description... Each sentence in the content description Knowledge information storage is established by combining known relation extraction frameworks, where knowledge information includes descriptions of {entity, attribute, attribute value}. For example, "There is a green orange on the table" generates knowledge information including {orange, quantity, one}, {orange, color, green}, and {orange, location, on the table} for storage.

[0120] Based on this, if the knowledge information is not included in the knowledge graph (where "not included" means the corresponding entity or entity attribute is not present in the knowledge graph), then the knowledge information is added to the knowledge graph to obtain the updated knowledge graph.

[0121] The method provided in this invention extracts entity relationships from video description text to obtain knowledge information, and updates the knowledge graph based on this knowledge information. The updated knowledge graph can then be used as auxiliary information in speech recognition decoding, thereby improving the accuracy of speech recognition.

[0122] Based on any of the above embodiments Figure 7 This is the fifth flowchart of the speech recognition method provided by the present invention, as shown below. Figure 7 As shown, a speech recognition method is provided:

[0123] S1, Obtain the environmental video of the speech environment to be recognized, and obtain the video description text of the environmental video;

[0124] S2, extract entity words or keywords from the generated video description text, and update the keyword library based on the entity extraction results;

[0125] Entity relations are extracted from the generated video description text, and the knowledge graph is updated based on the entity relation extraction results;

[0126] Based on the entity extraction results and / or entity relation extraction results, new keywords are determined; text retrieval is performed based on the new keywords to obtain new text corpus, and the language model is updated based on the new text corpus;

[0127] Based on at least one of the updated keyword database, updated knowledge graph, and updated language model, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

[0128] Specifically, the process of decoding the acoustic features of the speech to be recognized based on the updated keyword library to obtain the speech recognition result may include: determining the hot word incentive weight of each keyword based on the time label of each keyword in the updated keyword library; and decoding the acoustic features of the speech to be recognized based on the hot word incentive weight of each keyword to obtain the speech recognition result.

[0129] Decoding the acoustic features of the speech to be recognized based on the updated knowledge graph to obtain the speech recognition result may specifically include: fusing the graph features of the updated knowledge graph with the encoded features of the decoding result at the previous decoding time to obtain the fused features at the current decoding time; and decoding the acoustic features of the speech to be recognized based on the fused features at the current decoding time and the decoding result at the previous decoding time to obtain the speech recognition result, which is the final decoding result at the final decoding time.

[0130] The speech recognition device provided by the present invention is described below. The speech recognition device described below and the speech recognition method described above can be referred to in correspondence.

[0131] Based on any of the above embodiments Figure 8 This is a schematic diagram of the structure of the voice recognition device provided by the present invention, as shown below. Figure 8 As shown, the voice recognition device includes an environmental video determination unit 810, an update unit 820, and a voice recognition unit 830, wherein...

[0132] The environmental video determination unit 810 is used to determine the environmental video of the environment in which the speech to be recognized is located.

[0133] The update unit 820 is used to extract entities from the video description text based on the environmental video and update the keyword library based on the entity extraction results, and / or extract entity relations from the video description text and update the knowledge graph based on the entity relation extraction results.

[0134] The speech recognition unit 830 is used to perform speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

[0135] The speech recognition device provided in this invention extracts entities from the video description text of an environmental video and updates the keyword library based on the entity extraction results, and / or extracts entity relationships from the video description text and updates the knowledge graph based on the entity relationship extraction results. Based on this, speech recognition is performed on the speech to be recognized using the keyword library and / or knowledge graph. This allows for the formation of personalized knowledge tailored to the user, expanding the text corpus to match user habits or current environmental information, thereby improving the accuracy of speech recognition, increasing the success rate of voice command interaction, and enhancing the user experience.

[0136] Based on any of the above embodiments, the speech recognition unit is further configured to:

[0137] Based on the updated keyword library and / or the updated knowledge graph, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

[0138] Based on any of the above embodiments, the speech recognition unit is further configured to:

[0139] Based on the entity extraction results and / or entity relationship extraction results, new keywords are determined;

[0140] Text retrieval is performed based on the newly added keywords to obtain new text corpus, and the language model is updated based on the new text corpus. The language model is used to decode the acoustic features of the speech to be recognized.

[0141] Based on at least one of the updated keyword database, updated knowledge graph, and updated language model, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

[0142] Based on any of the above embodiments, the speech recognition unit is further configured to:

[0143] Based on the time tags of each keyword in the updated keyword library, determine the hot keyword incentive weight for each keyword;

[0144] Based on the hot word incentive weights of each keyword, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

[0145] Based on any of the above embodiments, the speech recognition unit is further configured to:

[0146] The updated knowledge graph features are fused with the encoded features of the decoding result from the previous decoding time to obtain the fused features for the current decoding time.

[0147] Based on the fusion features at the current decoding moment and the decoding result at the previous decoding moment, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result, which is the decoding result at the final decoding moment.

[0148] Based on any of the above embodiments, the updating unit is further configured to:

[0149] Based on the video description text, entity extraction is performed on each statement in the video description text to obtain the keywords contained in each statement;

[0150] If the keyword is not included in the keyword library, the keyword is added to the keyword library to obtain an updated keyword library.

[0151] Based on any of the above embodiments, the updating unit is further configured to:

[0152] Entity relations are extracted from each statement in the video description text to obtain knowledge information for each statement, including entities, entity attributes, and attribute values.

[0153] If the knowledge information is not present in the knowledge graph, the knowledge information is added to the knowledge graph to obtain an updated knowledge graph.

[0154] Based on any of the above embodiments Figure 9 This is a schematic diagram of the structure of the human-computer interaction device provided by the present invention, as shown below. Figure 9 As shown, the human-computer interaction device includes a microphone 910, a camera 920, and a processor 930 connected in sequence;

[0155] Microphone 910 is used to acquire the speech to be recognized;

[0156] Camera 920 is used to acquire environmental video of the environment in which the voice to be recognized is located, and transmit the environmental video to the processor;

[0157] The processor 930 is used for video description text based on environmental video, performing entity extraction on the video description text, updating the keyword library based on the entity extraction results, and / or performing entity relation extraction on the video description text, updating the knowledge graph based on the entity relation extraction results; performing speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph, and performing human-computer interaction based on the speech recognition results.

[0158] Specifically, the human-computer interaction device can be used to engage in voice dialogue with users. The interaction process can be broken down into three stages: voice recognition, conversation processing, and voice synthesis. Voice recognition, as the first stage, significantly impacts the success rate of the interaction and the user experience. This human-computer interaction device can be applied to various voice interaction scenarios, particularly service-oriented or companion-oriented voice command interaction scenarios such as smart homes and smart vending machines, improving the success rate of voice command interactions in these scenarios.

[0159] The human-computer interaction device may include a microphone and a camera, wherein the microphone is used to acquire the speech to be recognized; the camera is used to acquire environmental video of the environment in which the speech to be recognized is located, and transmit the environmental video to the processor.

[0160] After receiving the environmental video, the processor first obtains the video description text of the environmental video, performs entity extraction on the video description text, and updates the keyword library based on the entity extraction results, and / or performs entity relation extraction on the video description text, and updates the knowledge graph based on the entity relation extraction results.

[0161] Based on this, speech recognition is performed on the speech to be recognized using the updated keyword library and / or the updated knowledge graph, and human-computer interaction is performed based on the speech recognition results.

[0162] The specific implementation methods of each step can be referred to the description of the above embodiments, and will not be repeated here.

[0163] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a speech recognition method, which includes: determining an environmental video of the environment in which the speech to be recognized is located; performing entity extraction on the video description text based on the environmental video, and updating a keyword library based on the entity extraction results; and / or performing entity relation extraction on the video description text, and updating a knowledge graph based on the entity relation extraction results; and performing speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

[0164] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech recognition method provided by the above methods. The method includes: determining an environmental video of the environment in which the speech to be recognized is located; performing entity extraction on the video description text based on the environmental video, and updating a keyword library based on the entity extraction results; and / or performing entity relation extraction on the video description text, and updating a knowledge graph based on the entity relation extraction results; and performing speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

[0166] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the speech recognition method provided by the above methods. The method includes: determining an environmental video of the environment in which the speech to be recognized is located; performing entity extraction on the video description text based on the environmental video, and updating a keyword library based on the entity extraction results; and / or performing entity relation extraction on the video description text, and updating a knowledge graph based on the entity relation extraction results; and performing speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Determine the environmental video of the environment in which the speech to be recognized is located; Based on the video description text of the environmental video, entity extraction is performed on the video description text, and the keyword library is updated based on the entity extraction results; and / or, entity relation extraction is performed on the video description text, and the knowledge graph is updated based on the entity relation extraction results. Speech recognition is performed on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

2. The speech recognition method according to claim 1, characterized in that, The step of performing speech recognition on the speech to be recognized based on the updated keyword database and / or the updated knowledge graph includes: Based on the updated keyword library and / or the updated knowledge graph, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

3. The speech recognition method according to claim 2, characterized in that, The process of decoding the acoustic features of the speech to be recognized based on the updated keyword database and / or the updated knowledge graph to obtain the speech recognition result includes: Based on the entity extraction results and / or entity relationship extraction results, new keywords are determined; Text retrieval is performed based on the newly added keywords to obtain new text corpus, and the language model is updated based on the new text corpus. The language model is used to decode the acoustic features of the speech to be recognized. Based on at least one of the updated keyword database, updated knowledge graph, and updated language model, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

4. The speech recognition method according to claim 2, characterized in that, The process of decoding the acoustic features of the speech to be recognized based on the updated keyword database to obtain the speech recognition result includes: Based on the time tags of each keyword in the updated keyword library, determine the hot keyword incentive weight for each keyword; Based on the hot word incentive weights of each keyword, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result.

5. The speech recognition method according to claim 2, characterized in that, The process of decoding the acoustic features of the speech to be recognized based on the updated knowledge graph to obtain the speech recognition result includes: The updated knowledge graph features are fused with the encoded features of the decoding result from the previous decoding time to obtain the fused features for the current decoding time. Based on the fusion features at the current decoding moment and the decoding result at the previous decoding moment, the acoustic features of the speech to be recognized are decoded to obtain the speech recognition result, which is the decoding result at the final decoding moment.

6. The speech recognition method according to any one of claims 1-5, characterized in that, The process of extracting entities from the video description text based on the environmental video and updating the keyword library based on the entity extraction results includes: Based on the video description text, entity extraction is performed on each statement in the video description text to obtain the keywords contained in each statement; If the keyword is not included in the keyword library, the keyword is added to the keyword library to obtain an updated keyword library.

7. The speech recognition method according to any one of claims 1-5, characterized in that, The step of extracting entity relations from the video description text and updating the knowledge graph based on the entity relation extraction results includes: Entity relations are extracted from each statement in the video description text to obtain knowledge information for each statement, including entities, entity attributes, and attribute values. If the knowledge information is not present in the knowledge graph, the knowledge information is added to the knowledge graph to obtain an updated knowledge graph.

8. A voice recognition device, characterized in that, include: An environmental video determination unit is used to determine the environmental video of the environment in which the speech to be recognized is located. The update unit is used to extract entities from the video description text based on the environmental video, update the keyword library based on the entity extraction results, and / or extract entity relations from the video description text, update the knowledge graph based on the entity relation extraction results. The speech recognition unit is used to perform speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph.

9. A human-computer interaction device, characterized in that, Includes a camera, microphone, and processor connected in sequence; The microphone is used to acquire the speech to be recognized; The camera is used to acquire environmental video of the environment in which the voice to be recognized is located, and to transmit the environmental video to the processor; The processor is configured to perform entity extraction on the video description text based on the environmental video, and update the keyword library based on the entity extraction results, and / or perform entity relation extraction on the video description text, and update the knowledge graph based on the entity relation extraction results; perform speech recognition on the speech to be recognized based on the updated keyword library and / or the updated knowledge graph, and perform human-computer interaction based on the speech recognition results.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speech recognition method as described in any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 7.

Images