A speech recognition method, apparatus, device, and storage medium

Abstract

The application provides a speech recognition method, device and equipment and a storage medium. The speech recognition method comprises the following steps: obtaining target speech and target video of a specified field, wherein the target speech is commentary speech of video picture content of the target video; detecting an entity from a video picture of the target video to obtain a first target entity; obtaining knowledge information of the first target entity from a knowledge graph of the specified field; and performing speech recognition on the target speech with the aid of the knowledge information of the first target entity to obtain a speech recognition result of the target speech. The application starts from the characteristics of commentary speech, proposes to detect an entity from a video picture of a target video, obtain knowledge information of the detected entity from a knowledge graph, and then perform speech recognition on target speech with the aid of the obtained entity knowledge information. When the target speech is recognized, the knowledge information of the detected entity from the video picture of the target video is used, and a more accurate speech recognition result can be obtained.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus, device, and storage medium. Background Technology

[0002] Commentary voice generally refers to the audio provided by a commentator explaining the content of a video clip. For example, game commentary voice refers to the audio provided by a game commentator explaining the content of a game video. In certain fields, commentary voice recognition services are becoming increasingly important, with growing demand and increasingly higher requirements for the quality of such services.

[0003] Most current speech recognition solutions are designed for general speech, and because they focus on recognizing general speech, they perform well in this area. However, narration is usually speech specific to a particular domain. Therefore, general speech recognition solutions are not very applicable to narration; that is, using general speech recognition solutions to recognize narration results in poor performance. Summary of the Invention

[0004] In view of this, the present invention provides a speech recognition method, apparatus, device, and storage medium to solve the problem that the recognition effect is poor when using a recognition scheme for general speech to recognize narration speech. The technical solution is as follows:

[0005] A speech recognition method, comprising:

[0006] Acquire target speech and target video in a specified domain, wherein the target speech is the narration of the video content of the target video;

[0007] Entities are detected from the video frame of the target video to obtain the first target entity;

[0008] Obtain knowledge information of the first target entity from the knowledge graph of the specified domain;

[0009] Using the knowledge information of the first target entity, speech recognition is performed on the target speech to obtain the speech recognition result of the target speech.

[0010] Optionally, detecting entities from the video frames of the target video to obtain the first target entity includes:

[0011] The target video is encoded into features that can characterize the video frame information of the target video to obtain the visual features of the target video;

[0012] Based on the visual features of the target video, entities are extracted from the video frames of the target video to obtain the first target entity.

[0013] Optionally, the step of using the knowledge information of the first target entity to perform speech recognition on the target speech to obtain the speech recognition result of the target speech includes:

[0014] The target speech is encoded into features that can characterize the speech content information of the target speech, thereby obtaining the speech features of the target speech;

[0015] The speech features of the target speech are fused with the visual features of the target video to obtain fused features;

[0016] Encode the knowledge information of the first target entity into features that can characterize the knowledge information of the first target entity to obtain the knowledge features of the first entity;

[0017] Based on the fusion features and the first entity knowledge features, the speech recognition result of the target speech is determined.

[0018] Optionally, determining the speech recognition result of the target speech based on the fusion features and the first entity knowledge features includes:

[0019] At each decoding moment:

[0020] Based on the fusion features and the first entity knowledge features, determine the features required for decoding at this decoding moment;

[0021] Based on the features required for decoding at that decoding moment, the speech recognition result of the target speech at that decoding moment is determined.

[0022] Optionally, before determining the features required for decoding at the decoding moment based on the fused features and the first entity knowledge features, the method further includes:

[0023] The second target entity is obtained by detecting entities from the speech recognition results of the target speech at historical decoding times;

[0024] Obtain knowledge information about the second target entity from the knowledge graph of the specified domain;

[0025] Encode the knowledge information of the second target entity into features that can characterize the knowledge information of the second target entity to obtain the knowledge features of the second entity;

[0026] The step of determining the features required for decoding at this decoding moment based on the fused features and the first entity knowledge features includes:

[0027] Based on the fusion features, the first entity knowledge features, and the second entity knowledge features, the features required for decoding at this decoding moment are determined.

[0028] Optionally, the process of acquiring the knowledge graph for the specified domain includes:

[0029] Obtain multiple narration audio clips with text labeled with audio content in the specified domain;

[0030] The text of the audio content annotated from multiple narrations is regularized, and the regularized text is segmented into multiple words.

[0031] By combining word frequency, several entity words are obtained from the multiple words, and the relationship information between the several entity words is obtained;

[0032] Based on the aforementioned entity words and the relationship information between them, a knowledge graph for the specified domain is constructed. The knowledge graph includes several knowledge nodes and directed edges between them. Each knowledge node is an entity word. There is a relationship between the two knowledge nodes connected by each directed edge. Each directed edge contains the relationship information between the two knowledge nodes connected by the directed edge.

[0033] Optionally, obtaining the knowledge information of the first target entity from the knowledge graph of the specified domain includes:

[0034] The knowledge information of the first target entity is obtained by extracting the first-level knowledge nodes of the first target entity and the relationship information between the first target entity and the first-level knowledge nodes from the knowledge graph of the specified domain.

[0035] Optionally, the knowledge graph includes several knowledge nodes and directed edges between the knowledge nodes. Each knowledge node is an entity word. There is a relationship between the two knowledge nodes connected by each directed edge. Each directed edge contains the relationship information between the two knowledge nodes connected by the directed edge.

[0036] The step of obtaining knowledge information about the second target entity from the knowledge graph of the specified domain includes:

[0037] The knowledge information of the second target entity is obtained by extracting two or more levels of knowledge nodes of the second target entity and the relationship information between the second target entity and the two or more levels of knowledge nodes from the knowledge graph of the specified domain.

[0038] Optionally, the step of detecting entities from the video frame of the target video to obtain a first target entity; obtaining knowledge information of the first target entity from the knowledge graph of the specified domain; and using the knowledge information of the first target entity to perform speech recognition on the target speech to obtain the speech recognition result of the target speech includes:

[0039] The target speech and the target video are processed based on a pre-trained speech recognition model to obtain the speech recognition result of the target speech;

[0040] The speech recognition model is trained using training data from the training set in the specified domain. The training data includes training speech with text labeled with speech content and training videos labeled with entities. The training speech is the narration of the video content of the training video.

[0041] Optionally, the training process of the speech recognition model includes:

[0042] Based on the speech recognition model, entities are detected from the video frames of the training video to obtain the entity detection results corresponding to the training video, and the knowledge information of the detected entities is obtained from the knowledge graph of the specified domain to obtain the entity knowledge information corresponding to the training video.

[0043] Based on the speech recognition model, and supplemented by entity knowledge information corresponding to the training video, speech recognition is performed on the training speech to obtain the speech recognition result of the training speech;

[0044] A first loss is determined based on the speech recognition results of the training speech and the speech content text labeled in the training speech, and a second loss is determined based on the entity detection results corresponding to the training video and the entities labeled in the training video.

[0045] The parameters of the speech recognition model are updated based on the first loss and the second loss.

[0046] A speech recognition device includes: a data acquisition module, a recognition auxiliary information acquisition module, and a speech recognition result determination module;

[0047] The data acquisition module is used to acquire target speech and target video in a specified field, wherein the target speech is the narration speech of the video content of the target video;

[0048] The identification assistance information acquisition module is used to detect entities from the video frame of the target video to obtain a first target entity, and to obtain knowledge information of the first target entity from the knowledge graph of the specified domain.

[0049] The speech recognition result determination module is used to perform speech recognition on the target speech with the aid of knowledge information of the first target entity, and obtain the speech recognition result of the target speech.

[0050] A voice recognition device includes: a memory and a processor;

[0051] The memory is used to store programs;

[0052] The processor is configured to execute the program to implement each step of the speech recognition method described above.

[0053] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the speech recognition method described in any of the preceding claims.

[0054] The speech recognition method, apparatus, device, and storage medium provided by this invention first acquire target speech and target video (the target speech is the narration of the target video content) within a specified domain. Then, entities are detected from the video frames of the target video to obtain a first target entity. Next, knowledge information of the first target entity is obtained from a knowledge graph of the specified domain. Finally, the knowledge information of the first target entity is used to perform speech recognition on the target speech to obtain the speech recognition result. Considering that the target speech is the narration of the target video frame, and it has a certain correlation with the video frame content, this invention, based on this characteristic of the target speech being the narration, proposes to detect entities from the video frames of the target video and obtain knowledge information of the detected entities from a knowledge graph of the specified domain. Then, the obtained entity knowledge information is used to perform speech recognition on the target speech. By supplementing the speech recognition of the target speech (narration) with the knowledge information of the entities detected from the video frames of the target video, a more accurate speech recognition result can be obtained. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the hardware architecture involved in the present invention;

[0057] Figure 2 A flowchart illustrating the speech recognition method provided in an embodiment of the present invention;

[0058] Figure 3 This is an example of a knowledge graph in the gaming field provided in an embodiment of the present invention;

[0059] Figure 4 A schematic diagram illustrating the process of obtaining a knowledge graph in a specified domain, provided in an embodiment of the present invention;

[0060] Figure 5This is a flowchart illustrating the process of using knowledge information of a first target entity to perform speech recognition on target speech and obtain the speech recognition result of the target speech, as provided in an embodiment of the present invention.

[0061] Figure 6 This is a schematic diagram of the process for training a speech recognition model according to an embodiment of the present invention;

[0062] Figure 7 This is a schematic diagram of the structure of the speech recognition model provided in an embodiment of the present invention;

[0063] Figure 8 This is a schematic diagram of the structure of the speech recognition device provided in an embodiment of the present invention;

[0064] Figure 9 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] Most current speech recognition solutions are based on general speech recognition models. This involves pre-training a general speech recognition model and then using the trained model to recognize the speech input. While these general-purpose models perform well for recognizing general speech, their accuracy is lower when applied to recognizing narration in specific domains.

[0067] The inventors of this case, through research on speech recognition tasks in specific domains, discovered that speech recognition tasks in specific domains are significantly more difficult than speech recognition tasks in general domains. Taking game commentary speech recognition tasks in the gaming field as an example: many factors affect the accuracy of game commentary speech recognition, such as the commentator's Mandarin proficiency, regional accent, game-specific terminology, and game scene noise. In addition, many game commentators may also be game players, meaning that different game commentators may have different levels of education, ages, and accents. In other words, the accuracy of game commentary speech recognition is also affected by complex factors such as the commentator's education level, age, and accent. All of these factors significantly reduce the accuracy of game commentary speech recognition.

[0068] To significantly improve the recognition accuracy of narration in specific domains, the inventors of this case focused on these domains and discovered that while some existing speech recognition solutions addressed these domains, they did not significantly improve recognition performance. Researching these solutions, the inventors found that most were single-modal. Based on this finding, they conceived of using a multimodal speech recognition approach. Specifically, they envisioned using the speaker's lip-reading to assist in speech recognition, thereby improving the accuracy. The method involves training a speech recognition model using the speaker's training speech and a corresponding video (containing the speaker's face). The speech to be recognized and its corresponding video (containing the speaker's face) are then input into the speech recognition model.

[0069] This study investigated the aforementioned multimodal speech recognition scheme and found that it has several shortcomings: First, it does not fully consider the characteristics of the narration; second, the scheme requires capturing the speaker's (narrator's) facial information via a camera, which, due to user privacy concerns, renders the multimodal speech recognition model unsuitable for many situations.

[0070] Given the numerous shortcomings of the aforementioned multimodal speech recognition schemes, the inventors of this case continued their research. Through this research, the inventors discovered that narration is the voice of the commentator describing and analyzing the content of the video footage. This means that the content of the narration is related to the content of the video footage being described. Taking game commentary as an example, since game commentary is the voice of the game commentator describing and analyzing the content of the game video footage, the content of the game video footage and the content of the narration are related. For example, if the content of the game video is "The robot is walking from the river in the middle lane, and at this moment, Huo Wu gets the first blood," the corresponding narration content would be "Huo Wu gets the first blood in the bottom lane." Based on this discovery, the inventors continued their research and, through continuous efforts, finally proposed a speech recognition method with better performance.

[0071] The basic concept of the proposed speech recognition method is to detect entities from the video footage of the narrator's narration, acquire knowledge information about these entities, and then use this knowledge information, along with the entity knowledge detected from the video footage, to recognize the narration's speech. This speech recognition method fully considers the characteristics of narration's speech, significantly improving the accuracy of speech recognition. Furthermore, since this method does not require capturing the narrator's facial information through a camera, it does not infringe on the narrator's privacy.

[0072] Before introducing the speech recognition method provided by this invention, the hardware architecture involved in this invention will be described first.

[0073] In one possible implementation, such as Figure 1 As shown, the hardware architecture involved in this invention may include: electronic device 101 and server 102.

[0074] For example, electronic device 101 can be any electronic product that can interact with a user, such as a PC, laptop, tablet, mobile phone, learning machine, smart TV, etc.

[0075] It should be noted that, Figure 1 This is just one example; there can be many types of electronic devices, not limited to... Figure 1 The laptop in the middle.

[0076] For example, server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center. Server 102 may include processors, memory, and network interfaces, etc.

[0077] For example, electronic device 101 can establish a connection and communicate with server 102 through a wireless communication network; for example, electronic device 101 can establish a connection and communicate with server 102 through a wired communication network.

[0078] Electronic device 101 can acquire target speech and target video (target speech is the narration of the video content of the target video), and send the target speech and target video to server 102. Server 102 performs speech recognition on the target speech according to the speech recognition method provided by the present invention.

[0079] In another possible implementation, the hardware architecture involved in this invention may include: an electronic device.

[0080] The electronic device is an electronic product with strong data processing capabilities. The electronic device can acquire target speech and target video (the target speech is the narration of the video content of the target video), and perform speech recognition on the target speech according to the speech recognition method provided by the present invention.

[0081] Those skilled in the art should understand that the above-described electronic devices and servers are merely examples, and other existing or future electronic devices or servers that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0082] The speech recognition method provided by the present invention will be described in the following embodiments.

[0083] Please see Figure 2 The diagram illustrates a flowchart of a speech recognition method provided in an embodiment of the present invention. This speech recognition method may include:

[0084] Step S201: Obtain the target speech and target video in the specified domain.

[0085] Among them, the target speech is the speech to be recognized, which is the narration of the video content of the target video, that is, the narrator's narration of the video content of the target video.

[0086] For example, the target speech and target video refer to speech and video in the gaming field. Specifically, the target video is a game video, and the target speech is the narration of the game video's visual content, i.e., the speech of a game commentator explaining the video content. In recent years, my country's game live streaming industry has developed rapidly. Since 2017, the market size and user base of the game live streaming industry have shown a rapid growth trend. In 2020, the market size reached approximately 30 billion yuan, a year-on-year increase of 44%, and the number of game live streaming users reached 350 million, a year-on-year increase of 15%. Currently, the game live streaming industry has entered a mature development stage. At the same time, game live streaming platforms are constantly enriching their business models and showing a trend of expanding into overseas markets. Major game platforms are also exhibiting diversified and globalized development phenomena. Therefore, in the gaming field, game commentary speech recognition services are becoming increasingly important.

[0087] Besides being applicable to the gaming industry, target audio and target video can also be used in other fields, such as sports. Specifically, target video can be sports event videos, and target audio can be the narration of the video content of sports event videos, that is, the narration of sports commentators on the video content of sports event videos.

[0088] Step S202: Detect entities from the video frame of the target video to obtain the first target entity.

[0089] For example, if the target video shows "the robot is walking from the river in the middle lane, and at this time, Huo Wu gets the first blood", then entities such as "robot", "river" and "Huo Wu" can be detected from the target video.

[0090] Specifically, the process of detecting entities from the video frames of a target video to obtain the first target entity may include: firstly, encoding the target video into features that can characterize the video frame information of the target video to obtain the visual features of the target video; and then, based on the visual features of the target video, extracting entities from the video frames of the target video to obtain the first target entity.

[0091] Step S203: Obtain knowledge information of the first target entity from the knowledge graph of the specified domain.

[0092] The knowledge information of the first target entity is obtained from the knowledge graph of the specified domain, that is, the knowledge of the first target entity is augmented.

[0093] The knowledge graph for a specified domain can include several knowledge nodes and directed edges between them. Each knowledge node is an entity in the specified domain. There is a relationship between the two knowledge nodes connected by each directed edge, and each directed edge contains the relationship information between the two knowledge nodes connected by the directed edge.

[0094] Please see Figure 3 This example illustrates a knowledge graph in the gaming domain. In the graph, "Sun Wukong," "monkey," "Huaguo Mountain," "bottom lane," and "protective spell" are all knowledge nodes. The "home" on the outgoing edge represents the relationship between the knowledge nodes "Sun Wukong" and "Huaguo Mountain." The "nickname" on the outgoing edge represents the relationship between the knowledge nodes "Sun Wukong" and "monkey." The "skill 2" on the outgoing edge represents the relationship between the knowledge nodes "Sun Wukong" and "protective spell." The information on other outgoing edges is similar.

[0095] Optionally, the first-level knowledge point information of the first target entity can be extracted from the knowledge graph of the specified domain as the knowledge information of the first target entity. The first-level knowledge point information of the first target entity includes the first-level knowledge nodes of the first target entity and the relationship information between the first-level knowledge nodes of the first target entity, such as... Figure 3 As shown, the first-level knowledge nodes of the knowledge node "Sun Wukong" are "Monkey", "Flower Fruit Mountain", "Bottom Road", "Great Sage's Divine Might", "Ruyi Jingu Bang", "Fighting Charge" and "Protective Spell". The relationship information between the knowledge node "Sun Wukong" and the first-level knowledge node "Monkey" is "Nickname", the relationship information between the knowledge node "Sun Wukong" and the first-level knowledge node "Flower Fruit Mountain" is "Home", ..., the relationship information between the knowledge node "Sun Wukong" and the first-level knowledge node "Protective Spell" is "Skill 2".

[0096] Considering that the first-level knowledge point information of the first target entity is more relevant to the speech content of the speech to be recognized, the first-level knowledge point information of the first target entity can be extracted from the knowledge graph of the specified domain. Of course, this embodiment is not limited to this. For example, the second-level knowledge point information of the first target entity (including the second-level knowledge nodes of the first target entity and the relationship information between the second-level knowledge nodes of the first target entity) can also be extracted from the knowledge graph of the specified domain as the knowledge information of the first target entity. The specific number of levels of knowledge point information of the first target entity extracted can be determined according to the specific scenario.

[0097] Step S204: Using the knowledge information of the first target entity, perform speech recognition on the target speech to obtain the speech recognition result of the target speech.

[0098] Since the target speech is the narration of the video content of the target video, the speech content of the target speech is related to the video content of the target video. In view of this, the present invention extracts useful information, namely the first target entity, from the video content of the target video, and extracts the knowledge information of the first target entity from the knowledge graph of the specified domain. The knowledge information of the first target entity is used as speech recognition auxiliary information to assist in target speech recognition.

[0099] The speech recognition method provided in this invention first acquires target speech and target video in a specified domain. Then, it detects entities from the video frame of the target video to obtain a first target entity. Next, it obtains knowledge information of the first target entity from a knowledge graph in the specified domain. Finally, it uses the knowledge information of the first target entity to perform speech recognition on the target speech to obtain the speech recognition result. Considering that the target speech is the narration of the target video frame, and it has a certain correlation with the content of the target video frame, this invention, based on this characteristic of the target speech being the narration, proposes to detect entities from the video frame of the target video and obtain knowledge information of the detected entities from a knowledge graph in the specified domain. Then, it uses the obtained entity knowledge information to perform speech recognition on the target speech. When performing speech recognition on the target speech, i.e., the narration, and supplementing it with the knowledge information of the entities detected from the video frame of the target video, a more accurate speech recognition result can be obtained. The speech recognition method provided in this invention significantly improves the user experience of speech recognition services in narration scenarios.

[0100] In another embodiment of the present invention, the process of acquiring a knowledge graph for a specified domain is described.

[0101] Please see Figure 4 The diagram illustrates a process for obtaining a knowledge graph for a specified domain, which may include:

[0102] Step S401: Obtain multiple narration audio clips with text labeled with audio content in the specified domain.

[0103] For example, if the specified domain is the game domain, multiple game commentary voices labeled with audio content text can be obtained.

[0104] Step S402: The text of the audio content annotated with multiple narrations is normalized, and the normalized text is segmented into words to obtain multiple words.

[0105] The normalization processing performed on the acquired audio content text may include, but is not limited to, filtering out useless characters, sentences, etc., according to set rules.

[0106] When segmenting the regularized text, existing segmentation methods can be used.

[0107] Step S403: Combine word frequency to obtain several entity words from multiple words, and obtain the relationship information between several entity words.

[0108] There are several ways to extract several entity words from multiple words by combining word frequency. This embodiment provides the following two optional implementation methods:

[0109] The first implementation method is to sort the words in descending order of frequency, and then distinguish between entity words and non-entity words for each of the first M words (the specific value of M can be determined according to the specific scenario). The words that are determined to be entity words are selected as the final entity words, thus obtaining a number of entity words. (Of course, the words can also be sorted in ascending order of frequency, and the last M words can be distinguished between entity words and non-entity words to obtain a number of entity words.)

[0110] The second implementation method is as follows: For each word, the word is distinguished as an entity word or a non-entity word, and the word frequency is determined as whether it is greater than a preset word frequency threshold. If the word is an entity word and its word frequency is greater than the preset word frequency threshold, then the word is selected as the final entity word, thereby obtaining a number of entity words.

[0111] After obtaining a number of entity words, the relationship information between the entity words can be obtained based on the text in which the entity words are located.

[0112] It should be noted that when obtaining several entity words from multiple words, word frequency may not be considered. That is, each word can be distinguished as an entity word or a non-entity word, and the words that are determined to be entity words can be used as the final selected entity words to obtain several entity words.

[0113] Step S404: Construct a knowledge graph for a specified domain based on several entity words and the relationship information between them.

[0114] Specifically, several knowledge nodes (each knowledge node being an entity word) are constructed based on several entity words. Directed edges are then built between these knowledge nodes based on the relationships between the entity words, and the relationship information between the two connected knowledge nodes is added to each directed edge. Optionally, the construction of the knowledge graph can be combined with the explanatory style of a specified domain.

[0115] The above process can be used to obtain a knowledge graph for a specific domain.

[0116] In another embodiment of the present invention, the implementation process of "step S204: using the knowledge information of the first target entity to perform speech recognition on the target speech and obtain the speech recognition result of the target speech" in the above embodiment will be described.

[0117] There are several ways to perform speech recognition on the target speech, supplemented by the knowledge information of the first target entity, and obtain the speech recognition result of the target speech. In one possible implementation, the target speech can first be encoded into features that can represent the speech content information of the target speech to obtain the speech features of the target speech, and the knowledge information of the first target entity can be encoded into features that can represent the knowledge information of the first target entity to obtain the first entity knowledge features. Then, the speech recognition result of the target speech can be determined based on the speech features of the target speech and the first entity knowledge features.

[0118] To fully utilize the video footage information of the target video to improve speech recognition performance, this embodiment provides another method for performing speech recognition on the target speech using knowledge information of the first target entity, thereby obtaining the speech recognition result of the target speech. Figure 5 As shown, it may include:

[0119] Step S501: Encode the target speech into features that can characterize the speech content information of the target speech, and obtain the speech features of the target speech.

[0120] The target speech is encoded to obtain speech features that can represent the speech content information of the target speech.

[0121] Step S502: Fuse the speech features of the target speech with the visual features of the target video to obtain fused features.

[0122] Among them, the visual features of the target video are the visual features that can characterize the video image information of the target video.

[0123] Step S503: Encode the knowledge information of the first target entity into features that can represent the knowledge information of the first target entity, and obtain the knowledge features of the first entity.

[0124] The knowledge information of the first target entity is encoded to obtain the first entity knowledge features that can represent the knowledge information of the first target entity.

[0125] It should be noted that this embodiment does not specifically limit the execution order of step S503; step S503 only needs to be executed before step S504.

[0126] Step S504: Determine the speech recognition result of the target speech based on the fusion features and the first entity knowledge features.

[0127] The above implementation method uses the video frame information of the target video and the knowledge information of the entities detected from the video frame of the target video to assist the target speech recognition, thus obtaining a more accurate speech recognition result.

[0128] Specifically, the process of determining the speech recognition result of the target speech based on the fusion features and the first entity knowledge features may include: at each decoding time, determining the features required for decoding at that decoding time based on the fusion features and the first entity knowledge features, and determining the speech recognition result of the target speech at that decoding time based on the features required for decoding at that decoding time.

[0129] In one possible implementation, the features required for decoding at a given decoding moment can be determined solely based on the fused features and the first entity knowledge features. Then, based on these features, the speech recognition result of the target speech at that decoding moment can be determined. Introducing entity knowledge information detected from the target video frame during speech recognition can effectively improve the accuracy of target speech recognition.

[0130] Considering that sometimes the video frame information of the target video is redundant, or the video frame content of the target video is weakly correlated with the speech content of the target speech, the introduced knowledge information of the first target entity may affect the recognition result of the target speech in such cases. In order to still obtain relatively accurate speech recognition results in the above situations, this invention proposes another implementation method:

[0131] The system detects entities from the speech recognition results of the target speech at historical decoding moments to obtain the second target entity (i.e., historical context information). It then obtains the knowledge information of the second target entity from the knowledge graph of the specified domain (i.e., performs knowledge enhancement on the second target entity). The knowledge information of the second target entity is encoded into features that can represent the knowledge information of the second target entity to obtain the second entity knowledge features. Subsequently, based on the fused features, the first entity knowledge features, and the second entity knowledge features, the features required for decoding at this decoding moment are determined. Based on the features required for decoding at this decoding moment, the speech recognition result of the target speech at this decoding moment is determined.

[0132] When performing speech recognition on target speech, introducing knowledge information from a second target entity can effectively alleviate the problem that the knowledge information from the first target entity may affect the recognition effect when the video footage information of the target video is redundant or when the video footage content of the target video is weakly correlated with the speech content of the target speech.

[0133] Optionally, the process of obtaining knowledge information of the second target entity from the knowledge graph of the specified domain may include: obtaining knowledge information of the second target entity at two or more levels from the knowledge graph of the specified domain, that is, knowledge nodes of the second target entity at two or more levels and relationship information between the second target entity and knowledge nodes at two or more levels, thereby obtaining knowledge information of the second target entity.

[0134] Considering that the second target entity is an entity detected from the speech recognition results at a historical decoding moment, the knowledge information of the second target entity at two or more levels is more relevant to the speech content of the speech to be recognized. Therefore, the knowledge information of the second target entity at two or more levels can be obtained from the knowledge graph of the specified domain.

[0135] In one possible implementation, the speech recognition method provided in the above embodiments can be implemented based on a pre-trained speech recognition model. It should be noted that the model-based implementation of the speech recognition method provided in the above embodiments is merely an example, and the present invention does not limit the specific implementation form of the speech recognition method.

[0136] Specifically, the process of implementing speech recognition based on a speech recognition model can include:

[0137] Step a1: Obtain the target speech and target video in the specified domain.

[0138] The target speech is the speech to be recognized, which is the narration of the video content of the target video. The target video is used to assist in the recognition of the target speech.

[0139] Step a2: Input the target speech and target video into the pre-trained speech recognition model to obtain the speech recognition result of the target speech.

[0140] After the target speech and target video are input into the speech recognition model, the speech recognition model detects entities from the video frame of the target video to obtain the first target entity, obtains the knowledge information of the first target entity from the knowledge graph of the specified domain, and performs speech recognition on the target speech with the knowledge information of the first target entity, and outputs the speech recognition result of the target speech.

[0141] Optionally, when determining the speech recognition result at each decoding moment, the speech recognition model can also detect entities from the speech recognition results at historical decoding moments to obtain a second target entity, and obtain knowledge information of the second target entity from the knowledge graph of the specified domain. Then, with the knowledge information of the first target entity and the knowledge information of the second target entity, the recognition result of the target speech at the current decoding moment can be determined.

[0142] The aforementioned speech recognition model is trained using training data from a training set within a specified domain. The training data includes training speech with labeled text and training videos with labeled entities (the entities labeled in the training videos are the entities within the video frames). The training speech is the narration from the training video frames. Optionally, the labeled text for the training speech can be obtained by first inputting the training speech into a pre-trained general speech recognition model to obtain the recognition result, and then manually correcting the result. This method has high labeling efficiency. However, this embodiment is not limited to this; the training speech can also be directly labeled manually.

[0143] The training objectives of the speech recognition model include: making the speech recognition results obtained by performing speech recognition on the training speech using entity knowledge information corresponding to the training video as a supplement, and making the entities detected from the video frames of the training video as consistent with the entities labeled in the training video. Here, the entity knowledge information corresponding to the training video refers to the knowledge information of the entities detected from the video frames of the training video.

[0144] The speech recognition method based on a speech recognition model provided in this invention can detect entities from the video frame of a target video and obtain knowledge information of the detected entities from a knowledge graph of a specified domain. Then, the acquired entity knowledge information can be used to perform speech recognition on the target speech. When performing speech recognition on the target speech, the knowledge information of the entities detected from the video frame of the target video can be used to obtain a more accurate speech recognition result.

[0145] In another embodiment of the present invention, the training process of the speech recognition model in the above embodiment is described.

[0146] Please see Figure 6 The diagram illustrates the process of training a speech recognition model, which may include:

[0147] Step S601: Obtain training data from the training set.

[0148] The training set includes multiple training data sets. Each training data set includes training audio with text labeled with audio content and training video with entities labeled. The training audio is the narration audio of the training video in the training data set.

[0149] Step S602: Based on the speech recognition model, detect entities from the video frames of the training videos in the acquired training data to obtain the entity detection results corresponding to the training videos.

[0150] Figure 7This embodiment illustrates one structure of a speech recognition model, supplemented by... Figure 7 The model structure shown illustrates the training process of the speech recognition model. It should be noted that... Figure 7 The model structure shown is merely an example, and this embodiment does not limit the structure of the speech recognition model. Figure 7 Any structure shown that can detect entities from the video frame of the input video, obtain knowledge information of the detected entities from a knowledge graph of a specified domain, and perform speech recognition on the input speech using the obtained entity knowledge information, falls within the scope of protection of this invention.

[0151] like Figure 7 As shown, the speech recognition model may include a visual encoding module 701 and a first entity detection module 702. The training video is input into the visual encoding module 701, which encodes the input training video and outputs visual features of the training video (the visual features of the training video can characterize the video frame information of the training video). The visual features output by the visual encoding module 701 are input into the first entity detection module 702, which extracts entities from the video frames of the training video based on the visual features output by the visual encoding module 701.

[0152] Optionally, the visual encoding module 701 may use a YOLOv5s network. It should be noted that this embodiment does not limit the visual encoding module 701 to use a YOLOv5s network; any network that can encode video into visual features is acceptable.

[0153] Step S603: Based on the speech recognition model, obtain the knowledge information of the detected entities from the knowledge graph of the specified domain, and use it as the entity knowledge information corresponding to the training video.

[0154] like Figure 7 As shown, the speech recognition model may include a first entity knowledge acquisition module 703. Entities detected from the video frames of the training video are input into the first entity knowledge acquisition module 703, which then extracts and outputs the knowledge information of the input entities from a knowledge graph of a specified domain.

[0155] Optionally, the first entity knowledge acquisition module 703 can extract the first-level knowledge node information of the input entity (including the first-level knowledge nodes of the input entity and the relationship information between the first-level knowledge nodes of the input entity) from the knowledge graph of the specified domain, as the knowledge information of the input entity.

[0156] Step S604: Based on the speech recognition model and supplemented by entity knowledge information corresponding to the training video, perform speech recognition on the training speech in the acquired training data to obtain the speech recognition result of the training speech.

[0157] like Figure 7 As shown, the speech recognition model may include a speech encoding module 704, a feature fusion module 705, and a speech recognition decoding module 706. Training speech is input into the speech encoding module 704 for encoding, and the speech encoding module 704 outputs the speech features of the training speech (the speech features of the training speech can represent the speech content information of the training speech). The speech features of the training speech and the visual features of the training video are input into the feature fusion module 705. The feature fusion module 705 fuses the input speech features and input visual features. For example, the feature fusion module 705 can determine the weights corresponding to the speech features and visual features respectively, and then sum the speech features and visual features according to the determined weights to obtain the fused features. It should be noted that the sum of the weights corresponding to the speech features and visual features is 1. The larger the weight of the visual feature, the greater the correlation between the video content and the speech content; conversely, the smaller the weight of the visual feature, the smaller the correlation between the video content and the speech content. The above fusion method enables the model to focus on using speech features to determine the speech recognition result when the video content and the speech content are weakly correlated or uncorrelated. The fused features output by the feature fusion module 705 are input to the speech recognition decoding module 706 for decoding.

[0158] Optionally, the speech coding module 704 may use a 16-layer Conformer network. It should be noted that this embodiment does not limit the speech coding module 704 to use a 16-layer Conformer network; any network that can encode speech into speech features is acceptable.

[0159] Optionally, the speech encoding module can process the input speech with a 10ms frame shift and a 25ms window. Similarly, the visual encoding module processes the input video at 25fps, that is, extracts 25 frames from a 1s video stream and then encodes the 25-frame image sequence.

[0160] In one possible implementation, the visual features output by the visual encoding module 701 can be input to the feature fusion module 705 and the first entity detection module 702, respectively. That is, the visual features output by the visual encoding module 701 participate in both the entity detection task and the speech recognition task. To avoid the two tasks interfering with each other due to using the same visual features, in another possible implementation, the visual features output by the visual encoding module 701 can be processed, and the processed visual features can be input to the feature fusion module 705. That is, the visual features used in the speech recognition task are the processed visual features output by the visual encoding module 701, while the visual features used in the entity detection task are the visual features output by the visual encoding module 701. Figure 7As shown, the speech recognition model may further include a feature processing module 707. The visual features output by the visual encoding module 701 are input into the feature processing module 707 for processing and then input into the feature fusion module 705. The feature processing module 707 can process the input visual features into visual features that can better represent the video image information. Optionally, the feature processing module 705 may employ a convolutional neural network (CNN).

[0161] like Figure 7 As shown, the speech recognition model also includes a first entity knowledge encoding module 708. The entity knowledge information output by the first entity knowledge acquisition module 703 is input into the first entity knowledge encoding module 708 for encoding. The first entity knowledge encoding module 708 outputs entity knowledge features that represent the input entity knowledge information, which are used as entity knowledge features corresponding to the training video.

[0162] like Figure 7 As shown, the speech recognition model also includes a second entity detection module 709, a second entity knowledge acquisition module 710, and a second entity knowledge encoding module 711. The second entity detection module 709 detects entities from the speech recognition results of the training speech at historical decoding times. The entities detected by the second entity detection module 709 are input into the second entity knowledge acquisition module 711. The second entity knowledge acquisition module 711 extracts knowledge information of the input entities from a knowledge graph of a specified domain and outputs it. The entity knowledge information output by the second entity knowledge acquisition module 710 is input into the second entity knowledge encoding module 711. The second entity knowledge encoding module 711 encodes the input entity knowledge information and outputs entity knowledge features representing the input entity knowledge information.

[0163] like Figure 7 As shown, the fused features output by the feature fusion module 705, the entity knowledge features output by the first entity knowledge encoding module 708, and the entity knowledge features output by the second entity knowledge encoding module 711 are input together with the fused features output by the feature fusion module 705 into the speech recognition decoding module 706. The speech recognition decoding module 706 determines the features required for decoding at the current decoding time based on the input features, and determines the speech recognition result of the target speech at the current decoding time based on the features required for decoding at the current decoding time.

[0164] The speech recognition decoding module 706 may include an attention module and a decoder. The attention module determines the features required for decoding at the current decoding time (i.e., features related to the speech recognition result to be predicted at the current decoding time) based on the fused features output by the feature fusion module 705, the entity knowledge features output by the first entity knowledge encoding module 708, and the entity knowledge features output by the second entity knowledge encoding module 711. The decoder decodes the features required for decoding at the current decoding time to obtain the speech recognition result of the target speech at the current decoding time. Optionally, the decoder may be a language model based on acoustic conditions.

[0165] Step S605: Determine the first loss based on the speech recognition results of the training speech and the speech content text labeled in the training speech, and determine the second loss based on the entity detection results corresponding to the training video and the entities labeled in the training video.

[0166] The first loss is the loss of the speech recognition model on the speech recognition task, and the second loss is the loss of the speech recognition model on the entity detection task.

[0167] Step S606: Update the parameters of the speech recognition model based on the first loss and the second loss.

[0168] Specifically, the first loss and the second loss are fused, and the parameters of the speech recognition model are updated based on the fused loss.

[0169] There are several ways to fuse the first loss and the second loss. In one possible implementation, the first loss and the second loss can be directly summed. In another possible implementation, the first loss and the second loss can be weighted and summed. The fusion method of weighted summation is shown in the following formula:

[0170] Loss tot0l =λLoss asr +θLoss ed (1)

[0171] Among them, Loss asr This represents the first prediction loss, i.e., the loss of the speech recognition model on the speech recognition task. ed Let λ represent the second prediction loss, i.e., the loss of the speech recognition model on the entity detection task. asr The corresponding weights, θ is the loss. ed The corresponding weights, λ and θ, can be set according to the specific scenario. Loss total This refers to the loss after fusion.

[0172] Repeat steps S601 to S606 until the training termination conditions are met, such as model convergence or reaching the set number of training iterations.

[0173] To fully utilize video content information and improve the recognition effect of narration, this invention trains a speech recognition model using data from both the narrator's voice and the video footage they are narrating. Therefore, the speech recognition model in this invention is a multimodal speech recognition model. Compared to multimodal speech recognition models that rely on lip reading for recognition, this invention's model uses knowledge of entities within the video footage to identify the narration, resulting in better recognition performance. Furthermore, since this invention does not require capturing the narrator's face via camera, it avoids infringing on the narrator's privacy, thus broadening its application scope.

[0174] After obtaining the trained speech recognition model, it can be used for speech recognition. For the target speech and target video, both can be input into the trained speech recognition model for processing to obtain the speech recognition result of the target speech. The process of the speech recognition model processing the target speech and target video is similar to the process of processing the training speech and training video described above, and will not be described in detail here.

[0175] In one possible implementation, a large amount of labeled training data can be used to train the speech recognition model to achieve better performance. However, considering that acquiring a large amount of labeled training data is time-consuming and costly, another possible implementation is to pre-train the speech coding module, visual coding module, and feature fusion module with a large amount of unlabeled training data (e.g., 100 hours of unlabeled training data). Then, a model is constructed that includes the above modules (trained to obtain the speech coding module, visual coding module, and feature fusion module), feature processing module, first entity detection module, first entity knowledge acquisition module, first entity knowledge encoding module, second entity detection module, second entity knowledge acquisition module, second entity knowledge encoding module, and speech recognition decoding module as the initial speech recognition model. Finally, a small amount of labeled training data (e.g., 10 hours of labeled training data) is used to fine-tune the initial speech recognition model to obtain a speech recognition model with better performance.

[0176] For the second implementation method mentioned above, in addition to obtaining the training set mentioned in the above embodiments, which includes a training set with labeled training data (the above training set is used as the first training set), a second training set is also required. The second training set includes multiple training data sets. Each training data set in the second training set includes unlabeled training audio and unlabeled training video. The training audio is the narration audio of the video content of the training video in its training data set.

[0177] The following section describes the process of training the speech coding module, visual coding module, and feature fusion module using the training data from the second training set.

[0178] The process of training the speech coding module, visual coding module, and feature fusion module using training data from the second training set may include:

[0179] Step b1: For each piece of training data in the second training set, obtain the speech features of the training speech in the training data based on the pre-trained general speech recognition model, and use them as the speech features corresponding to the training data.

[0180] The speech features corresponding to each training data in the second training set can be obtained through step b1.

[0181] Step b2: Cluster the speech features corresponding to each training data in the second training set to obtain multiple speech features, and assign a category label to each speech feature. The category label of each speech feature is then determined as the category label of the corresponding training data.

[0182] Existing clustering methods (such as K-Means clustering) can be used to cluster the speech features corresponding to each training data in the second training set. Through clustering, multiple classes of speech features can be obtained. After obtaining multiple classes of speech features, a category label can be assigned to each class of speech features. For example, if four classes of speech features are obtained through clustering, category labels "1", "2", "3", and "4" can be assigned to the four classes of speech features respectively. After setting the category label for each class of speech features, the category label of each speech feature can be used as the category label of the corresponding training data. For example, if the category label of the speech feature corresponding to a training data in the second training set is "1", then the category label "1" can be used as the category label of that training data.

[0183] The category label for each training data point in the second training set can be obtained through step b2.

[0184] Step b3: Combine the training data with class labels into a third training set.

[0185] Step b4: Using the third training set, combined with the data classification task, train the speech coding module, the visual coding module, and the feature fusion module.

[0186] Specifically, the process of training the speech coding module, visual coding module, and feature fusion module using the third training set and in conjunction with the data classification task can include:

[0187] Step b41: Construct a data classification model that includes a speech coding module, a visual coding module, a feature fusion module, and a classification module.

[0188] Step b42: Obtain training data from the third training set.

[0189] Step b43: Use the speech encoding module of the training speech input data classification model in the acquired training data to obtain the speech features of the training speech, and use the visual encoding module of the training video input data classification model in the acquired training data to obtain the visual features of the training video.

[0190] Step b44: Input the speech features of the training speech and the visual features of the training video into the feature fusion module of the data classification model to obtain the fused features corresponding to the training data.

[0191] The feature fusion module of the data classification model performs feature fusion on the input speech features and visual features, and outputs the fused features corresponding to the training data.

[0192] Step b45: Input the fusion features corresponding to the training data into the classification module of the data classification model to obtain the category prediction results of the training data.

[0193] The classification module of the data classification model predicts the category of the training data based on the input fusion features and outputs the category prediction results of the training data.

[0194] Step b46: Determine the category prediction loss based on the category prediction results and category labels of the training data.

[0195] Optionally, the category prediction loss can be the cross-entropy loss. The calculation method of the cross-entropy loss is a prior art, and will not be described in detail in this embodiment.

[0196] Step b47: Predict the loss based on the category and update the parameters of the data classification model.

[0197] Repeat steps b42 to b47 until the training termination condition is met (such as model convergence or reaching the set number of training iterations).

[0198] After training, the trained speech coding module, visual coding module, and feature fusion module can be obtained.

[0199] This invention also provides a speech recognition device. The speech recognition device provided in this invention will be described below. The speech recognition device described below can be referred to in correspondence with the speech recognition method described above.

[0200] Please see Figure 8 The diagram shows a schematic of the structure of a speech recognition device provided in an embodiment of the present invention, which may include: a data acquisition module 801, a recognition auxiliary information acquisition module 802, and a speech recognition result determination module 803.

[0201] The data acquisition module 801 is used to acquire target speech and target video in a specified field, wherein the target speech is the narration speech of the video content of the target video.

[0202] The identification auxiliary information acquisition module 802 is used to detect entities from the video frame of the target video, obtain a first target entity, and obtain knowledge information of the first target entity from the knowledge graph of the specified domain.

[0203] The speech recognition result determination module 803 is used to perform speech recognition on the target speech with the aid of the knowledge information of the first target entity, and obtain the speech recognition result of the target speech.

[0204] Optionally, the identification auxiliary information acquisition module 802 includes: a video encoding module and a first entity detection module.

[0205] The video encoding module is used to encode the target video into features that can characterize the video frame information of the target video, thereby obtaining the visual features of the target video.

[0206] The first entity detection module is used to extract entities from the video frames of the target video based on the visual features of the target video, and obtain the first target entity.

[0207] Optionally, the speech recognition result determination module 803 includes: a speech encoding module, a feature fusion module, a first entity knowledge encoding module, and a speech recognition decoding module.

[0208] A speech encoding module is used to encode the target speech into features that can characterize the speech content information of the target speech, thereby obtaining the speech features of the target speech.

[0209] The feature fusion module is used to fuse the speech features of the target speech with the visual features of the target video to obtain fused features.

[0210] The first entity knowledge encoding module is used to encode the knowledge information of the first target entity into features that can characterize the knowledge information of the first target entity, thereby obtaining the first entity knowledge features.

[0211] The speech recognition decoding module is used to determine the speech recognition result of the target speech based on the fusion features and the first entity knowledge features.

[0212] Optionally, when determining the speech recognition result of the target speech based on the fused features and the first entity knowledge features, the speech recognition decoding module is specifically used for:

[0213] At each decoding moment: based on the fused features and the first entity knowledge features, determine the features required for decoding at that decoding moment; based on the features required for decoding at that decoding moment, determine the speech recognition result of the target speech at that decoding moment.

[0214] Optionally, the speech recognition device provided in this embodiment of the invention further includes: a second entity detection module, a second entity knowledge acquisition module, and a second entity knowledge encoding module.

[0215] The second entity detection module is used to detect entities from the speech recognition results of the target speech at historical decoding times to obtain the second target entity.

[0216] The second entity knowledge acquisition module is used to acquire knowledge information of the second target entity from the knowledge graph of the specified domain.

[0217] The second entity knowledge encoding module is used to encode the knowledge information of the second target entity into features that can characterize the knowledge information of the second target entity, thereby obtaining the second entity knowledge features.

[0218] When the speech recognition decoding module determines the features required for decoding at the current decoding moment based on the fused features and the first entity knowledge features, it is specifically used to determine the features required for decoding at the current decoding moment based on the fused features, the first entity knowledge features, and the second entity knowledge features.

[0219] Optionally, the speech recognition device provided in this embodiment of the invention may further include: a knowledge graph acquisition module. The knowledge graph acquisition module is used for:

[0220] Obtain multiple narration audio clips with text labeled with audio content in the specified domain;

[0221] The text of the audio content annotated from multiple narrations is regularized, and the regularized text is segmented into multiple words.

[0222] By combining word frequency, several entity words are obtained from the multiple words, and the relationship information between the several entity words is obtained;

[0223] Based on the aforementioned entity words and the relationship information between them, a knowledge graph for the specified domain is constructed. The knowledge graph includes several knowledge nodes and directed edges between them. Each knowledge node is an entity word. There is a relationship between the two knowledge nodes connected by each directed edge. Each directed edge contains the relationship information between the two knowledge nodes connected by the directed edge.

[0224] Optionally, when the first entity knowledge acquisition module obtains the knowledge information of the first target entity from the knowledge graph of the specified domain, it is specifically used to extract the first-level knowledge nodes of the first target entity and the relationship information between the first target entity and the first-level knowledge nodes from the knowledge graph of the specified domain to obtain the knowledge information of the first target entity.

[0225] Optionally, the second entity knowledge acquisition module obtains knowledge information of the second target entity from the knowledge graph of the specified domain. Specifically, it extracts two or more levels of knowledge nodes of the second target entity and the relationship information between the second target entity and the two or more levels of knowledge nodes from the knowledge graph of the specified domain to obtain the knowledge information of the second target entity.

[0226] Optionally, the implementation process of the auxiliary information acquisition module 802 and the speech recognition result determination module 803 can be implemented through a speech recognition model. Specifically, the target speech and the target video are processed using a pre-trained speech recognition model to obtain the speech recognition result of the target speech. The speech recognition model is trained using training data from the training set of the specified domain. The training data includes training speech with labeled text and training videos with labeled entities. The training speech is the narration of the video content of the training video.

[0227] The speech recognition device provided in this embodiment of the invention may further include: a model training module. The model training module is specifically used for:

[0228] Based on the speech recognition model, entities are detected from the video frames of the training video to obtain the entity detection results corresponding to the training video, and the knowledge information of the detected entities is obtained from the knowledge graph of the specified domain to obtain the entity knowledge information corresponding to the training video.

[0229] Based on the speech recognition model, and supplemented by entity knowledge information corresponding to the training video, speech recognition is performed on the training speech to obtain the speech recognition result of the training speech;

[0230] A first loss is determined based on the speech recognition results of the training speech and the speech content text labeled in the training speech, and a second loss is determined based on the entity detection results corresponding to the training video and the entities labeled in the training video.

[0231] The parameters of the speech recognition model are updated based on the first loss and the second loss.

[0232] Considering that the target speech is the narration of the target video, and that it has a certain correlation with the content of the target video, the speech recognition device provided in this embodiment of the invention detects entities from the video frame of the target video and obtains knowledge information of the detected entities from a knowledge graph of a specified domain. This knowledge information is then used to perform speech recognition on the target speech. By supplementing the speech recognition with the knowledge information of the entities detected from the video frame of the target video, a more accurate speech recognition result can be obtained. The speech recognition device provided in this embodiment of the invention improves the user experience of speech recognition services in narration scenarios.

[0233] This invention also provides a voice recognition device; please refer to [link to relevant documentation]. Figure 9 The diagram shows the structure of the voice recognition device, which may include: at least one processor 901, at least one communication interface 902, at least one memory 903 and at least one communication bus 904.

[0234] In this embodiment of the invention, the number of processor 901, communication interface 902, memory 903, and communication bus 904 is at least one, and processor 901, communication interface 902, and memory 903 communicate with each other through communication bus 904.

[0235] The processor 901 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0236] The memory 903 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0237] The memory stores a program, which the processor can call. The program is used for:

[0238] Acquire target speech and target video in a specified domain, wherein the target speech is the narration of the video content of the target video;

[0239] Entities are detected from the video frame of the target video to obtain the first target entity;

[0240] Obtain knowledge information of the first target entity from the knowledge graph of the specified domain;

[0241] Using the knowledge information of the first target entity, speech recognition is performed on the target speech to obtain the speech recognition result of the target speech.

[0242] Optionally, the refined and extended functions of the program can be found in the description above.

[0243] This invention also provides a computer-readable storage medium that stores a program suitable for execution by a processor, the program being used for:

[0244] Acquire target speech and target video in a specified domain, wherein the target speech is the narration of the video content of the target video;

[0245] Entities are detected from the video frame of the target video to obtain the first target entity;

[0246] Obtain knowledge information of the first target entity from the knowledge graph of the specified domain;

[0247] Using the knowledge information of the first target entity, speech recognition is performed on the target speech to obtain the speech recognition result of the target speech.

[0248] Optionally, the refined and extended functions of the program can be found in the description above.

[0249] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0250] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0251] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that, include: Acquire target speech and target video in a specified domain, wherein the target speech is the narration of the video content of the target video; The first target entity is obtained by directly detecting entities from the video frame of the target video; Obtain knowledge information of the first target entity from the knowledge graph of the specified domain; Entities are detected from the historical decoding results of the target speech to obtain a second target entity, and knowledge information of the second target entity is obtained from the knowledge graph of the specified domain. Using the knowledge information of the first target entity and the knowledge information of the second target entity, speech recognition is performed on the target speech to obtain the speech recognition result of the target speech.

2. The speech recognition method according to claim 1, characterized in that, The step of directly detecting entities from the video frame of the target video to obtain the first target entity includes: The target video is encoded into features that can characterize the video frame information of the target video to obtain the visual features of the target video; Based on the visual features of the target video, entities are extracted from the video frames of the target video to obtain the first target entity.

3. The speech recognition method according to claim 1, characterized in that, The process of using the knowledge information of the first target entity and the knowledge information of the second target entity to perform speech recognition on the target speech, and obtaining the speech recognition result of the target speech, includes: The target speech is encoded into features that can characterize the speech content information of the target speech, thereby obtaining the speech features of the target speech; The speech features of the target speech are fused with the visual features of the target video to obtain fused features; Encode the knowledge information of the first target entity into features that can characterize the knowledge information of the first target entity to obtain the knowledge features of the first entity; Encode the knowledge information of the second target entity into features that can characterize the knowledge information of the second target entity to obtain the knowledge features of the second entity; The speech recognition result of the target speech is determined based on the fusion features, the first entity knowledge features, and the second entity knowledge features.

4. The speech recognition method according to claim 3, characterized in that, The step of determining the speech recognition result of the target speech based on the fusion features and the first entity knowledge features includes: At each decoding moment: Based on the fusion features, the first entity knowledge features, and the second entity knowledge features, determine the features required for decoding at this decoding moment; Based on the features required for decoding at that decoding moment, the speech recognition result of the target speech at that decoding moment is determined.

5. The speech recognition method according to claim 1, characterized in that, The process of acquiring the knowledge graph for the specified domain includes: Obtain multiple narration audio clips with text labeled with audio content in the specified domain; The text of the audio content annotated from multiple narrations is regularized, and the regularized text is segmented into multiple words. By combining word frequency, several entity words are obtained from the multiple words, and the relationship information between the several entity words is obtained; Based on the aforementioned entity words and the relationship information between them, a knowledge graph for the specified domain is constructed. The knowledge graph includes several knowledge nodes and directed edges between them. Each knowledge node is an entity word. There is a relationship between the two knowledge nodes connected by each directed edge. Each directed edge contains the relationship information between the two knowledge nodes connected by the directed edge.

6. The speech recognition method according to claim 5, characterized in that, The step of obtaining knowledge information of the first target entity from the knowledge graph of the specified domain includes: The knowledge information of the first target entity is obtained by extracting the first-level knowledge nodes of the first target entity and the relationship information between the first target entity and the first-level knowledge nodes from the knowledge graph of the specified domain.

7. The speech recognition method according to claim 1, characterized in that, The knowledge graph includes several knowledge nodes and directed edges between knowledge nodes. Each knowledge node is an entity word. There is a relationship between the two knowledge nodes connected by each directed edge. Each directed edge contains the relationship information between the two knowledge nodes connected by the directed edge. The step of obtaining knowledge information about the second target entity from the knowledge graph of the specified domain includes: The knowledge information of the second target entity is obtained by extracting two or more levels of knowledge nodes of the second target entity and the relationship information between the second target entity and the two or more levels of knowledge nodes from the knowledge graph of the specified domain.

8. The speech recognition method according to any one of claims 1 to 7, characterized in that, The process involves detecting entities from the video frame of the target video to obtain a first target entity; acquiring knowledge information of the first target entity from a knowledge graph of a specified domain; detecting entities from the historical decoding results of the target speech to obtain a second target entity, and acquiring knowledge information of the second target entity from the knowledge graph of the specified domain; and using the knowledge information of the first and second target entities to perform speech recognition on the target speech to obtain a speech recognition result for the target speech, including: The target speech and the target video are processed based on a pre-trained speech recognition model to obtain the speech recognition result of the target speech; The speech recognition model is trained using training data from the training set in the specified domain. The training data includes training speech with text labeled with speech content and training videos labeled with entities. The training speech is the narration of the video content of the training video.

9. The speech recognition method according to claim 8, characterized in that, The training process of the speech recognition model includes: Based on a speech recognition model, entities are detected directly from the video frames of the training video to obtain entity detection results corresponding to the training video. Knowledge information of the entities contained in the entity detection results is obtained from the knowledge graph of the specified domain to obtain first entity knowledge information corresponding to the training video. Entities are also detected from the historical decoding results of the training speech, and knowledge information of the entities detected from the historical decoding results of the training speech is obtained from the knowledge graph of the specified domain to obtain second entity knowledge information corresponding to the training video. Based on the speech recognition model, and supplemented by the first entity knowledge information and the second entity knowledge information corresponding to the training video, speech recognition is performed on the training speech to obtain the speech recognition result of the training speech; A first loss is determined based on the speech recognition results of the training speech and the speech content text labeled in the training speech, and a second loss is determined based on the entity detection results corresponding to the training video and the entities labeled in the training video. The parameters of the speech recognition model are updated based on the first loss and the second loss.

10. A voice recognition device, characterized in that, include: The module consists of a data acquisition module, a recognition auxiliary information acquisition module, and a speech recognition result determination module. The data acquisition module is used to acquire target speech and target video in a specified field, wherein the target speech is the narration speech of the video content of the target video; The recognition auxiliary information acquisition module is used to directly detect entities from the video frame of the target video to obtain a first target entity, and obtain knowledge information of the first target entity from the knowledge graph of the specified domain; and to detect entities from the historical decoding results of the target speech to obtain a second target entity, and obtain knowledge information of the second target entity from the knowledge graph of the specified domain. The speech recognition result determination module is used to perform speech recognition on the target speech by supplementing the knowledge information of the first target entity and the knowledge information of the second target entity, so as to obtain the speech recognition result of the target speech.

11. A voice recognition device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the speech recognition method as described in any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the speech recognition method as described in any one of claims 1 to 9.

Images