Recording abstract generation method and device, equipment and medium
By combining voiceprint feature extraction and knowledge graph technology with dynamic graph optimization, the problem of misinterpretation of technical terms in audio recording summaries is solved, generating logically coherent audio recording summaries, improving efficiency and accuracy, and reducing the need for manual correction.
Patent Information
- Application Number
- CN202511001922.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-21
- Publication Date
- 2025-11-04
AI Technical Summary
Existing audio summarization technologies are prone to misinterpretation and misjudgment of technical terms during the generation process, which can lead to broken logical chains in the generated summaries. This requires manual correction, resulting in low efficiency and high costs.
By employing a voiceprint feature extraction model, a pre-trained natural language parsing model, and a knowledge graph, combined with dynamic graph optimization, a directed acyclic graph is constructed through spectral feature extraction, semantic embedding vectors, and term boundary detection. This process repairs broken markers and generates logically coherent audio summaries.
It improves the accuracy and structure of audio summarization, reduces the need for manual correction, enhances information processing efficiency, and lowers labor costs.
Smart Images
Figure CN120895040A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of natural language processing, and in particular to a recording summary generation method and device based on natural language processing, equipment and a medium. BACKGROUND
[0002] The principle of extracting a recording summary is to extract the content as summary information after converting a telephone recording into text, which is realized by using computer technology. At present, the application field is mostly a summary extraction process based on speech. The current mainstream recording summary technology is mostly focused on keyword extraction or end-to-end generation at the text level. In the extraction process, mostly only the text summary extraction process is used. In some professional fields, different people express professional terms differently, or some sentences are not recorded in the recording process. Therefore, in the process of generating a summary, the extracted text content cannot be further understood and corrected, so it is easy to cause misreading and misjudgment of professional terms, the logical chain of the generated summary text is broken, and the generated result often deviates greatly from the result that the user wants. The user needs to manually correct the summary text, which leads to low efficiency of information processing and increases the labor cost. SUMMARY
[0003] The embodiments of the present application provide a recording summary generation method, device, equipment and medium based on natural language processing, which combines voiceprint feature extraction, knowledge graph semantic enhancement and dynamic graph optimization method, reduces the probability of manual correction of the summary, and improves the accuracy and structured degree of the summary.
[0004] In a first aspect, the embodiments of the present application provide a recording summary generation method, comprising:
[0005] receiving an audio stream, and converting the audio stream into speech text by a voiceprint feature extraction model;
[0006] inputting the speech text into a pre-trained natural language parsing model to obtain a candidate term corresponding to the speech text; the natural language parsing model is a neural network model trained based on a labeled sample speech text;
[0007] matching the candidate term with a knowledge graph to obtain a term relationship path of the candidate term;
[0008] constructing a directed acyclic graph including a broken mark according to the term relationship path, and repairing the broken mark based on the context of the speech text;
[0009] generating a recording summary corresponding to the audio stream according to the candidate term and the directed acyclic graph.
[0010] In a possible implementation manner of the first aspect, the receiving of the audio stream and the conversion of the audio stream into speech text by the voiceprint feature extraction model comprises:
[0011] preprocessing the audio stream, extracting spectral features through a convolutional layer of the voiceprint feature extraction model and outputting a spectral feature matrix of the audio stream;
[0012] capturing time sequence dependence of the spectral feature matrix through a recurrent neural network layer of the voiceprint feature extraction model, and obtaining a time sequence encoding vector of the audio stream
[0013] fusing multi-scale features of the time sequence encoding vector through an attention mechanism, and obtaining an output text probability distribution of the audio stream;
[0014] generating speech text according to the output text probability distribution of the audio stream.
[0015] In a possible implementation manner of the first aspect, the speech text is input into a pre-trained natural language parsing model to obtain candidate terms corresponding to the speech text, including:
[0016] inputting the speech text into a basic layer of the natural language parsing model to obtain a semantic embedding vector of the speech text;
[0017] inputting the semantic embedding vector into a term boundary detector of the natural language parsing model to obtain a position mark of the candidate term;
[0018] outputting a candidate term set corresponding to the speech text according to the position mark of the candidate term.
[0019] In a possible implementation manner of the first aspect, the knowledge graph includes: a term association graph, an incremental update layer and a cross-domain association index;
[0020] matching the candidate term with the knowledge graph to obtain a term relationship path of the candidate term, including:
[0021] querying an initial association path corresponding to the candidate term in the candidate term set in the term association graph;
[0022] inputting the initial association path into the incremental update layer to determine an expanded association path of the candidate term;
[0023] based on the cross-domain association index, performing semantic alignment on term relationships of candidate terms in different fields to obtain a cross-domain alignment path of the candidate term;
[0024] outputting a term relationship path in which the expanded association path and the cross-domain alignment path of the candidate term are fused.
[0025] In a possible implementation manner of the first aspect, a directed acyclic graph including a break mark is constructed, including:
[0026] initializing an empty directed acyclic graph, and taking the candidate term as a node;
[0027] According to the term relationship path of the candidate term, semantic similarity between term nodes is calculated as an edge weight;
[0028] By means of a dynamic programming algorithm, a non-connected subgraph in the directed acyclic graph is marked for rupture, and the missing information area is identified;
[0029] By means of a reverse filling mechanism, the term relationship at the rupture marking is repaired by using context semantic information.
[0030] In a possible implementation manner of the first aspect, repairing the term relationship at the rupture marking by using context semantic information comprises the following steps:
[0031] The rupture area in the directed acyclic graph is identified, and the semantic distance on both sides of the rupture is calculated;
[0032] Supplementary terms semantically related to the rupture area are retrieved from the knowledge graph;
[0033] The insertion probability of the supplementary terms is calculated by means of a probabilistic graph model, and the optimal path is selected to fill in the rupture;
[0034] The edge weight of the directed acyclic graph is updated.
[0035] In a possible implementation manner of the first aspect, generating the audio stream corresponding recording summary comprises the following steps:
[0036] Key term nodes and their path weights are extracted from the directed acyclic graph;
[0037] Based on the path weight ordering, a key term list is generated;
[0038] The continuous node sequence in the directed acyclic graph is clustered into events to generate an event summary;
[0039] A timestamp is added to each event summary, and the timestamp is used to mark the start and end time points of the event corresponding to the audio stream.
[0040] The first aspect of the present application has the beneficial effect compared with the prior art: by fusing the voiceprint feature, semantic analysis and knowledge graph technology, the robustness of speech to text is improved, and then the semantic association of the term is enhanced by using the knowledge graph, the missing of key information is reduced, and the logical coherence and integrity of the summary are ensured by means of the construction and rupture repair technology of the dynamic graph; the structural degree and practicability of the summary are further improved.
[0041] In a second aspect, the present application provides a recording summary generation device based on natural language processing, comprising:
[0042] A voiceprint feature extraction module is configured to receive an audio stream and convert the audio stream into speech text by means of a voiceprint feature extraction model;
[0043] The term analysis module is configured to input the speech text into a pre-trained natural language analysis model to obtain candidate terms corresponding to the speech text; the natural language analysis model is a neural network model trained based on annotated sample speech texts;
[0044] The term relationship matching module is configured to match the candidate terms with a knowledge graph to obtain a term relationship path of the candidate terms;
[0045] The dynamic graph construction module is configured to construct a directed acyclic graph including a broken mark according to the term relationship path;
[0046] The audio summary generation module is configured to generate an audio summary corresponding to the audio stream according to the candidate terms and the directed acyclic graph.
[0047] In a third aspect, an audio summary generation device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the device performs the method of any one of the first aspect.
[0048] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the method of any one of the first aspect is implemented.
[0049] It can be understood that the beneficial effects of the second aspect to the fourth aspect can be referred to the related description of the first aspect, which will not be repeated here.
[0050] Compared with the prior art, the beneficial effects of the embodiments of the present application are:
[0051] The present application extracts multi-dimensional features from the audio stream through the voiceprint feature extraction model, effectively captures spectral features and time sequence dependencies, improves the accuracy and efficiency of speech to text conversion, and reduces conversion errors. The pre-trained natural language analysis model is used to analyze the speech text, the semantic embedding vector and the term boundary detection are used to accurately locate and extract key candidate terms, irrelevant information interference is reduced, and the knowledge graph is used for term relationship matching to realize semantic alignment and dynamic expansion of cross-domain terms, enhance the comprehensiveness and accuracy of term association, and realize accurate judgment of professional terms. The generated summary text has good logical relationship, significantly reduces the labor cost, improves the information processing efficiency, and is suitable for quick content extraction and deep analysis in various different scenes. BRIEF DESCRIPTION OF DRAWINGS
[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.
[0053] Figure 1 is a flow diagram of the method for generating a recording summary based on natural language processing provided by the present application;
[0054] Figure 2 is a structural diagram of the device for generating a recording summary based on natural language processing provided by the embodiments of the present application;
[0055] Figure 3 is a structural diagram of the device for generating a recording summary based on natural language processing provided by the embodiments of the present application. DETAILED DESCRIPTION
[0056] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art will understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known devices, apparatuses, circuits, and methods are omitted in order not to obscure the description of the present application with unnecessary details.
[0057] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0058] It should also be understood that the term "and / or" as used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations thereof, and includes these combinations.
[0059] As used in the specification and the appended claims of the present application, the term "if" can be interpreted as "when" or "upon" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting [a described condition or event]" or "in response to detecting [a described condition or event]" depending on the context.
[0060] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions and cannot be understood as indicating or implying relative importance.
[0061] In the present application, the reference "one embodiment" or "some embodiments" means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in the present description are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but are not limited to", unless otherwise specifically emphasized.
[0062] Currently, the traditional method uses a simple speech recognition model (such as based on GMM-HMM or shallow neural network), which is sensitive to environmental noise, speaker accent, and speech rate changes, resulting in high error rate of transcribed text. The analysis of the transcribed text often relies on keyword matching or shallow syntax analysis, making it difficult to understand the deep logical relationship between terms (such as cause and effect, time sequence, hierarchy), for example, in the field of textiles, needle is a weaving method or a decorative technique to hide stitches, while in the medical field, it is a long-acting acupuncture point embedding therapy. At the same time, the pronunciation of needle and long needle is very similar, and in the process of sorting out the audio summary, it is easy to be considered as long needle, resulting in deviation in the subsequent understanding of the audio summary by the user. At the same time, when there are information gaps in the audio (such as sudden silence, background noise interference, or two people talking at the same time causing part of the content to be unable to be recognized), the summary is easy to miss key content, affecting the overall semantic coherence. Thus, there is a large difference between the generated audio summary and the actual audio content, which needs to be listened to again by the user, and the professional terms, logical relationships and missing content of the audio summary are further filled in an artificial form, resulting in low efficiency of the user in sorting out the audio summary.
[0063] Therefore, the present application provides a natural language processing-based audio summary generation method, which extracts multi-dimensional features from the audio stream through a voiceprint feature extraction model, effectively captures spectral features and time sequence dependencies, improves the accuracy and efficiency of speech-to-text conversion, and reduces conversion errors; a pre-trained natural language analysis model is used to analyze the speech text, key candidate terms are accurately positioned and extracted through semantic embedding vectors and term boundary detection, irrelevant information interference is reduced, and term relationship matching is performed in combination with a knowledge graph to realize semantic alignment and dynamic expansion of cross-domain terms, enhance the comprehensiveness and accuracy of term association, and thus realize accurate judgment of professional terms and improve the processing efficiency of audio summary.
[0064] The scheme provided in the present application will be described in detail below.
[0065] Figure 1 is a flowchart of the natural language processing-based audio summary generation method provided in the present application. Referring to Figure 1 , the method can include steps S1-S9, specifically including the following steps:
[0066] S1, receiving an audio stream, converting the audio stream into speech text through a voiceprint feature extraction model.
[0067] After receiving the audio stream, pre-processing (noise reduction, frame division, windowing) is first performed. The voiceprint feature extraction model adopts the following technical solution:
[0068] Acoustic features are extracted through a convolutional neural network (CNN), context dependence is captured in combination with a bidirectional long short-term memory network (Bi-LSTM), and a feature vector is output after weighting by an attention mechanism.
[0069] Illustratively, the CNN uses multiple convolutional layers (3 layers, 128 filters per layer, kernel size 5x5), and fuses Mel-frequency cepstral coefficients (MFCC) and first and second order differential coefficients to enhance dynamic feature expression.
[0070] Specifically, the CNN is trained using mixed data, i.e., training is performed using a combination of public datasets (such as LibriSpeech) and domain data (such as medical and conference recordings); wherein the CNN and Bi-LSTM are pre-trained on general data, and the attention mechanism is fine-tuned using domain data.
[0071] Then, illustratively, an Adam optimizer with a learning rate of 0.001 is used to fine-tune the CNN and Bi-LSTM in combination with a connectionist temporal classification (CTC) loss function.
[0072] Further, the attention mechanism can use a self-attention mechanism (such as Multi-Head Attention in Transformer) or an energy-based attention model (weighted by calculating the energy distribution of the frame) to improve the recognition ability for high signal-to-noise ratio frames.
[0073] S3, input the voice text into the pre-trained natural language parsing model to obtain the candidate terms corresponding to the voice text; the natural language parsing model is a neural network model trained based on the labeled sample voice text.
[0074] Specifically, the voice text is input into the pre-trained BERT model (such as BERT-base or domain adaptation version) for semantic embedding to obtain the context vector representation of each word. The context sensitivity of term recognition is improved through the semantic embedding vector, avoiding misrecognition caused by relying only on keyword matching.
[0075] Subsequently, a boundary detection algorithm (such as a sequence labeling model based on Transformer) is used to identify the position markers (such as "start-end" offsets) of the candidate terms in combination with a domain dictionary (such as a legal term library, a medical term library) and statistical features (such as word frequency, context entropy).
[0076] Among them, the term boundary detection reduces the term segmentation error (such as correctly identifying "natural language processing" as a whole term, rather than splitting it into "natural" and "language processing").
[0077] S5, match the candidate terms with the knowledge graph to obtain the term relationship path of the candidate terms.
[0078] The knowledge graph includes the following structures:
[0079] Term association graph: stores the static associations of terms in the field (such as "machine learning"→"neural network"→"backpropagation algorithm").
[0080] Incremental update layer: dynamically expands the path based on real-time data (such as the latest academic papers, industry reports) (for example, adds the association of "quantum computing" and "machine learning").
[0081] Cross-domain association index: maps terms from different domains to a unified semantic space through semantic alignment technology (such as Word2Vec, knowledge distillation) (such as establishing the association of "drug targets" and "biomarkers" in the medical and biotechnology fields).
[0082] Path construction process:
[0083] Retrieve the initial association path of the candidate terms in the knowledge graph.
[0084] The incremental update layer expands the path according to the latest data (such as the association of "algorithm optimization" to "edge computing").
[0085] The cross-domain association index generates cross-domain alignment paths through semantic similarity calculation (such as cosine similarity).
[0086] Specifically, the knowledge graph integrates the "Core Legal Terminology Library" and the "Case Association Rule Library". During dynamic graph construction:
[0087] (1) Take the key words of legal articles as root nodes (such as "burden of proof");
[0088] (2) When adjacent sentences appear associated terms (such as "inversion of the burden of proof") and voiceprint confidence is greater than 0.9, a strong weight edge is established;
[0089] (3) If the nodes on both sides of the logical fault contain associated terms such as "court cross-examination" and "evidence chain", automatically trigger cross-fault reconnection.
[0090] In another embodiment, in a recording related to AI pharmaceuticals, "molecular structure prediction" and "deep learning model" are related through cross-domain association indexing to form the path "molecular structure prediction -> bioinformatics -> deep learning model".
[0091] S7, according to the term relationship path, construct a directed acyclic graph including a break mark, and repair the break mark based on the context of the voice text;
[0092] The construction method of dynamic directed acyclic graph (DAG) is as follows:
[0093] The candidate terms are taken as nodes.
[0094] The edge weight of the candidate term is calculated based on the semantic similarity of the term relationship path (for example, by knowledge graph path length or node co-occurrence frequency).
[0095] The dynamic programming algorithm (such as shortest path search) identifies non-connected subgraphs (i.e. logical fault areas).
[0096] The semantic distance on both sides of the break is calculated, and the terms are supplemented from the knowledge graph; the probability of the candidate path is evaluated using a probabilistic graph model (such as a Bayesian network), the optimal path is selected to fill the break, and the edge weight is updated.
[0097] For example, the semantic association degree of the adjacent nodes of the logical fault is calculated where K i is the knowledge graph association degree, W i is the voiceprint confidence, if S is greater than or equal to a preset threshold (such as an association degree greater than or equal to 90%), a cross-fault bidirectional edge connection is established; otherwise, the fault is retained and a manual verification mark is marked.
[0098] S9, according to the candidate term and the directed acyclic graph, generate an audio stream corresponding to the recording summary.
[0099] According to the path weight of the nodes in the DAG (such as in-degree, out-degree, and total edge weight), a sorted list is generated.
[0100] The continuous key term nodes are clustered by topic (e.g., "technical principle", "experimental result", "application case") to form an event summary.
[0101] In combination with the time point information in the S1 step, the start and end times are added to each event summary to quickly locate the original audio content.
[0102] For example, the summary item is generated as: "15:00-15:30: Discuss algorithm optimization scheme, core terms: distributed training, GPU acceleration; 16:20-16:50: Show experimental results, key data: accuracy 92%, inference delay reduction 30%"
[0103] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0104] In the following, the technical solutions of the present application are further described in detail by combining specific scenarios:
[0105] Scenario description: In an AI academic conference, the speaker introduces an algorithm optimization scheme combining quantum computing and neural networks at a fast speed, and the recording contains a dense professional term and a logical jump (such as omitting part of the derivation steps).
[0106] Processing flow and details:
[0107] Input: a 30-minute conference recording (including background noise).
[0108] Voiceprint feature extraction and speech to text:
[0109] Segmentation and noise reduction: segment the audio every 2 seconds, and use a deep learning noise reduction model (such as WaveNet) to remove environmental noise.
[0110] Voiceprint feature extraction: CNN extracts spectral features, RNN captures temporal dependencies, and attention mechanism weights key frames (such as speaker emphasized parts).
[0111] Text conversion: CTC model outputs the first draft, and the recognition accuracy is improved from 82% to 92% (due to voiceprint enhancement).
[0112] Candidate term acquisition:
[0113] Natural language analysis model (domain customized BERT) identifies terms: "quantum computing", "neural network", "algorithm optimization", "quantum gate circuit", "gradient descent method", etc.
[0114] The term boundary detection: correctly identify "quantum gate circuit" instead of splitting, avoid mislabeling as "quantum gate" and "circuit".
[0115] The term relationship path construction:
[0116] Knowledge graph search path: "quantum computing" → "quantum gate circuit" → "neural network acceleration".
[0117] Incremental update: combined with the latest papers, add a new path "quantum optimization algorithm" → "gradient descent method improvement".
[0118] Cross-domain association: establish cross-domain association between "quantum computing" and "algorithm optimization" through semantic similarity.
[0119] Dynamic DAG construction and repair:
[0120] Detecting a logical break between "quantum gate circuit" and "neural network acceleration" (the speaker omitted the hardware implementation step).
[0121] The repair mechanism supplements the "quantum hardware architecture" node from the knowledge graph and calculates the probability graph model to select the optimal path.
[0122] Abstract generation:
[0123] Key event clustering: "15:00-15:30: algorithm principles of quantum computing combined with neural networks; 16:10-16:40: experimental verification and performance comparison".
[0124] Abstract content: "Proposed neural network algorithm based on quantum optimization, improved gradient descent method, achieved 40% efficiency improvement on XX dataset. Hardware architecture uses XX quantum gate circuit design."
[0125] According to the natural language processing-based audio summary generation method provided in the present application, the voiceprint feature extraction and attention mechanism are used to reduce the speech-to-text error rate. The pre-trained natural language parsing model is used to parse the speech text, and the semantic embedding vector and term boundary detection are used to accurately locate and extract key candidate terms, reduce irrelevant information interference, and combine the knowledge graph to match the term relationship, realize the semantic alignment and dynamic expansion of cross-domain terms, enhance the comprehensiveness and accuracy of term association, so as to realize the accurate judgment of professional terms, and the generated summary text has good logical relationship. The dynamic graph repair mechanism improves the logical coherence of the summary, reduces the information missing rate, and can also be labeled after filling the missing information, helping users quickly locate the unclear part of the audio. This method is suitable for quick content extraction and deep analysis in various scenarios.
[0126] Corresponding to the natural language processing-based audio summary generation method described in the above embodiment,Figure 2 A structural block diagram of a recording summary generation device based on natural language processing is shown, and only parts related to the embodiments of the present application are shown for ease of illustration.
[0127] Referring to Figure 2 The device comprises:
[0128] A voiceprint feature extraction module is configured to receive an audio stream and convert the audio stream into speech text through a voiceprint feature extraction model.
[0129] A term analysis module is configured to input the speech text into a pre-trained natural language analysis model to obtain candidate terms corresponding to the speech text. The natural language analysis model is a neural network model trained based on annotated sample speech text.
[0130] A term relationship matching module is configured to match the candidate terms with a knowledge graph to obtain a term relationship path of the candidate terms.
[0131] A dynamic graph construction module is configured to construct a directed acyclic graph including a broken mark according to the term relationship path.
[0132] A recording summary generation module is configured to generate a recording summary corresponding to the audio stream according to the candidate terms and the directed acyclic graph.
[0133] It should be noted that the information interaction, execution process, and the like between the above-mentioned devices / units are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought about can be referred to the method embodiments part, which will not be repeated here.
[0134] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, i.e., the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software. In addition, the specific names of the functional units and modules are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the device can be referred to the corresponding process in the foregoing method embodiments, which will not be repeated here.
[0135] The embodiments of the present application also provide a recording summary generation device, which is shown in Figure 3The communication device includes at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor implements the steps in any of the method embodiments described above when executing the computer program.
[0136] The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the method embodiments described above.
[0137] The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiments, which can be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium, and the computer program, when executed by a processor, can implement the steps in the method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium cannot be an electrical carrier signal and a telecommunication signal.
[0138] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.
[0139] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0140] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other manners. For example, the embodiments of the apparatus / network device described above are merely illustrative. For example, the division of the modules or units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or in other forms.
[0141] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.
[0142] The above-described embodiments are merely used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. A method for generating audio recording summaries, characterized in that, The method includes the following steps: Receive an audio stream and convert it into speech text using a voiceprint feature extraction model; The speech text is input into a pre-trained natural language parsing model to obtain candidate terms corresponding to the speech text; the natural language parsing model is a neural network model trained based on labeled sample speech text. The candidate terms are matched with the knowledge graph to obtain the terminology relationship paths of the candidate terms; Based on the terminology relationship path, a directed acyclic graph including break markers is constructed, and the break markers are repaired based on the context of the spoken text. Based on the candidate terms and the directed acyclic graph, a recording summary corresponding to the audio stream is generated.
2. The method as described in claim 1, characterized in that, The received audio stream is converted into speech text using a voiceprint feature extraction model, including: The audio stream is preprocessed, and spectral features are extracted through the convolutional layer of the voiceprint feature extraction model, and the spectral feature matrix of the audio stream is output. The temporal dependency of the spectral feature matrix is captured by the recurrent neural network layer of the aforementioned voiceprint feature extraction model to obtain the temporal encoding vector of the audio stream. The multi-scale features of the temporal coding vector are fused using an attention mechanism to obtain the output text probability distribution of the audio stream; The speech text is generated based on the output text probability distribution of the audio stream.
3. The method as described in claim 1, characterized in that, The step of inputting the speech text into a pre-trained natural language parsing model to obtain candidate terms corresponding to the speech text includes: The speech text is input into the base layer of the natural language parsing model to obtain the semantic embedding vector of the speech text; The semantic embedding vector is input into the term boundary detector of the natural language parsing model to obtain the position labels of candidate terms; Based on the location markers of the candidate terms, output the set of candidate terms corresponding to the speech text.
4. The method as described in claim 3, characterized in that, The knowledge graph includes: a term association graph, an incremental update layer, and a cross-domain association index; The step of matching the candidate terms with the knowledge graph to obtain the terminology relationship path of the candidate terms includes: Query the initial association path corresponding to the candidate term in the candidate term set in the term association graph; The initial association path is input into the incremental update layer to determine the extended association path of the candidate term; Based on the cross-domain association index, the term relationships of the candidate terms in different domains are semantically aligned to obtain the cross-domain alignment path of the candidate terms. Output the term relationship path, which is the fusion of the candidate terms, the extended association path, and the cross-domain alignment path.
5. The method as described in claim 1, characterized in that, The construction includes a directed acyclic graph of break markers, and the repair of break markers based on the context of the spoken text, including: Initialize an empty directed acyclic graph, and use the candidate terms as nodes; Based on the term relationship path of the candidate terms, the semantic similarity between term nodes is calculated as the edge weight; By using a dynamic programming algorithm, disconnected subgraphs in the directed acyclic graph are marked with breaks to identify regions with missing information. The terminology relationships at the break marks are repaired by using a reverse filling mechanism and contextual semantic information.
6. The method as described in claim 5, characterized in that, The method of repairing the terminology relationship at the break mark using contextual semantic information includes the following steps: Identify the fracture regions in the directed acyclic graph and calculate the semantic distance between the two sides of the fracture. Retrieve supplementary terms related to the semantics of the fracture region from the knowledge graph; The insertion probability of the supplementary term is calculated using a probabilistic graphical model, and the optimal path is selected to fill the gap. Update the edge weights of the directed acyclic graph.
7. The method as described in claim 1, characterized in that, Generating the recording summary corresponding to the audio stream includes the following steps: Extract key term nodes and their path weights from the directed acyclic graph; Based on the path weight sorting, a list of key terms is generated; Event clustering is performed on the continuous node sequence in the directed acyclic graph to generate an event summary; A timestamp is added to each of the event summaries, and the timestamp is used to mark the start and end times of the event corresponding to the audio stream.
8. A recording summary generation device, characterized in that, include: The voiceprint feature extraction module is used to receive audio streams and convert the audio streams into speech text through a voiceprint feature extraction model; The terminology parsing module is used to input the speech text into a pre-trained natural language parsing model to obtain candidate terms corresponding to the speech text; the natural language parsing model is a neural network model trained based on labeled sample speech text. The terminology relationship matching module is used to match the candidate terms with the knowledge graph to obtain the terminology relationship path of the candidate terms; A dynamic graph construction module is used to construct a directed acyclic graph including break markers based on the term relationship path, and repair the break markers based on the context of the speech text. The recording summary generation module is used to generate a recording summary corresponding to the audio stream based on the candidate terms and the directed acyclic graph.
9. A recording summary generation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.