Call shorthand method and management system based on large model

Abstract

The invention provides a call shorthand method and management system based on a large model, and relates to the technical field of call shorthand, and the method comprises the steps: collecting the voice of a user after the shorthand authority granted by the user is received, and enabling the collected voice of each user to form a voice stream; preprocessing the voice stream to obtain a processed voice stream; converting the processed voice stream into an initial text containing speaking content, a timestamp and a user identifier; inputting the initial text into a large language model, and executing the following tasks by the large language model: extracting fault statements based on speaking contents; and complementing and marking the fault statement based on the context content of the initial text to generate a shorthand text. And complementing fault statements by using context contents to solve the problems of branch breakage and semantic segmentation caused by dialogue faults.

Description

Technical Field

[0001] This invention relates to the field of call stenography technology, specifically to a call stenography method and management system based on a large model. Background Technology

[0002] With the popularization of remote collaboration and online meetings, call stenography technology has emerged, which automatically converts the voice content of a call into a text record, aiming to retain information, review the meeting afterward, and trace the decision.

[0003] However, when existing call stenography technology is applied to multi-participant group chat scenarios, the generated text records face severe challenges in terms of coherence and readability, and generally suffer from the problem of "dialogue discontinuity". As a result, the text records cannot accurately reflect the logical flow, interaction relationship and contextual semantics of the dialogue itself, resulting in fragmented and semantically disjointed stenographic text. Summary of the Invention

[0004] The purpose of this invention is to provide a method and management system for call stenography based on a large model. The technical problem to be solved is the existence of dialogue breaks when converting speech content into stenographic text.

[0005] This invention is achieved through the following technical solution:

[0006] The first aspect provides a call shorthand method based on a large model, including the following steps:

[0007] After receiving the shorthand permission granted by the user, the system collects the user's voice, and the collected voices from each user are combined into a voice stream.

[0008] The above speech stream is preprocessed to obtain the processed speech stream;

[0009] The above-processed audio stream is converted into initial text containing the spoken content, timestamps, and user identifiers;

[0010] The initial text above is input into a large language model, which then performs the following tasks:

[0011] Based on the above statements, extract the fragmented sentences;

[0012] Based on the context of the initial text, complete and mark the broken sentences to generate shorthand text.

[0013] The large language model utilizes the contextual content of the initial text to complete fragmented sentences. Since the contextual content of the initial text contains the logical flow of the dialogue, interaction relationships, and semantic information, the large language model integrates this information to generate sentences that are semantically coherent and logically sound. This solves the problem of fragmented and semantically disjointed shorthand texts caused by dialogue breaks in existing technologies, enabling shorthand texts to more accurately reflect the content and logic of the dialogue itself. While completing fragmented sentences, the model also marks them, allowing users to clearly see which parts have been completed. This helps users verify the accuracy of the completed content, thereby improving the credibility of the shorthand text.

[0014] Furthermore, the above-mentioned speech stream is preprocessed, including the following steps:

[0015] Extract the judgment features used to distinguish between main speech and noisy speech from the above speech stream;

[0016] Based on the above-mentioned characteristics, the speech in the speech stream is classified to obtain the main speech and noisy speech; the noisy speech is filtered out to obtain the processed speech stream.

[0017] The timestamps of each speech are obtained from the processed speech stream. If the timestamps of adjacent speech overlap, the adjacent speech is marked as overlapping speech.

[0018] Based on the timestamps of the adjacent speech, the time interval between adjacent speech is determined; when the time interval between adjacent speech is within the interruption time threshold range, the speech of the previous moment in the adjacent speech is marked as the interrupted speech.

[0019] The collected audio stream includes not only the main discussion dialogue but also noisy private conversations. Filtering out noisy audio reduces the amount of audio data input to the large oracle model, improving processing efficiency. During discussions, simultaneous speaking and interruptions may occur. When multiple users speak simultaneously, their voices interfere with each other, making complete audio recognition impossible. When a user interrupts another, the speech is interrupted, resulting in incomplete content. Therefore, marking simultaneous speaking and interruptions helps with subsequent completion operations.

[0020] Furthermore, based on the overlapping and interrupted speech marks mentioned above, the fragmented sentences are identified.

[0021] Quickly locate the above-mentioned broken sentences based on the markers to prepare for subsequent content completion.

[0022] Furthermore, complete the broken sentences, including the following steps:

[0023] Obtain the user identifier of the above-mentioned fault statement, obtain the next statement adjacent to the fault statement based on the user identifier, and determine whether the statement is connected to the fault statement.

[0024] If so, then add the above statement into the break statement to obtain the completed break statement, and mark the completed break statement as a completed statement;

[0025] If not, mark the above broken statement as a statement to be completed.

[0026] By identifying subsequent user comments, it's determined whether they are further elaborations on a broken sentence. If so, the elaboration is added to the broken sentence to make it complete. If subsequent comments do not provide further elaboration, the context cannot be used to infer the subsequent content of the broken sentence, requiring manual completion. Completed broken sentences are marked to facilitate user verification of accuracy when using shorthand text, thus improving the reliability of the shorthand text. Marking broken sentences requiring manual completion as pending completion facilitates subsequent completion of those sentences.

[0027] Furthermore, upon receiving a call interruption signal, the system obtains the user's statement to be completed based on the user identifier. If the user's statement to be completed is obtained, a completion instruction is sent to the user's terminal. The system then inputs the completion content of the corresponding broken statement from the terminal to obtain the completed broken statement.

[0028] Change the marker of the above fault statement from "statement to be completed" to "statement to be completed".

[0029] After the call is interrupted, the broken sentence that needs to be manually completed is fed back to the user, who then manually enters the completed content to make the broken sentence complete. After completion, the marker is changed.

[0030] The second aspect provides a call stenography management system based on a large model, which uses the aforementioned call stenography method; the call stenography management system includes:

[0031] The authorization unit is used to send an authorization request to the terminal and, upon receiving the authorization response from the terminal, to grant the terminal shorthand permission.

[0032] The voice acquisition unit is communicatively connected to the authorization unit. After receiving the shorthand permission from the terminal, the voice acquisition unit acquires the user's voice from the terminal and combines the acquired user voices into a set to obtain a voice stream.

[0033] The speech recognition unit is communicatively connected to the speech acquisition unit; the speech recognition unit is used to preprocess the speech stream to obtain a processed speech stream, and convert the processed speech stream into initial text containing the speech content, timestamps and user identifiers.

[0034] The text processing unit is communicatively connected to the speech recognition unit. The text processing unit invokes a large language model, inputting the initial text into the large language model, which then performs the following tasks:

[0035] Based on the above statements, extract the fragmented sentences;

[0036] Based on the context of the initial text, complete and mark the broken sentences to generate shorthand text.

[0037] Before collecting voice data, authorization from the end user is required. After granting shorthand permission, the end user's voice data is collected, and the collected voice data is recognized, converted into text, and then processed by a text processing unit to complete the fragmented sentences and finally output the shorthand text.

[0038] Furthermore, it also includes a communication unit, which is communicatively connected to the text processing unit and the authorization unit; the communication unit is used for real-time communication with the terminal.

[0039] Furthermore, after receiving the call interruption signal from the terminal, the communication unit sends a completion instruction to the text processing unit; the text processing unit obtains the user identifier of the terminal, and obtains the user's sentence completion mark based on the user identifier. If the user's sentence completion mark is obtained, a completion instruction is sent to the user's terminal; the completion content of the corresponding broken sentence is input from the terminal to obtain the completed broken sentence.

[0040] Change the marker of the above fault statement from "statement to be completed" to "statement to be completed".

[0041] The aforementioned communication unit enables communication between the terminal and the management system. After communication is interrupted, a completion command is sent to the user, which avoids interference with the discussion compared to performing completion during communication.

[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0043] The large language model uses the contextual content of the initial text to complete fragmented sentences. Since the contextual content of the initial text contains the logical flow, interaction relationships, and semantic information of the dialogue, the large language model integrates this information to generate sentences that are semantically coherent and logically sound. This solves the problem of fragmented and semantically disjointed shorthand texts generated by existing technologies due to dialogue breaks, enabling shorthand texts to more accurately reflect the content and logic of the dialogue itself.

[0044] By marking the missing sentences while completing them, users can clearly see which parts have been completed when using the shorthand text. This helps users check whether the completed content is correct, thereby improving the credibility of the shorthand text. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0046] Figure 1 Here is the main flowchart for the call shorthand method;

[0047] Figure 2 This is a system block diagram.

[0048] The attached diagram shows the markings and corresponding component names:

[0049] 1. Authorization unit; 2. Voice acquisition unit; 3. Voice recognition unit; 4. Text processing unit; 5. Communication unit. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0051] First embodiment:

[0052] Combination Figure 1 The call shorthand method based on a large model includes the following steps:

[0053] After receiving the shorthand permission granted by the user, the system collects the user's voice, and the collected voices from each user are combined into a voice stream.

[0054] The above speech stream is preprocessed to obtain the processed speech stream;

[0055] The above-processed audio stream is converted into initial text containing the spoken content, timestamps, and user identifiers;

[0056] The initial text above is input into a large language model, which then performs the following tasks:

[0057] Based on the above statements, extract the fragmented sentences;

[0058] Based on the context of the initial text, complete and mark the broken sentences to generate shorthand text.

[0059] The large language model utilizes the contextual content of the initial text to complete fragmented sentences. Since the contextual content of the initial text contains the logical flow of the dialogue, interaction relationships, and semantic information, the large language model integrates this information to generate sentences that are semantically coherent and logically sound. This solves the problem of fragmented and semantically disjointed shorthand texts caused by dialogue breaks in existing technologies, enabling shorthand texts to more accurately reflect the content and logic of the dialogue itself. While completing fragmented sentences, the model also marks them, allowing users to clearly see which parts have been completed. This helps users verify the accuracy of the completed content, thereby improving the credibility of the shorthand text.

[0060] One possible use case is a business meeting held by a marketing department, attended by the marketing director, marketing specialists, and representatives from partner companies. The meeting focused on in-depth discussions of new product promotion strategies, covering key issues such as market research analysis, channel selection, and budget allocation. Due to the rich content and large amount of information, to ensure no important information is missed and to facilitate subsequent compilation of meeting minutes and decision follow-up, the call shorthand method of this invention is used to record the meeting content. Details are as follows:

[0061] Before the meeting begins, explain to all participants that a call stenography method based on a large model will be used to record the meeting content, and request that everyone grant stenography permissions. After the participants agree, complete the process of granting stenography permissions on the mobile devices.

[0062] Subsequently, multiple microphones installed in the conference room began to operate, capturing the voices of each participant in real time and recording them sequentially according to the order of their speeches to form a continuous audio stream. For example, the Marketing Director spoke first: "Ladies and gentlemen, today we will mainly discuss new product promotion strategies..." Then, Marketing Specialist A said: "According to market research conducted in the first half of the year, our brand awareness among young consumers is still relatively low."

[0063] The collected audio stream may contain some background noise, such as air conditioner noise, chair moving noise, private conversations, etc.; noise reduction processing is performed on the audio stream to remove interfering noise, and the volume of the audio is standardized to ensure that the volume of different speakers is relatively consistent, resulting in a clear processed audio stream.

[0064] The pre-processed speech stream is identified using speech recognition technology and converted into text. Simultaneously, a timestamp is added to each speech segment to record the start and end times, and a corresponding user identifier is added to each segment based on pre-defined participant information. For example, the generated initial text is as follows:

[0065] [00:00:05-00:00:20][Marketing Director]: Everyone, today we will mainly discuss new product promotion strategies.

[0066] [00:00:21-00:00:35][Marketing Specialist A]: According to market research conducted in the first half of the year, our brand awareness among young consumers remains relatively low.

[0067] [00:00:36-00:00:45][Marketing Specialist B] We've tried this before.

[0068] [00:00:45-00:00:48][Other] Excuse me.

[0069] [00:00:49-00:01:00][Marketing Specialist B] Tried offline promotion, but the results were not very good.

[0070] [00:01:01-00:01:35][Partner Representative]: While promoting offline, we can also increase promotional efforts on social media platforms.

[0071] The large language model analyzes the initial text and identifies any broken sentences. For example, when Marketing Specialist B is interrupted during a speech, one sentence is incomplete: "We tried before." "We tried before" is a broken sentence, semantically incomplete, making it difficult to understand its specific meaning. After the interruption ends, Marketing Specialist B says, "We tried offline promotions before, but the results weren't very good." Based on the context of the initial text, the large language model completes the broken sentence, inferring that Marketing Specialist B intended to express "We tried offline promotions before, but the results weren't very good." The completed content is added to the shorthand text using special markers (such as adding the identifier "*", bolding the text, changing the color, etc.), generating the complete shorthand text, as shown below:

[0072] [00:00:05-00:00:20][Marketing Director]: Everyone, today we will mainly discuss new product promotion strategies.

[0073] [00:00:21-00:00:35][Marketing Specialist A]: According to market research conducted in the first half of the year, our brand awareness among young consumers remains relatively low.

[0074] *[00:00:36-00:01:00][Marketing Specialist B] We tried offline promotion before, but the results weren't very good.*

[0075] [00:01:01-00:01:35][Partner Representative]: While promoting offline, we can also increase promotional efforts on social media platforms.

[0076] Second embodiment:

[0077] Based on the first embodiment, the above-mentioned speech stream is preprocessed, including the following steps:

[0078] Extract the discrimination features from the above speech stream to distinguish between main speech and noisy speech; the discrimination features may be speech frequency, volume, and energy intensity.

[0079] Based on the above-mentioned characteristics, the speech in the speech stream is classified to obtain the main speech and noisy speech; the noisy speech is filtered out to obtain the processed speech stream.

[0080] The timestamps of each speech are obtained from the processed speech stream. If the timestamps of adjacent speech overlap, the adjacent speech is marked as overlapping speech.

[0081] Based on the timestamps of the adjacent speech, the time interval between adjacent speech is determined; when the time interval between adjacent speech is within the interruption time threshold range, the speech of the previous moment in the adjacent speech is marked as the interrupted speech.

[0082] The collected audio stream includes not only the main discussion dialogue but also noisy private conversations. Filtering out noisy audio reduces the amount of audio data input to the large oracle model, improving processing efficiency. During discussions, simultaneous speaking and interruptions may occur. When multiple users speak simultaneously, their voices interfere with each other, making complete audio recognition impossible. When a user interrupts another, the speech is interrupted, resulting in incomplete content. Therefore, marking simultaneous speaking and interruptions helps with subsequent completion operations.

[0083] In a specific implementation, in order to quickly locate the above-mentioned broken sentences, the broken sentences are determined based on the marked overlapping and interrupted speech, in order to prepare for subsequent content completion; that is, sentences marked as overlapping and interrupted are broken sentences.

[0084] One possible use case is a cross-regional online academic seminar where experts discuss cutting-edge research findings. When Expert A presents their viewpoint, their speech energy is stable and the frequency is concentrated within a specific range; while noise such as car horns has large energy fluctuations and chaotic frequencies, and private conversations are at low volume. Based on these characteristics, car horn noise and other noise are filtered out, resulting in a processed speech stream containing only the main speech of the experts. When Expert B and Expert C speak almost simultaneously, their speech timestamps overlap, and these are marked as overlapping speech for easier subsequent analysis. If Expert E interrupts Expert D while Expert D is speaking, Expert E's interruption should occur within the interruption time threshold range (assumed to be 0.5-1 second), and Expert D's speech is marked as interrupted.

[0085] Third embodiment:

[0086] Based on the second embodiment, complete the tomographic statements, including the following steps:

[0087] Obtain the user identifier of the above-mentioned fault statement, and obtain the next statement adjacent to the fault statement based on the user identifier (or obtain multiple statements of the user continuously and determine whether the obtained statements are connected to the fault statement), and determine whether the statement is connected to the fault statement.

[0088] If so, then add the above statement into the break statement to obtain the completed break statement, and mark the completed break statement as a completed statement;

[0089] If not, mark the above broken statement as a statement to be completed.

[0090] By identifying subsequent user comments, it's determined whether they are further elaborations on a broken sentence. If so, the elaboration is added to the broken sentence to make it complete. If subsequent comments do not provide further elaboration, the context cannot be used to infer the subsequent content of the broken sentence, requiring manual completion. Completed broken sentences are marked to facilitate user verification of accuracy when using shorthand text, thus improving the reliability of the shorthand text. Marking broken sentences requiring manual completion as pending completion facilitates subsequent completion of those sentences.

[0091] Fourth embodiment:

[0092] Based on the third embodiment, after receiving the call interruption signal, the user's statement to be completed is obtained based on the user identifier. If the user's statement to be completed is obtained, a completion instruction is sent to the user's terminal. The completion content of the corresponding broken statement is input from the terminal to obtain the completed broken statement.

[0093] Change the marker of the above fault statement from "statement to be completed" to "statement to be completed".

[0094] After the call is interrupted, the broken sentence that needs to be manually completed is fed back to the user, who then manually enters the completed content to make the broken sentence complete. After completion, the marker is changed.

[0095] Fifth embodiment:

[0096] Combination Figure 2 A call stenography management system based on a large model, which uses the aforementioned call stenography method; the call stenography management system includes:

[0097] Authorization unit 1 is used to send an authorization request to the terminal and, upon receiving an authorization response from the terminal, to enable the terminal's shorthand permission.

[0098] Voice acquisition unit 2 is communicatively connected to authorization unit 1. After receiving the shorthand permission of the terminal, the voice acquisition unit 2 acquires the user's voice of the terminal, and the acquired user voices are combined into a set to obtain a voice stream.

[0099] The speech recognition unit 3 is communicatively connected to the speech acquisition unit 2; the speech recognition unit 3 is used to preprocess the speech stream to obtain a processed speech stream, and convert the processed speech stream into initial text containing the speech content, timestamp and user identifier.

[0100] Text processing unit 4 is communicatively connected to speech recognition unit 3; text processing unit 4 is used to invoke a large language model, input the initial text into the large language model, and have the large language model perform the following tasks:

[0101] Based on the above statements, extract the fragmented sentences;

[0102] Based on the context of the initial text, complete and mark the broken sentences to generate shorthand text.

[0103] Before collecting voice data, authorization from the end user is required. After granting shorthand permission, the end user's voice data is collected, and the collected voice data is recognized and converted into text. The converted text is then processed by text processing unit 4 to complete the fragmented sentences and finally output the shorthand text.

[0104] Sixth embodiment:

[0105] Based on the fifth embodiment, a communication unit 5 is also included, which is communicatively connected to the text processing unit 4 and the authorization unit 1; the communication unit 5 is used for real-time communication with the terminal.

[0106] In a specific embodiment, after receiving a call interruption signal from the terminal, the communication unit 5 sends a completion instruction to the text processing unit 4; the text processing unit 4 obtains the user identifier of the terminal, obtains the user's sentence completion mark based on the user identifier, and if the user's sentence completion mark is obtained, sends a completion instruction to the user's terminal; the completion content of the corresponding broken sentence is input from the terminal to obtain the completed broken sentence.

[0107] Change the marker of the above fault statement from "statement to be completed" to "statement to be completed".

[0108] The communication unit 5 mentioned above enables communication between the terminal and the management system. After communication is interrupted, a completion command is sent to the user, which avoids interference with the discussion compared to performing completion during communication.

[0109] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of call transcription based on a large model, characterized by, The method comprises the following steps: After receiving the shorthand permission granted by the user, the voice of the user is collected, and each user voice collected forms a voice stream; The voice stream is preprocessed to obtain a processed voice stream; The processed voice stream is converted into an initial text containing speech content, a timestamp, and a user identifier; The initial text is input into a large language model, and the large language model performs the following tasks: Based on the speech content, extract fault statements; Based on the context content of the initial text, complete and label the fault statements to generate shorthand text.

2. The large model-based conversation shorthand method according to claim 1, wherein, The preprocessing of the voice stream comprises the following steps: Extracting a judgment feature for distinguishing main voice and noise voice from the voice stream; Based on the judgment feature, the voice in the voice stream is classified to obtain main voice and noise voice; the noise voice is filtered to obtain a processed voice stream; From the processed voice stream, the timestamp of each voice is obtained, and when the timestamps of adjacent voices overlap, the adjacent voices are marked as overlapping voices; According to the timestamps of the adjacent voices, the time interval of the adjacent voices is determined; when the time interval of the adjacent voices is in the interruption time threshold interval, the previous time voice in the adjacent voices is marked as an interrupted voice. 3.The large model-based conversation shorthand method according to claim 2, wherein, Determine the fault statement according to the marked overlapping voice and interrupted voice.

4. The large model based conversation shorthand method of claim 3, wherein, Completing the fault statement comprises the following steps: Obtain the user identifier of the fault statement, obtain the next statement adjacent to the fault statement based on the user identifier, and determine whether the statement connects the fault statement; If yes, the statement is supplemented into the fault statement to obtain a completed fault statement, and the completed fault statement is marked as a completed statement; If not, the fault statement is marked as a statement to be completed.

5. The large model based conversation shorthand method of claim 4, wherein, After receiving the call interruption signal, the user identifier of the user is obtained, and if the user identifier of the user is obtained, a completion instruction is sent to the terminal of the user; The completion content corresponding to the fault statement is input from the terminal to obtain the completed fault statement; Change the mark of the fault statement from the statement to be completed to the completed statement.

6. A dictation management system based on a large model, characterized by, The call shorthand management system uses the call shorthand method of any one of claims 1 to 5; The call shorthand management system comprises: An authorization unit (1) configured to send an authorization request to a terminal and open the shorthand permission of the terminal after receiving an authorization response sent by the terminal; A voice collection unit (2) in communication connection with the authorization unit (1); the voice collection unit (2) collects the voice of the user of the terminal after receiving the shorthand permission of the terminal, and forms a set by collecting each user voice to obtain a voice stream; A speech recognition unit (3) in communication connection with the voice collection unit (2); the speech recognition unit (3) is configured to preprocess the voice stream to obtain a processed voice stream, and convert the processed voice stream into an initial text containing speech content, a timestamp, and a user identifier; The text processing unit (4) is connected in communication with the speech recognition unit (3); the text processing unit (4) is used for calling a large language model, inputting the initial text into the large language model, and executing the following tasks by the large language model: Based on the speech content, extract fault sentences; Based on the context content of the initial text, complete and mark the fault sentences to generate shorthand texts. 7.The large model based conversation shorthand management system according to claim 6, wherein, Further comprising a communication unit (5) connected in communication with the text processing unit (4) and the authorization unit (1); the communication unit (5) is used for real-time communication with the terminal. 8.The large model based conversation shorthand management system according to claim 7, wherein, The communication unit (5) sends a supplement instruction to the text processing unit (4) after receiving a call interruption signal of the terminal; the text processing unit (4) acquires the user identifier of the terminal, acquires the to-be-completed sentence mark of the user based on the user identifier, and sends a completion instruction to the terminal of the user if the to-be-completed sentence mark of the user is acquired; The completion content corresponding to the fault sentence is input from the terminal to obtain the completed fault sentence; The mark of the fault sentence is changed from the to-be-completed sentence to the completed sentence.

Images