Rich text audio summarization method based on LLM multi-feature fusion slicing

Abstract

The invention relates to the technical field of audio processing, in particular to a rich text audio summarization method based on LLM multi-feature fusion slicing, which comprises the following steps: acquiring a conference audio file through a recording and reception module in a recording and collecting device, analyzing the audio file through an audio analysis module, and performing theme cutting on the audio file; the natural language processing and analyzing module judges whether two adjacent statement topics are the same topic or not to obtain statements segmented according to the topics, summarization content of each topic is obtained through the LLM module, finally, the map-reduce process is executed, summarization of each topic is summarized, and an overall summarization is obtained. The method can be effectively applied to various automatic summary summaries with rich text information, is not limited to the time duration of conference audios, and has a good audio text processing effect.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a rich text audio summarization method based on LLM multi-feature fusion slicing. Background Art

[0002] An audio file is a file that contains audio data such as music, sound effects, speech, etc. These files are usually designed to be played on various media players, computer programs, mobile devices, etc. There are many different audio file formats, each with its specific uses and characteristics.

[0003] Rich text audio refers to audio containing a large amount of text information, such as conferences and debates. At present, there are the following technical problems in the processing of such audio: the content of rich text audio is lengthy, and the meeting recorder often has to listen to the entire conference audio for a long time before he can finish listening to the entire conference audio and form a summary report. As a result, the processing efficiency of the entire rich text audio is greatly reduced and a lot of time is required. In this regard, there is an urgent need to provide a rich text audio summary based on LLM (short for large language model) multi-feature fusion slicing. Summary of the invention

[0004] The main purpose of the present invention is to overcome the deficiencies in the prior art and provide a rich text audio summarization method based on LLM multi-feature fusion slicing.

[0005] The technical solution adopted by the present invention to achieve its technical purpose is: a rich text audio summarization method based on LLM multi-feature fusion slicing, specifically including the following steps: S1. Obtain and record audio files through the recording and sound receiving module in the recording collection device; S2, identifying the speaker identity, the corresponding speaker speech content and the corresponding speaker speech speed in the audio file through an audio analysis module; S3, obtaining features from the speaker's sentences through the audio analysis module, including speech rate change features, text vector features and transition keywords, and performing sentence topic segmentation; Text vector features include keyword tag extraction and topic extraction, such as transition keywords such as "next" or "next topic"; S4. By setting up a natural language processing parsing module to perform feature fusion, train an SVM support vector machine binary classification model, and judge whether two adjacent sentence topics are the same topic based on the fusion features between adjacent sentence topics; SVM support vector machine is a machine learning method used to solve classification problems; S5, combining steps S3 and S4, obtaining sentences segmented by topic; S6. After dividing the topics, we use the LLM module to summarize each topic based on the map-reduce method to obtain the summary content of each topic; LLM module refers to the Large Language Model module, and map-reduce refers to mapping-reduction, a divide-and-conquer method that is used to split a large problem into small problems, solve the small problems, and then reduce the small problems. It is a computer thinking method that is widely used in many fields such as big data and artificial intelligence. S7. Then summarize each topic and execute the map-reduce process to get the overall summary.

[0006] Preferably, a voiceprint recognition module is also included. For each speaker, the identity of the speaker is combined with the actually defined identity through the voiceprint recognition module to obtain the speech content of the specific person in the meeting or debate.

[0007] Preferably, the audio analysis module uses whisperX to segment the speaker's sentences, thereby identifying the speaker, identifying the speech content, and extracting the speech rate using the speech content. WhisperX is an audio analysis algorithm for speech recognition and speaker recognition.

[0008] Preferably, it is characterized in that: wherein, speaking speed = total number of words / audio duration, the unit is min.

[0009] Preferably, the natural language processing parsing module uses bge-m3 to calculate the semantic features of sentences, and integrates speech rate, text vector features and transition keywords to obtain fusion features, and trains an SVM support vector machine binary classification model to determine whether adjacent topics are sentences of the same topic.

[0010] Among them, bge-m3 is a text vectorization model algorithm, which is used to convert text into feature vectors. It is one of the most widely used and powerful models among the current open source pre-trained text vectorization models. It can extract text features from text and vectorize it. This process is called text vectorization.

[0011] Therefore, the features used above mainly include: the dense matrix features of the text itself corresponding to the text vector, the sparse matrix features corresponding to the transition keywords, and the speaking speed. In order to reduce algorithm misjudgment and enhance robustness, the sliding window concept is added. Each sentence will be judged against the previous three sentences. If the three previous sentences are not similar, it is considered to be the sentence starting a new topic.

[0012] Preferably, the LLM module writes prompt words, summarizes each topic based on the speech content, and then summarizes the topic to obtain a summary of the entire meeting.

[0013] Preferably, the voiceprint module uses X-vector to perform speaker verification on each speaker to obtain the identity of each speaker. X-vector is an existing technology.

[0014] Preferably, the recording collection device comprises an adjustment rod assembly, a fixing seat assembly, a recording and sound receiving assembly, a rotating seat assembly and a base; One end of the adjusting rod assembly is fixedly mounted on the base, and the other end is fixedly connected to the fixing seat assembly. The rotating seat assembly is rotatably mounted on the fixing seat assembly, and the recording and receiving assembly is fixedly mounted above the rotating seat assembly.

[0015] Preferably, the adjusting rod assembly comprises an outer skin, a metal wire, a connecting protrusion and an inner concave plate; the outer skin is made of a soft leather material, and the metal wire can be made of other soft metal wire materials such as aluminum wire or iron wire.

[0016] The connecting convex head and the inner concave plate are respectively fixedly connected to the inside of the two ends of the outer skin, and the metal wire is fixedly connected between the connecting convex head and the inner concave plate and is located inside the outer skin; The plurality of connecting convex heads and the inner concave plates are fixed end to end, the connecting convex head at the outermost end is fixedly plugged on the base, and the inner concave plate at the outermost end is fixedly connected to the bottom of the fixing seat assembly; By virtue of the material properties of the outer skin and the metal wire, the adjusting rod assembly can be bent or adjusted at any angle at each rod section.

[0017] Preferably, the fixing seat assembly comprises a fixing plate, a mounting groove, a spring and a ball; The bottom of the fixed plate is fixed to the inner concave plate at the outermost end, and the mounting grooves are provided at both ends of the inner portion thereof. One end of the spring is fixed inside the mounting groove, and the other end is fixed to the ball. The ball can freely pass through the mounting groove. The rotating seat assembly comprises a rotating cylinder, an annular groove and spherical particles; The bottom of the middle end of the rotating cylinder is rotatably connected to the top of the fixed disk via a rotating shaft, the edge of the rotating cylinder is provided with the annular groove, and the spherical particles are fixedly arranged inside the annular groove; When the rotating cylinder is rotatably connected to the fixed disk, the balls are pushed into the annular groove by the spring to contact the spherical particles.

[0018] By cooperating with the fixed seat assembly and the rotating seat assembly, the rotating cylinder can be rotated arbitrarily on the fixed disk. At the same time, when the ball is pushed into the annular groove by the spring to contact the spherical particles, the rotating cylinder has a damping feeling when rotating due to the bumps between the spherical particles, and can pause at any position.

[0019] Preferably, the recording and sound receiving assembly comprises a rotating shell, a recording and sound receiving module, a protective cover and a through hole; The bottom of the rotating shell is fixed to the top of the rotating cylinder, and a plurality of through holes are provided on the top thereof, wherein the through holes are arranged in a honeycomb hole structure, so that the recording and sound receiving module can collect audio; The recording and receiving module is fixedly arranged inside the rotating shell, the protective cover is sleeved on half of the outer wall of the rotating shell, and its bottom is fixed on a fixed plate. By rotating the rotating cylinder, the rotating shell is rotated, so that the protective cover blocks the through hole.

[0020] Compared with the prior art, the present invention has the following beneficial effects: This rich text audio summary method based on LLM multi-feature fusion slicing converts voice data into text data, which is convenient for subsequent natural language processing and large language model analysis. It uses natural language processing and machine learning technology to predict the topic changes between sentences and perform map-reduce summaries on the topics. It can adapt to infinitely long audio texts and accurately summarize the meeting's general summary, topic summary, and speaker summary. This method can be effectively applied to various automatic summaries with rich text information, and is not limited to the length of the conference audio, and has a good audio text processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a structural flowchart of the rich text audio summarization method based on LLM multi-feature fusion slicing.

[0022] Figure 2 It is a schematic diagram of the main structure of the recording collection device.

[0023] Figure 3 It is a schematic diagram of the main cross-sectional structure of the recording collection device.

[0024] Figure 4 for Figure 3 A magnified view of the structure in the middle.

[0025] Figure 5 It is a bottom-up schematic diagram of the rotating cylinder, annular groove and spherical particle structure.

[0026] Among them: 1. Adjustment rod assembly; 101. Outer skin; 102. Metal wire; 103. Connecting boss; 104. Inner concave plate; 2. Fixed seat assembly; 201. Fixed plate; 202. Mounting groove; 203. Spring; 204. Ball; 3. Recording and radio assembly; 301. Rotating shell; 302. Recording and radio module; 303. Protective cover; 304. Through hole; 4. Rotating seat assembly; 401. Rotating cylinder; 402. Annular groove; 403. Spherical particles. DETAILED DESCRIPTION

[0027] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0028] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0029] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below through the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention. Example 1

[0030] See also Figure 1 , a rich text audio summarization method based on LLM multi-feature fusion slicing, specifically including the following steps: S1. Acquire and record audio files through the recording and sound receiving module in the recording collection device.

[0031] S2, identifying the speaker identity, the corresponding speaker speech content and the corresponding speaker speech speed in the audio file through the audio analysis module; where the speech speed = total number of words / audio duration, in minutes; The audio parsing module uses whisper X to segment the speaker's sentences, thereby identifying the speaker and the content of the speech, and extracting the speech rate based on the speech content. whisperX is an audio parsing algorithm for speech recognition and speaker recognition.

[0032] S3, obtaining features from the speaker's sentences through the audio analysis module, including speech rate change features, text vector features and transition keywords, and performing sentence topic segmentation; Text vector features include keyword tag extraction and topic extraction, such as transition keywords such as "next" or "next topic".

[0033] S4. By setting up a natural language processing parsing module to perform feature fusion, train an SVM support vector machine binary classification model, and judge whether two adjacent sentence topics are the same topic based on the fusion features between adjacent sentence topics; SVM support vector machine is a machine learning method used to solve classification problems; The natural language processing parsing module uses bge-m3 to calculate the semantic features of sentences, and integrates speech rate, text vector features and transition keywords to obtain fusion features, and trains the SVM support vector machine binary classification model to determine whether adjacent topics are sentences of the same topic.

[0034] Among them, bge-m3 is a text vectorization model algorithm, which is used to convert text into feature vectors. It is one of the most widely used and powerful models among the current open source pre-trained text vectorization models. It can extract text features from text and vectorize it. This process is called text vectorization.

[0035] Therefore, the features used above mainly include: the dense matrix features of the text itself corresponding to the text vector, the sparse matrix features corresponding to the transition keywords, and the speaking speed. In order to reduce algorithm misjudgment and enhance robustness, the sliding window concept is added. Each sentence will be judged against the previous three sentences. If the three previous sentences are not similar, it is considered to be the sentence starting a new topic.

[0036] S5, combining steps S3 and S4, obtaining sentences segmented by topic; S6. After dividing the topics, we use the LLM module to summarize each topic based on the map-reduce method to obtain the summary content of each topic; LLM module refers to the Large Language Model module, and map-reduce refers to mapping-reduction, a divide-and-conquer method that is used to split a large problem into small problems, solve the small problems, and then reduce the small problems. It is a computer thinking method that is widely used in many fields such as big data and artificial intelligence. The LLM module is to write prompt words, summarize each topic based on the speech content, and then summarize the topic to get a summary of the entire meeting.

[0037] S7. Then summarize each topic and execute the map-reduce process to get the overall summary.

[0038] S8. It also includes a voiceprint recognition module. For each speaker, the speaker's identity is combined with the actual defined identity through the voiceprint recognition module to obtain the specific person's speech content in the meeting or debate. The voiceprint module uses X-vector to perform speaker verification on each speaker to obtain the identity of each speaker. X-vector is an existing technology used.

[0039] Specifically, when in use, the conference audio file is first obtained through the recording and receiving module in the recording collection device, the audio file is analyzed and divided into topics through the audio analysis module, the natural language processing analysis module determines whether the topics of two adjacent sentences are the same topic, and obtains sentences divided according to the topic, and then obtains the summary content of each topic through the LLM module, and finally executes the map-reduce process, and then summarizes the summary of each topic to obtain an overall summary. Example 2

[0040] See also Figure 2-5 On the basis of the above embodiments, the rich text audio summarization method based on LLM multi-feature fusion slicing, the recording collection device includes an adjusting rod assembly 1, a fixed seat assembly 2, a recording and sound receiving assembly 3, a rotating seat assembly 4 and a base 5; one end of the adjusting rod assembly 1 is fixedly installed on the base 5, and the other end is fixedly connected to the fixed seat assembly 2, the rotating seat assembly 4 is rotatably installed on the fixed seat assembly 2, and the recording and sound receiving assembly 3 is fixedly installed above the rotating seat assembly 4.

[0041] Specifically, during use, when the recording collection device is needed to collect audio, the direction or height can be adjusted by adjusting the rod assembly 1 so that the recording and sound receiving assembly 3 faces the direction of sound collection, and then the recording and sound receiving assembly 3 can be adjusted to switch on and off through the setting of the fixed seat assembly 2 and the rotating seat assembly 4.

[0042] The solution in this embodiment can be selectively combined with the solutions in other embodiments for use. Example 3

[0043] See also Figure 2-5 , Based on the above embodiments, the rich text audio summarization method based on LLM multi-feature fusion slicing, the adjustment rod component 1 includes an outer skin 101, a metal wire 102, a connecting convex head 103 and an inner concave plate 104; The connecting convex head 103 and the concave plate 104 are respectively fixedly connected to the inside of the two ends of the outer skin 101, and the metal wire 102 is fixedly connected between the connecting convex head 103 and the concave plate 104, and is located inside the outer skin 101; multiple connecting convex heads 103 and concave plates 104 are fixed head to tail, the connecting convex head 103 at the outermost end is fixedly plugged on the base 5, and the concave plate 104 at the outermost end is fixedly connected to the bottom of the fixing seat assembly 2; The outer skin 101 is made of soft leather material, and the metal wire 102 can be made of other soft metal wire materials such as aluminum wire or iron wire. Through the material properties of the outer skin 101 and the metal wire 102, the adjusting rod assembly 1 can be bent or adjusted at any angle at each section of the rod.

[0044] The fixing seat assembly 2 includes a fixing plate 201, a mounting groove 202, a spring 203 and a ball 204; The bottom of the fixed disk 201 is fixed to the inner concave plate 104 at the outermost end, and mounting grooves 202 are opened at both ends thereof. One end of the spring 203 is fixed inside the mounting groove 202, and the other end is fixed to the ball 204, and the ball 204 can freely pass through the mounting groove 202; the rotating seat assembly 4 includes a rotating cylinder 401, an annular groove 402 and spherical particles 403; the middle bottom of the rotating cylinder 401 is rotatably connected to the top of the fixed disk 201 through a rotating shaft, and an annular groove 402 is opened on the edge of the rotating cylinder 401, and the spherical particles 403 are fixedly arranged inside the annular groove 402; when the rotating cylinder 401 is rotatably connected to the fixed disk 201, the ball 204 is pushed inside the annular groove 402 by the spring 203 to contact the spherical particles 403.

[0045] By cooperating with the fixed seat assembly 2 and the rotating seat assembly 4, the rotating cylinder 401 can be rotated arbitrarily on the fixed disk 201. At the same time, when the ball 204 is pushed into the annular groove 402 by the spring 203 and contacts the spherical particles 403, the rotating cylinder 401 has a damping feeling when rotating due to the bumps between the spherical particles 403, and can pause at any position.

[0046] The recording and sound receiving assembly 3 includes a rotating shell 301, a recording and sound receiving module 302, a protective cover 303 and a through hole 304; The bottom of the rotating shell 301 is fixed to the top of the rotating cylinder 401, and a plurality of through holes 304 are opened on the top thereof. The through holes 304 are arranged in a honeycomb hole structure, so that the recording and sound receiving module 302 can collect audio. The recording and sound receiving module 302 is fixedly arranged inside the rotating shell 301, and the protective cover 303 is sleeved on the half outer wall of the rotating shell 301, and the bottom thereof is fixed on the fixed disk 201. By rotating the rotating cylinder 401, the rotating shell 301 is rotated, so that the protective cover 303 covers the through holes 304. The through hole 304 is half-disposed on the top of the rotating shell 301. Through the coordinated arrangement of the rotating shell 301 and the protective cover 303, the protective cover 303 can expose or block the through hole 304. When the through hole 304 is exposed, the recording and radio module 302 can receive sound normally or better. When the through hole 304 is blocked, the recording and radio module 302 is separated from the outside and can stop receiving sound. At the same time, the inside of the rotating shell 301 can be protected from dust.

[0047] The solution in this embodiment can be selectively combined with the solutions in other embodiments for use.

[0048] It should be noted that although the above embodiments have been described in this article, the scope of patent protection of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures, equivalent processes or equivalent functional transformations made using the contents of the present specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields are all included in the scope of protection of the patent of the present invention.

Claims

1. A rich text audio summarization method based on LLM multi-feature fusion slicing, characterized in that: The specific steps are as follows: S1. Obtain and record audio files through the recording and sound receiving module in the recording collection device; S2, identifying the speaker identity, the corresponding speaker speech content and the corresponding speaker speech speed in the audio file through an audio analysis module; S3, obtaining features from the speaker's sentences through the audio analysis module, including speech rate change features, text vector features and transition keywords, and performing sentence topic segmentation; S4, by setting up a natural language processing parsing module to perform feature fusion, train an SVM support vector machine binary classification model, and determine whether the topics of two adjacent sentences are the same topic based on the fusion features between the topics of adjacent sentences; S5, combining steps S3 and S4, obtaining sentences segmented by topic; S6. After dividing the topics, we use the LLM module to summarize each topic based on the map-reduce method to obtain the summary content of each topic; S7. Then summarize each topic and execute the map-reduce process to get the overall summary.

2. According to claim 1, a rich text audio summarization method based on LLM multi-feature fusion slicing is characterized by: It also includes a voiceprint recognition module. For each speaker, the identity of the speaker is combined with the actual defined identity through the voiceprint recognition module to obtain the specific person's speech content in the meeting or debate.

3. According to claim 1, a method for summarizing rich text audio based on LLM multi-feature fusion slices is characterized in that: The audio analysis module uses whisper X to segment the speaker's sentences, thereby identifying the speaker and the speech content, and extracting the speech speed using the speech content; Among them, speaking speed = total number of words / audio duration.

4. According to claim 1, a method for summarizing rich text audio based on LLM multi-feature fusion slices is characterized in that: The natural language processing parsing module uses bge-m3 to calculate the semantic features of sentences, and integrates speech rate, text vector features and transition keywords to obtain fusion features, and trains an SVM support vector machine binary classification model to determine whether adjacent topics are sentences of the same topic.

5. According to claim 1, a method for summarizing rich text audio based on LLM multi-feature fusion slices is characterized by: The LLM module writes prompt words, summarizes each topic based on the speech content, and then summarizes the topic to obtain a summary of the entire meeting.

6. The method for summarizing rich text audio based on LLM multi-feature fusion slices according to claim 2 is characterized in that: The voiceprint module performs speaker verification on each speaker to obtain the identity of each speaker.

7. The method for summarizing rich text audio based on LLM multi-feature fusion slices according to claim 1 is characterized in that: The recording collection device comprises an adjustment rod assembly (1), a fixing seat assembly (2), a recording and sound receiving assembly (3), a rotating seat assembly (4) and a base (5); One end of the adjusting rod assembly (1) is fixedly mounted on the base (5), and the other end is fixedly connected to the fixed seat assembly (2); the rotating seat assembly (4) is rotatably mounted on the fixed seat assembly (2); and the recording and receiving assembly (3) is fixedly mounted above the rotating seat assembly (4).

8. The method for summarizing rich text audio based on LLM multi-feature fusion slices according to claim 7 is characterized in that: The regulating rod assembly (1) comprises an outer skin (101), a metal wire (102), a connecting protrusion (103) and an inner concave plate (104); The connecting convex head (103) and the concave plate (104) are respectively fixedly connected to the inside of the two ends of the outer skin (101); the metal wire (102) is fixedly connected between the connecting convex head (103) and the concave plate (104) and is located inside the outer skin (101); The plurality of connecting protrusions (103) and the inner concave plates (104) are fixed end to end, the connecting protrusion (103) at the outermost end is fixedly plugged onto the base (5), and the inner concave plate (104) at the outermost end is fixedly connected to the bottom of the fixing seat assembly (2).

9. The method for summarizing rich text audio based on LLM multi-feature fusion slices according to claim 8 is characterized in that: The fixing seat assembly (2) comprises a fixing plate (201), a mounting groove (202), a spring (203) and a ball (204); The bottom of the fixing plate (201) is fixed to the inner concave plate (104) at the outermost end, and the mounting grooves (202) are provided at both ends of the fixing plate. One end of the spring (203) is fixed inside the mounting groove (202), and the other end is fixed to the ball (204). The ball (204) can freely pass through the mounting groove (202). The rotating seat assembly (4) comprises a rotating cylinder (401), an annular groove (402) and spherical particles (403); The bottom of the middle end of the rotating cylinder (401) is rotatably connected to the top of the fixed disk (201) via a rotating shaft, the edge of the rotating cylinder (401) is provided with the annular groove (402), and the spherical particles (403) are fixedly arranged inside the annular groove (402); When the rotating cylinder (401) is rotatably connected to the fixed disk (201), the rolling balls (204) are pushed into the annular groove (402) by the spring (203) to come into contact with the spherical particles (403).

10. The method for summarizing rich text audio based on LLM multi-feature fusion slices according to claim 9 is characterized in that: The recording and sound receiving component (3) comprises a rotating shell (301), a recording and sound receiving module (302), a protective cover (303) and a through hole (304); The bottom of the rotating shell (301) is fixed to the top of the rotating cylinder (401), and a plurality of through holes (304) are provided on the top thereof, wherein the through holes (304) are arranged in a honeycomb hole structure to facilitate the recording and sound receiving module (302) to collect audio; The recording and receiving module (302) is fixedly arranged inside the rotating shell (301), the protective cover (303) is sleeved on half of the outer wall of the rotating shell (301), and its bottom is fixed on the fixed disk (201), and the rotating shell (301) is rotated by rotating the rotating cylinder (401), so that the protective cover (303) covers the through hole (304).

Images