Large model-based data processing method, apparatus and device, and storage medium

Abstract

The invention provides a data processing method and device based on a large model, equipment and a storage medium, and relates to the technical field of data processing, in particular to the technical fields of artificial intelligence, big data, large models and the like. The specific implementation scheme is as follows: performing voice activity detection on a target dialogue audio to obtain a plurality of target audio clips with voice activities; obtaining target text content based on the plurality of target audio clips; performing intention extraction on the target text content by utilizing a large model, and extracting at least one of a question text related to a question and an answer text related to an answer; based on the time information in the target dialogue audio and the extracted text, a target dialogue text with the time information associated with the target dialogue audio is obtained, and the target dialogue text comprises at least one of a question and answer pair with the time information and a question text with the time information.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to the technical fields of artificial intelligence, big data, and big models. Background Art

[0002] Offline quality inspection is a common and universal scenario for dual-recording quality inspection. The existing stock of data that needs to be inspected according to the dual-recording regulations is huge. The use of offline quality inspection can greatly relieve the workload of manual review and reduce the labor cost of quality inspection work. Moreover, it can also solve the risks and omissions caused by manual quality inspection to a certain extent.

[0003] In the scenario of offline quality inspection of dual recording, the current solution is mainly to record dot information during the audio and video recording process to distinguish the various stages in the audio and video. In other words, to determine the specific process of the audio and video. However, many audio and video that need to be quality inspected do not have dot information during the recording process, making it impossible to know the specific process of the entire audio and video, resulting in the inability to effectively perform quality inspection. Summary of the invention

[0004] The present invention provides a data processing method, device, equipment and storage medium based on a large model.

[0005] According to one aspect of the present disclosure, a data processing method based on a large model is provided, comprising:

[0006] Performing voice activity detection on the target conversation audio to obtain multiple target audio segments with voice activities;

[0007] Based on the multiple target audio segments, obtaining target text content;

[0008] Using the large model, extract the intent of the target text content, and extract at least one of the following: question text related to the question, and reply text related to the answer;

[0009] Based on the time information in the target conversation audio and the extracted text, a target conversation text with time information associated with the target conversation audio is obtained, wherein the target conversation text includes at least one of the following: a question-answer pair with time information, and a question text with time information.

[0010] According to another aspect of the present disclosure, there is provided a data processing device based on a large model, comprising:

[0011] An audio segmentation unit, used to perform voice activity detection on the target conversation audio to obtain multiple target audio segments with voice activities;

[0012] An audio conversion unit, configured to obtain target text content based on the multiple target audio segments;

[0013] A text processing unit is used to use a large model to extract the intent of the target text content and extract at least one of the following: question text related to the question, and reply text related to the answer; based on the time information in the target conversation audio and the extracted text, a target conversation text with time information associated with the target conversation audio is obtained, wherein the target conversation text includes at least one of the following: a question and answer pair with time information, and a question text with time information.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] In this way, the disclosed scheme can obtain multiple target audio segments based on the target conversation audio to obtain the target text content, and then perform intent extraction on the target text content to obtain the question text related to the question and / or the reply text related to the answer, and then further process the extracted text (the question text related to the question, or the reply text related to the answer) according to the time information in the target conversation audio to obtain the target conversation text with time information. In this way, the conversation text is automatically and accurately extracted from the target conversation audio. At the same time, audio segmentation of the target conversation audio is realized, which provides strong support for the subsequent improvement of the quality inspection efficiency of audio data.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 This is a schematic flow chart of a data processing method based on a large model according to an embodiment of the present application. Figure 1 ;

[0024] Figure 2 This is a schematic flow chart of a data processing method based on a large model according to an embodiment of the present application. Figure 2 ;

[0025] Figure 3 This is a schematic flow chart of a data processing method based on a large model according to an embodiment of the present application. Figure 3 ;

[0026] Figure 4 This is a schematic flow chart of a data processing method based on a large model according to an embodiment of the present application. Figure 4 ;

[0027] Figure 5 is a schematic diagram of the result of performing voice activity detection on the target conversation audio according to an embodiment of the present application;

[0028] Figure 6 This is a flowchart of a data processing method based on a large model in a specific example according to an embodiment of the present application. Figure 1 ;

[0029] FIG. 7( a ) is a schematic diagram of a processing flow of a video to audio conversion module according to an embodiment of the present application;

[0030] FIG7( b ) is a schematic diagram of a training and reasoning process of a general text classification model according to an embodiment of the present application;

[0031] FIG7( c ) is a schematic diagram of speech template matching according to an embodiment of the present application;

[0032] Figure 8 This is a flowchart of a data processing method based on a large model in a specific example according to an embodiment of the present application. Figure 2 ;

[0033] Fig. 9 is a structural schematic diagram of a data processing device based on a large model according to an embodiment of the present application;

[0034] Fig.10 It is a block diagram of an electronic device used to implement the data processing method based on a large model in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0036] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0037] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0038] Currently, many audio and video files that need to be quality-checked do not have any marking information during the recording process (i.e., timestamps, event descriptions, and other marking information added at key links in the audio and video). As a result, it is impossible to know the specific processes of each stage in the entire audio and video recording process, and thus it is impossible to improve quality inspection capabilities through AI empowerment.

[0039] Based on this, the disclosed scheme provides a data processing method, which can automatically and accurately extract the dialogue text for audio data or video data without marking information, and then know the stage it is in to achieve segmented marking. This provides strong support for the subsequent improvement of the quality inspection efficiency of audio data or video data.

[0040] Specifically, Figure 1 This is a schematic flow chart of a data processing method based on a large model according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0041] Further, the method includes at least part of the following contents. Figure 1 As shown, including:

[0042] Step S101: performing voice activity detection on the target conversation audio to obtain a plurality of target audio segments having voice activities.

[0043] For example, in one example, after performing voice activity detection on the target conversation audio, the target conversation audio can be segmented according to the detection results to obtain multiple target audio segments with relatively "active" voice activities.

[0044] Step S102: obtaining target text content based on the multiple target audio segments.

[0045] Step S103: using the large model, extracting the intent of the target text content, and extracting at least one of the following: question text related to the question, and reply text related to the answer.

[0046] Step S104: Based on the time information in the target conversation audio and the extracted text, a target conversation text with time information associated with the target conversation audio is obtained.

[0047] Here, the target dialogue text includes at least one of the following: a question-answer pair with time information, and a question text with time information.

[0048] For example, in one example, the question-answer pair with time information may specifically include a question text with time information and a reply text with time information and related to the question text. For example, the question-answer pair with time information is specifically:

[0049] "0:00:00~0:00:16.235714: In order to standardize the sales behavior of sales personnel and to better protect your legal rights and interests, according to relevant regulations, we will record the key links of my sales process by audio and video. Do you agree?",

[0050] "0:00:16.235714~0:00:17.219000: Agree, agree."

[0051] In this way, the disclosed scheme can obtain multiple target audio segments based on the target conversation audio to obtain the target text content, and then perform intent extraction on the target text content to obtain the question text related to the question and / or the reply text related to the answer, and then further process the extracted text (the question text related to the question, or the reply text related to the answer) according to the time information in the target conversation audio to obtain the target conversation text with time information. In this way, the conversation text is automatically and accurately extracted from the target conversation audio. At the same time, audio segmentation of the target conversation audio is realized, which provides strong support for the subsequent improvement of the quality inspection efficiency of audio data.

[0052] It should be noted that there will be obvious pauses in actual conversation scenarios. Therefore, in the present disclosure, voice activity detection is performed on the target conversation audio to obtain the audio activity part. For example, a voice activity detection (VAD) algorithm can be used to perform voice activity detection on the target conversation audio. This provides strong support for the subsequent accurate acquisition of the target text content. At the same time, it also improves the efficiency of text generation, and further provides strong support for improving the accuracy of audio segmentation.

[0053] Here, in one example, the target conversation audio may specifically be audio data recorded using an audio acquisition device, for example, audio data obtained after recording using an audio acquisition device in a dual recording scenario.

[0054] Alternatively, in another example, the target conversation audio may be audio data obtained by performing audio extraction on a video captured by a video capture device, for example, audio data obtained by performing audio extraction on a video captured by a video capture device in a dual recording scenario. For example, in one example, the Fast Forward moving picture experts group (FFmpeg) tool is used to perform audio extraction on a recorded video to obtain the target conversation audio.

[0055] Furthermore, in one example, after extracting audio data from a video, the audio data may be format converted. For example, the audio format of the extracted audio data may be converted into a waveform audio file format (Waveform Audio File Format, WAV), thereby obtaining a target conversation audio that meets the preset audio format requirements. This facilitates subsequent rapid text transcription of the target conversation audio.

[0056] It is understandable that the disclosed solution does not impose any specific restrictions on the source and extraction method of the target conversation audio.

[0057] It should be pointed out that the "dual recording" scenario referred to in the present disclosure may specifically refer to a scenario in which the voice or video of both parties in the conversation is recorded. For example, in a scenario in which a salesperson and a customer are conducting business or transactions, the conversation content and operation process between the salesperson and the customer can be recorded through a collection device (such as a video collection device or an audio collection device, etc.).

[0058] In a specific example, the following method can be used to extract intent; specifically, the above-mentioned extraction of intent from the target text content (e.g., step S103) can specifically include: using a large model (e.g., a large language model (LLM)) to extract intent from the target text content. In this way, the question text and the answer text can be efficiently and accurately extracted from the target text content, which greatly improves the efficiency of data processing and ensures the accuracy of data extraction.

[0059] Figure 2 This is a schematic flow chart of a data processing method based on a large model according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0060] Further, the method includes at least part of the following contents. Figure 2 As shown, including:

[0061] Step S201: performing voice activity detection on the target conversation audio to obtain a plurality of target audio segments having voice activities.

[0062] It should be noted that the relevant content about the target dialogue audio and the target audio segment can be referred to the above examples and will not be repeated here.

[0063] Step S202: Based on the multiple target audio segments, obtain a text segment of each target audio segment.

[0064] For example, in one example, a speech recognition model (such as a Convolution-augmented transformer (Conformer) model) can be used to perform speech recognition on each target audio segment to obtain a text segment of each target audio segment. Furthermore, in order to facilitate the subsequent rapid acquisition of the target dialogue text with time information, the text segment of each target audio segment can also carry time information.

[0065] Step S203: performing denoising processing on the text segment of each target audio segment to obtain the denoised text segment of each target audio segment.

[0066] Step S204: splicing the text segments of each target audio segment after denoising to obtain target text content.

[0067] Step S205: using the large model, extracting the intent of the target text content, and extracting at least one of the following: question text related to the question, and reply text related to the answer.

[0068] Step S206: Based on the time information in the target conversation audio and the extracted text, a target conversation text with time information associated with the target conversation audio is obtained.

[0069] Here, the target dialogue text includes at least one of the following: a question-answer pair with time information, and a question text with time information.

[0070] In this way, the disclosed scheme can perform denoising on the text segments of each target audio segment after obtaining the text segments of each target audio segment. In this way, the text quality of the obtained target text content is effectively improved, providing data support for subsequent intent extraction. At the same time, it also provides strong support for the subsequent improvement of the quality inspection efficiency of audio data.

[0071] It should be noted that the text segment of the target audio segment mentioned above can be regarded as a conversation scene, such as a conversation scene between a salesperson and a customer. The conversation content in the conversation scene may contain audio data that is irrelevant to the business content. Therefore, in order to improve the efficiency of subsequent quality inspection, after obtaining the text segment of the target audio segment, denoising processing can be performed to remove irrelevant content.

[0072] Specifically, in a specific example, the denoising process may be performed in the following manner; specifically, the denoising process of the text segments of each target audio segment described above to obtain the denoised text segments of each target audio segment (for example, step S203) may specifically include:

[0073] Step S203 - 1 : Using the first model, the text segment of the target audio segment is processed to determine whether there are sentences in the text segment that are irrelevant to the target business scenario.

[0074] Here, the target business scenario refers to the business scenario targeted by the target conversation audio.

[0075] Step S203 - 2 : when it is determined that there are sentences irrelevant to the target business scenario, the sentences irrelevant to the target business scenario are deleted from the text segment of the target audio segment to obtain the denoised text segment of the target audio segment.

[0076] Here, it can be understood that, when it is determined that there are no sentences irrelevant to the target business scenario, in other words, each sentence in the text segment of the target audio segment is related to the target business scenario, then no deletion processing is required.

[0077] In this way, the disclosed solution can use the first model to identify sentences that are irrelevant to the target business scenario from the text segments of each target audio segment, and then delete such sentences. In this way, the denoising of the text segments of each target audio segment is achieved, and the text quality of the obtained target text content is effectively improved, laying the foundation for subsequent improvement of quality inspection efficiency, such as improving the quality inspection efficiency of dual-recording data.

[0078] Further, in an example, the above-described processing of the text segment of the target audio segment using the first model to determine whether there are sentences irrelevant to the target business scenario in the text segment (for example, step S203-1) may specifically include:

[0079] Step S203-1-1: using the first model, classify the sentences in the text segment of the target audio segment to obtain a classification result for the text segment of the target audio segment.

[0080] Here, the classification result indicates that the sentence in the text segment belongs to one of the following: noise, question, answer; noise indicates irrelevant to the target business scenario; question indicates a question related to the target business scenario, and answer indicates a reply related to the target business scenario.

[0081] Step S203 - 1 - 2 : Based on the classification result of the text segment of the target audio segment, determine whether there is noise in the text segment.

[0082] That is to say, the first model can be specifically a pre-trained classification model, for example, it can be a multi-classification model, etc. Furthermore, in this example, the classification ability of the first model is used to determine the category to which the sentences in the text segment of the target audio segment belong. When it is determined that there are sentences classified as noise, the sentences classified as noise are deleted. This lays the foundation for improving the text quality of the obtained target text content, and at the same time, it also lays the foundation for improving quality inspection efficiency, such as improving the quality inspection efficiency of dual-recording data.

[0083] Figure 3 This is a schematic flow chart of a data processing method based on a large model according to an embodiment of the present application. Figure 3The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figure 1 and Figure 2 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0084] Further, the method includes at least part of the following contents. Figure 3 As shown, including:

[0085] Step S301: Perform voice activity detection on the target conversation audio to obtain multiple target audio segments with voice activities.

[0086] It should be noted that the relevant content about the target dialogue audio and the target audio segment can be referred to the above examples and will not be repeated here.

[0087] Step S302: Based on the multiple target audio segments, obtain a text segment of each target audio segment.

[0088] It should be noted that the relevant content of the text segment of the target audio segment can be referred to the above example, which will not be repeated here.

[0089] Step S303: performing denoising processing on the text segment of each target audio segment to obtain the denoised text segment of each target audio segment.

[0090] Step S304: splicing the text segments of each target audio segment after denoising to obtain target text content.

[0091] Step S305: using the large model, extracting the intent of the target text content, and extracting at least one of the following: question text related to the question, and reply text related to the answer.

[0092] Step S306: Based on the time information in the target conversation audio and the extracted text, a target conversation text with time information associated with the target conversation audio is obtained.

[0093] Here, the target dialogue text includes at least one of the following: a question-answer pair with time information, and a question text with time information.

[0094] Step S307: determining the question-answering session involved in the text segment of each target audio segment after denoising.

[0095] Here, step S307 may be executed after step S303, or may be executed after obtaining the target dialogue text. The disclosed solution does not specifically limit the execution order of step S307.

[0096] For instance, in one example, the text segment of the target audio segment can be first vectorized, for example, the text segment of the target audio segment is input into a vectorization service engine to obtain the vectorized text content; secondly, in the vector index library, a query is performed to obtain K (an integer greater than or equal to 1) recall results with the highest semantic similarity to the vectorized text content; finally, based on the K recall results, the speech template corresponding to the text segment of the denoised target audio segment is determined, and then based on the speech template corresponding to the text segment of the denoised target audio segment, the question and answer link involved in the text segment of the denoised target audio segment is determined.

[0097] Here, the vector index library is constructed based on data obtained after vectorizing multiple preset speech templates.

[0098] Step S308: merging the contents of the question-and-answer session associated with the target dialogue text to obtain a target dialogue text marked with the question-and-answer session.

[0099] For example, the contents belonging to the same question-and-answer session in the target dialogue text are merged to obtain a target dialogue text marked with a question-and-answer session. Alternatively, the contents of question-and-answer sessions with upper and lower dependencies in the target dialogue text can also be merged to obtain a target dialogue text marked with a question-and-answer session. The above is only an exemplary description. In actual applications, merging can be performed based on scenario requirements, and the disclosed solution does not impose specific restrictions on this.

[0100] In this way, the disclosed solution can automatically and efficiently obtain the target conversation text annotated with question and answer sessions and time information based on the target conversation audio, effectively solving the problem that the existing audio and video that needs quality inspection cannot determine the specific process due to the lack of marking information, and thus cannot effectively perform artificial intelligence (AI) quality inspection. At the same time, it also effectively avoids the problems of high manual quality inspection costs and omissions in manual quality inspection due to the inability to perform AI quality inspection, thereby laying the foundation for improving quality inspection efficiency, such as improving the quality inspection efficiency of dual-recording data.

[0101] Figure 4 This is a schematic flow chart of a data processing method based on a large model according to an embodiment of the present application. Figure 4 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices. It can be understood that the above Figures 1 to 3 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0102] Further, the method includes at least part of the following contents. Figure 4 As shown, including:

[0103] Step S401: Perform voice activity detection on the target conversation audio to obtain a plurality of initial audio segments with voice activities.

[0104] For example, in one example, the target conversation audio is detected using the voice activity detection (VAD) algorithm to obtain Figure 5 The detection result shown in FIG. 1 may include multiple voice activity intervals, such as voice activity interval 1, voice activity interval 2, ..., voice activity interval 7. At this time, the target conversation audio may be segmented according to each voice activity interval in the detection result to obtain initial audio segments corresponding to each voice activity interval, and these initial audio segments are saved in WAV format respectively. For example, 7 initial audio segments as shown below may be obtained:

[0105] Initial audio clip 1: 001_0_17219.wav;

[0106] Initial audio clip 2: 002_18770_39100.wav;

[0107] Initial audio clip 3: 003_39410_45959.wav;

[0108] Initial audio clip 4: 004_48619_50519.wav;

[0109] Initial audio clip 5: 005_56849_57639.wav;

[0110] Initial audio clip 6: 006_58719_59299.wav;

[0111] Initial audio clip 7: 007_60219_75700.wav.

[0112] Here, it should be pointed out that the naming format of the above initial audio segment is idx_start_end.wav, wherein idx represents the sequence number of the initial audio segment, start represents the start time of the initial audio segment in the target conversation audio, and end represents the end time of the initial audio segment in the target conversation audio.

[0113] Step S402: determining an initial audio segment whose frequency falls within a target frequency range from the multiple initial audio segments, and taking the initial audio segment falling within the target frequency range as a target audio segment.

[0114] It is understandable that non-human voice noise data (such as keyboard typing, the sound of vehicles passing on the road, etc.) may exist in the target conversation audio. Therefore, in order to further improve the accuracy of the obtained target text content and avoid noise interference, the disclosed solution can also perform non-human voice filtering on multiple initial audio clips, such as using the target frequency range to perform non-human voice filtering on multiple initial audio clips.

[0115] Further, in an example, the target frequency range mentioned above may specifically be the frequency range of human voice. Further, the frequency value of the initial audio segment may specifically be the average value or median of the frequency values ​​corresponding to each time frame in the initial audio segment.

[0116] For example, taking 7 initial audio clips as an example, first, the frequency value corresponding to each time frame in the initial audio clip is used to obtain the average frequency value of the initial audio clip, so as to obtain the average frequency value of each initial audio clip; secondly, from the 7 initial audio clips, the initial audio clips whose average frequency values ​​fall into the target frequency range are determined: initial audio clip 1, initial audio clip 2, initial audio clip 3 and initial audio clip 7; finally, the initial audio clip 1 and the initial audio clip 2 that fall into the target frequency range are taken as target audio clip 2, and the initial audio clip 3 and the initial audio clip 7 that fall into the target frequency range are both taken as target audio clips.

[0117] Alternatively, in another example, it is also possible to determine whether the audio intensity of the initial audio segment that falls within the target frequency range is greater than or equal to a preset threshold. If so, it indicates that the initial audio segment contains human voice. In this case, the initial audio segment can be used as the target audio segment; otherwise, that is, the audio intensity of the initial audio segment that falls within the target frequency range is less than the preset threshold, indicating that the initial audio segment only contains non-human voice noise. In this case, the initial audio segment needs to be removed, thereby achieving non-human voice filtering of multiple initial audio segments.

[0118] Further, in an example, before determining an initial audio segment whose frequency falls within the target frequency range from the multiple initial audio segments and using the initial audio segment falling within the target frequency range as the target audio segment (for example, before step S402), determining the target frequency range specifically includes:

[0119] Determine the frequency spectrum information corresponding to the time frame included in each initial audio segment;

[0120] Based on the spectrum information corresponding to the time frames included in each initial audio segment, obtaining the frequency value corresponding to the time frames included in each initial audio segment;

[0121] The target frequency range is determined based on the frequency values ​​corresponding to the time frames included in each initial audio segment.

[0122] In this way, the disclosed scheme provides a refined scheme for obtaining the target audio range, which is simple, efficient, and can quickly obtain the target audio range. In this way, it provides strong support for obtaining a target audio segment with high precision. At the same time, it also lays a foundation for improving the text quality of the obtained target text content, and further lays a foundation for improving quality inspection efficiency, such as improving the quality inspection efficiency of dual-recording data.

[0123] Step S403: obtaining target text content based on the multiple target audio segments.

[0124] Step S404: using the large model, extract the intent of the target text content, and extract at least one of the following: question text related to the question, and reply text related to the answer.

[0125] Step S405: Based on the time information in the target conversation audio and the extracted text, a target conversation text with time information associated with the target conversation audio is obtained.

[0126] Here, the target dialogue text includes at least one of the following: a question-answer pair with time information, and a question text with time information.

[0127] In this way, the disclosed solution provides a refined solution for extracting multiple target audio segments from the target conversation audio, and the solution can easily and efficiently extract high-quality target audio segments, thus providing strong support for accurately extracting key text information and obtaining high-quality conversation text in the subsequent process. In addition, for audio data or video data without dot information, the disclosed solution can still automatically and accurately extract the conversation text, and then implement segmented dot, thus providing strong support for improving the efficiency of quality inspection of audio data or video data in the subsequent process.

[0128] The following is a further explanation of the disclosed solution in conjunction with specific examples and drawings; the disclosed solution provides a video (or audio) segmentation method based on a large model. For example, in this example, it can be divided into the following modules according to function: video to audio module, speech recognition module, dialogue denoising module, intention extraction module, time backtracking module, speech matching module, link merging module, among which the intention extraction module and the speech matching module can both be implemented using the large model technology. Figure 6 As shown in the figure, the modules are interconnected to form a complete end-to-end video segmentation service, which can greatly improve the accuracy and efficiency of video review in the dual-recording offline quality inspection scenario.

[0129] Specifically, Figure 6As shown, the disclosed solution specifically includes:

[0130] (1) Video to Audio Module

[0131] The video to audio module is mainly used to extract audio from the video to be inspected to obtain the dialogue audio of the video to be inspected (that is, corresponding to the target dialogue audio mentioned above). For example, the Fast Forward Picture Experts Group (FFmpeg) tool is used to extract audio from the video to be inspected to obtain the target dialogue audio.

[0132] It should be noted that in actual scenarios, the image stream in the video to be inspected may be out of sync with the audio stream in the conversation audio. In this case, the time information of the image stream in the video to be inspected and the time information of the audio stream in the target conversation audio can also be obtained, and then the image stream in the video and the audio stream in the conversation audio can be aligned according to the time interval between the time information of the image stream and the time information of the audio stream.

[0133] Here, after obtaining the target conversation audio, the audio format of the target conversation audio can also be converted into a required audio format, such as 16K sampling rate, mono, and WAV format, so as to facilitate the subsequent transcription of the conversation audio.

[0134] (II) Speech Recognition Module

[0135] The video-to-audio module is mainly used to process the target conversation audio to obtain multiple text segments with timestamps (i.e., text segments corresponding to the target audio segments mentioned above). The speech recognition module can also specifically include three sub-modules: audio segmentation, non-human voice filtering, and audio transcription. In this way, the human voice pauses in the target conversation audio can be separated, providing strong support for the subsequent establishment of the conversation scene. Specifically, as shown in Figure 7(a), it includes:

[0136] (1) Audio Slicing Submodule

[0137] In an actual dual-recording scenario, since there are usually obvious pauses in the conversation process between a text-to-speech (TTS) broadcaster (such as an account manager) and a customer, the target conversation audio can be segmented based on this audio feature to obtain multiple audio segments (corresponding to the multiple initial audio segments mentioned above).

[0138] For example, the voice activity detection (VAD) algorithm is used to detect the target conversation audio and obtain the following Figure 5The detection results shown are used to segment the target conversation audio according to the detection results, and the initial audio segments obtained after segmentation are saved in WAV format respectively to obtain multiple initial audio segments in WAV format (for example, the audio segments shown below are obtained), and the timestamps of each audio segment in the target conversation audio are recorded at the same time.

[0139] The initial audio clips obtained after segmentation are as follows:

[0140] 001_0_17219.wav;

[0141] 002_18770_39100.wav;

[0142] 003_39410_45959.wav;

[0143] 004_48619_50519.wav;

[0144] 005_56849_57639.wav;

[0145] 006_58719_59299.wav;

[0146] 007_60219_75700.wav;

[0147] 008_76270_77450.wav;

[0148] 009_88150_93420.wav;

[0149] 010_94550_95830.wav;

[0150] 011_97380_127310.wav;

[0151] 012_128540_131400.wav;

[0152] 013_132040_132810.wav;

[0153] 014_132810_166999.wav;

[0154] 015_166999_169309.wav;

[0155] 016_170069_171979.wav.

[0156] It should be noted that the naming format of the above initial audio segment is idx_start_end.wav, wherein idx represents the sequence number of the initial audio segment, start represents the start time of the initial audio segment in the target conversation audio, and end represents the end time of the initial audio segment in the target conversation audio.

[0157] (2) Non-human voice filtering submodule

[0158] In an actual dual-recording scenario, the audio and video recording process may be in an open environment, so there will be non-human voice noise (such as keyboard tapping, the sound of vehicles passing on the road, etc.), that is, the initial audio segment obtained after the above segmentation may contain non-human voice. Based on this, the non-human voice filtering submodule can also be used to remove the non-human voice in the initial audio segment obtained above, so as to achieve non-human voice filtering of the initial audio segment.

[0159] Specifically, first, the human voice frequency is calculated using the fast Fourier transform. For example, the specific steps are as follows:

[0160] Step a: Divide the initial audio clip into multiple time frames;

[0161] Step b: Use fast Fourier transform to calculate the spectrum information of each time frame (comparison

[0162] Such as amplitude spectrum and power spectrum, etc.);

[0163] Step c: For a time frame, determine the frequency spectrum information of the time frame.

[0164] The initial frequency value corresponding to the frequency point;

[0165] Step d: Determine the frequency of the human

[0166] at least one target frequency value of a sound frequency range;

[0167] At this point, repeat the above steps c and d to obtain the target frequency corresponding to each time frame.

[0168] rate value;

[0169] Step e: Calculate the average value of all target frequency values ​​as the vocal frequency of the initial audio segment.

[0170] In this way, the human voice frequency range (corresponding to the target frequency range described above) can be obtained, and then the overall strength of the human voice in the initial audio signal can be evaluated based on the human voice frequency range. For example, it is determined whether the overall strength of the human voice in the initial audio signal that falls within the human voice frequency range is greater than a preset threshold. If so, it is considered that the initial audio signal contains human voice, otherwise, it can be considered that the initial audio information does not contain human voice. Such a processing method can effectively determine whether there is human voice in the audio clip and eliminate the interference of non-human voice noise, thus effectively improving the transcription accuracy and speech recognition effect.

[0171] For example, non-human voice filtering is performed on each initial audio segment obtained after audio segmentation to obtain the target audio segment shown below:

[0172] 001_0_17219.wav;

[0173] 002_18770_39100.wav;

[0174] 003_39410_45959.wav;

[0175] 007_60219_75700.wav;

[0176] 009_88150_93420.wav;

[0177] 011_97380_127310.wav;

[0178] 012_128540_131400.wav;

[0179] 014_132810_166999.wav;

[0180] 015_166999_169309.wav;

[0181] 016_170069_171979.wav.

[0182] (3) Audio transcription submodule

[0183] The audio transcription submodule is mainly used to transcribe the target audio segments after non-human voice filtering using the Automatic Speech Recognition (ASR) model to obtain the text segments of the target audio segments. Furthermore, the text segments of the target audio segments can be assigned corresponding timestamps according to the timestamps (such as start time and end time) recorded in the audio segmentation submodule.

[0184] For example, for the target audio segments obtained above, the Conformer model can be used to transcribe the audio to obtain text segments with timestamps. For example, the following text segments are obtained by transcribing the target audio segments above:

[0185] 0:00:00~0:00:17.219000: In order to standardize the sales behavior of sales personnel and to better protect

[0186] To protect your legal rights and interests, we will record and videotape the key links of my sales process.

[0187] Do you agree to this? Agree.

[0188] 0:00:18.770000~0:00:39.100000:We need to remind you that this recording and video recording process is

[0189] It is very important for you to protect your rights in the future. Please read the specific contents of the document you signed carefully and answer the relevant questions truthfully.

[0190] If I make any promise to you that is inconsistent with the written document, I suggest you contact me in writing.

[0191] Please confirm in the form to better protect your legal rights and interests.

[0192] 0:00:39.410000~0:00:45.959000:Please test the tester of road B gmh test exhibition

[0193] Show qualification certificate.

[0194] 0:01:00.219000~0:01:15.700000:Hello Mr. ××, before confirming the purchase, in order to protect your legal rights, we need to collect your face and identity information for personal verification. Please hold the back of your ID card facing

[0195] The camera completes the collection of identity information.

[0196] 0:01:28.150000~0:01:33.420000: Please hold your ID card and face the camera to complete the ID verification.

[0197] Information collection.

[0198] 0:01:37.380000 0:02:07.310000: You are about to purchase the ×× product of ×× test B route

[0199] The product number is 0, 1, 4, 6, 2. High risk, high income, say goodbye to part-time work. The purchase amount is 20.

[0200] This product is a financial plan of ×× Financial Management Subsidiary Company, not ×× Savings or

[0201] The financial product and the operating entity are the Earth First Financial Group. ×× Finance is the sales agency of this product.

[0202] Are you aware of the investment management responsibility behind the product?

[0203] 0:02:08.540000~0:02:11.400000:Got it.

[0204] 0:02:12.810000~0:02:46.999000: You need to purchase the three books of our bank's financial products

[0205] The product type of the product is financial sales, high risk, high income, say goodbye to working, and go one step further. The risk rating of this product in our bank is R5. Its risk is the highest. The return fluctuation is very large. The risk of principal loss is very large. The corresponding financial products include private equity funds, etc. Your risk rating in our bank is A2, which belongs to the stable type and is lower than R5.

[0206] period and risk tolerance investment objectives investment expectations investment goals may I ask if you make the decision to purchase independently

[0207] This product does not believe that the product is completely suitable for my product.

[0208] 0:02:46.999000~0:02:49.309000:Are you sure?

[0209] 0:02:50.069000~0:02:51.979000:Confirmed.

[0210] (III) Dialogue denoising module

[0211] In an actual dual recording scenario, the text segment of the target audio segment can be regarded as multiple rounds of conversations between the account manager and the customer, and the content of each round of conversation is theoretically conducted in accordance with the dual recording script related to the business content. However, since the recording process is in an open environment, some conversation content unrelated to the business content may be recorded during the recording of a normal question-and-answer session. Therefore, the disclosed solution needs to use a conversation denoising module to filter the text segments of each target audio segment obtained by audio transcription to achieve the purpose of conversation denoising.

[0212] Specifically, the dialogue denoising module is mainly used to process the text segment of the target audio segment through a classification model to determine whether there are sentences in the text segment that are irrelevant to the business content.

[0213] Here, taking the Universal Text Classification (UTC) model as an example, as shown in FIG7(b), the specific steps of using the model for denoising include:

[0214] (1) Offline: Model training

[0215] Step a: Use a text annotation tool (Doccano) to annotate each text data in the text dataset under the dialogue scenario so that each text data corresponds to a label, thereby obtaining a text dataset with labels.

[0216] Here, the labels include question, answer, and noise, among which the "question" label indicates that the current text data belongs to the text of the account manager reporting the product, or the text of the question raised to the customer, the "answer" label indicates that the current text data is the customer's reply text, and the "noise" label indicates that the current text data is irrelevant to the current business scenario.

[0217] Step b: Divide the labeled text dataset into a training set, a validation set, and a test set according to a preset ratio (for example, 8:1:1).

[0218] Step c: Use the training set to train the UTC model, and use the validation set and the test set to evaluate the trained model. Then, according to the evaluation results, fine-tune the parameters of the UTC model until the training obtains a UTC model whose evaluation results meet the preset requirements.

[0219] Step d: Use containerization technology to deploy the trained UTC model on the server and combine it with the backend service framework to provide inference services.

[0220] Here, in the actual reasoning scenario, a fast application programming interface (Fast API) backend framework developed by a specific programming language (such as Python) can also be used to provide reasoning services, and the GPU can be enabled to improve the reasoning performance.

[0221] (2) Online: Model Inference

[0222] Step e: Input the text segment obtained by audio transcription into the trained UTC model to classify the sentences in the text segment and obtain the classification result of the text segment. Then, according to the classification result of the text segment, determine whether the sentence in the text segment belongs to "noise". If the sentence in the text segment belongs to "noise", the sentence can be deleted from the text segment, or the text segment where the sentence is located can be deleted. In this way, by traversing each text segment, each text segment after dialogue denoising can be obtained.

[0223] (IV) Intent Extraction Module

[0224] In an actual dual-recording scenario, in the multiple rounds of conversations between the account manager and the customer, in addition to conversation content unrelated to the business content, there are also cases of rushing to answer (that is, the customer answers the account manager's question in advance without waiting for the account manager to finish the words specified in the dual-recording scenario). As a result, when the conversation audio is segmented, it is impossible to find obvious pauses in the human voice, and thus it is impossible to separate the audio related to the customer's reply and the audio related to the account manager's question from the conversation audio.

[0225] To solve the above problems, the disclosed solution needs to utilize an intent extraction module, such as utilizing a large model, to extract question text related to the question and reply text related to the answer from multiple rounds of conversations between the account manager and the customer.

[0226] Here, the task of the big model is to distinguish between the text belonging to "questions" and the text belonging to "answers" in the text content. For example, the big model can roughly divide the text in the text content into three categories, namely "questions", "answers", and "please take an action". Among them, "questions" means that the currently extracted text belongs to the question text of the account manager; "answers" means that the currently extracted text belongs to the customer's reply text; "please take an action" means that the currently extracted text belongs to the question text raised by the account manager and does not require a voice answer from the customer, but needs to be responded to through actions.

[0227] Specifically, the specific steps of intent extraction include:

[0228] Step a: According to the time sequence, the text segments of each target audio segment after denoising are merged to obtain the merged text content (that is, corresponding to the target text content mentioned above).

[0229] Step b: Splice the merged target text content into the initial prompt template (prompt) to obtain the target prompt template, input the target prompt template into the large model to obtain the intent extraction result, and from the intent extraction result, mark the question text belonging to "question" or "please take an action" as "Q", and mark the reply text belonging to "answer" as "A".

[0230] Here, in order to guide the large model to extract the intent of the text content, a suitable prompt can also be designed. For example, in one example, the initial prompt can be specifically:

[0231] "prompt":"You are a professional financial analyst and will follow the following requirements from the dialogue information

[0232] Extract some key information from the recording. \n[Dialogue Information] contains questions between the account manager and the customer during the dual recording process.

[0233] Answer the dialogue text. Your task is to notice the question and answer information in the dialogue information, such as the account manager's question

[0234] questions, the customer's answers, etc.; can understand the meaning behind the conversation, including the intentions represented by the customer's answers

[0235] Etc. \n[Dialogue information]\n{content}\n[Sample answer]\n{template}\n\n[Answer restrictions]\n1.

[0236] The information is true and can be found in [Dialogue Information]. Fabrication is not allowed. \n2. Do not include in the output

[0237] '"' and other html tags

[0238] Furthermore, the text content of Example 1 is spliced ​​into the above initial prompt template to obtain the target prompt template, and after the target prompt template is input into the large model, the following extraction results can be obtained:

[0239] Example 1 text content:

[0240] 0:00:00~0:00:17.219000: In order to standardize the sales behavior of sales personnel and to better protect

[0241] To protect your legal rights and interests, we will record and videotape the key links of my sales process.

[0242] Do you agree to this? Agree.

[0243] 0:00:18.770000~0:00:39.100000:We need to remind you that this recording and video recording process is

[0244] It is very important for you to protect your rights in the future. Please read the specific contents of the document you signed carefully and answer the relevant questions truthfully.

[0245] If I make any promise to you that is inconsistent with the written document, I suggest you contact me in writing.

[0246] Please confirm in the form to better protect your legal rights and interests.

[0247] 0:00:39.410000~0:00:45.959000:Please ask the tester of road B gmh test to show

[0248] Qualification certificate.

[0249] 0:01:00.219000~0:01:15.700000:Hello Mr. ××, before confirming the purchase, in order to protect your legal rights, we need to collect your face and identity information for personal verification. Please hold the back of your ID card facing

[0250] The camera completes the collection of identity information.

[0251] Extraction results:

[0252] Q: Do you agree?

[0253] It is recommended that you confirm with me in writing in order to better protect your legal rights and interests.

[0254] Please ask the test guy who is doing the B road test gmh test to show his qualification certificate.

[0255] Please hold the back of your ID card in front of the camera to complete the identity information collection.

[0256] A: I agree.

[0257] (V) Time Backtracking Module

[0258] The time backtracking module first uses the extracted sentences as division nodes and segments the merged target text content, so as to use the text between the identifier "Q" and the previous identifier "A" as the inquiry stage, or use the text between the identifier "Q" and the previous identifier "Q" as the inquiry stage, and use the text identified as "A" alone as the willingness response stage, thus obtaining the QA division result.

[0259] Furthermore, the timestamps of each stage in the QA division result are traced back to obtain the timestamps of each stage. For example, if "Q" and "A" in the inquiry stage correspond to the same time period, the timestamps can be segmented according to the word length.

[0260] For example, the timestamp of the text corresponding to the inquiry stage ([Do you agree? Agree.]) is 0 to 9 seconds. At this time, the timestamp can be segmented according to the word length. For example, based on the rule of one second per word, we can get: "Q": [0--7 seconds Do you agree? ], "A": [7--9 seconds Agree. ]

[0261] The following are some examples:

[0262] (1) QA division process:

[0263] 0:00:00~0:00:17.219000: In order to standardize the sales behavior of sales personnel and to better protect

[0264] To protect your legal rights and interests, we will record and videotape the key links of my sales process.

[0265] (Q1) Do you agree? (A2) I agree.

[0266] 0:00:18.770000~0:00:39.100000:We need to remind you that this recording and video recording process is

[0267] It is very important for you to protect your rights in the future. Please read the specific contents of the document you signed carefully and answer the relevant questions truthfully.

[0268] Question: If I make any promise to you that is inconsistent with the written document (Q2), please contact me.

[0269] In order to better safeguard your legal rights and interests, we will confirm it in written form.

[0270] 0:00:39.410000~0:00:45.959000:(Q3) Please ask the tester gmh to test the B route

[0271] testShow qualification certificate.

[0272] 0:01:00.219000~0:01:15.700000:Hello Mr. ××, before confirming the purchase, in order to protect your legal rights, we need to collect your face and identity information for personal verification (Q4). Please hold your ID card and

[0273] Face the camera to complete identity information collection.

[0274] (2) QA division results:

[0275] Question 1: To regulate the sales behavior of sales personnel and to better protect your legal

[0276] According to relevant regulations, we will record the key links of my sales process by audio and video

[0277] Do you agree?

[0278] Reply text 2: Agree, agree.

[0279] Question 2: We need to remind you that this recording and videotaping process is very important for you to protect your rights in the future.

[0280] Please read the specific contents of the document you signed carefully and answer the relevant questions truthfully.

[0281] Any commitment that is inconsistent with the written document is recommended to be confirmed in writing by me for correction.

[0282] Better safeguard your legal rights.

[0283] Question text 3: Please ask the test guy gmh test of ×× test B road to show his qualification certificate.

[0284] Question text 4: Hello Mr. ××, before confirming the purchase, in order to protect your legal rights and interests, we need to collect your

[0285] Please hold the back of your ID card facing the camera to complete the identity information.

[0286] collection.

[0287] Furthermore, according to the timestamp, the text content of each part in each QA division result is traced back in time to obtain a dialogue text with a timestamp (that is, corresponding to the target dialogue text mentioned above).

[0288] Here, if the contents of multiple parts in the QA division result are all in the same time interval, the time interval can be divided according to the number of text words in each part of the text content in the same time interval, wherein the time interval is obtained based on the start time and end time contained in the timestamp.

[0289] For example, the text segment is "0:00:00-0:00:09: Do you agree? Agree.", and the QA segmentation results are: Q: "Do you agree?", A: "Agree.", the text contents of each part of the QA segmentation result are in the same time interval (ie, "0:00:00-0:00:09"). At this time, the time interval is segmented according to the proportion of the number of words in the text content of each part, that is, Q: "Do you agree?" is in the time interval of 0:00:00-0:00:07, and A: "Agree." is in the time interval of 0:00:07-0:00:09.

[0290] Continuing with the above extraction results as an example, after the merged text content is divided according to the extraction results and the following QA division results are obtained, the time of each part of the text content in the QA division result is traced back according to the timestamp in the speech recognition module, and the following time traceback results are obtained:

[0291] The time backtracking results are as follows:

[0292] 0:00:00~0:00:16.235714:In order to standardize the sales behavior of sales personnel and to better protect

[0293] To protect your legal rights and interests, we will record and videotape the key links of my sales process.

[0294] Do you agree to record this section?

[0295] 0:00:16.235714~0:00:17.219000:Agree. Agree.

[0296] 0:00:18.770000~0:00:39.100000:We need to remind you that this recording and video recording process is

[0297] It is very important for you to protect your rights in the future. Please read the specific contents of the document you signed carefully and answer the relevant questions truthfully.

[0298] If I make any promise to you that is inconsistent with the written document, I suggest you contact me in writing.

[0299] Please confirm in the form to better protect your legal rights and interests.

[0300] 0:00:39.410000~0:00:45.959000:Please ask the tester gmh test to show the B route

[0301] Qualification certificate.

[0302] 0:01:00.219000~0:01:15.700000:Hello Mr. ××, before confirming the purchase, in order to protect your legal rights, we need to collect your face and identity information for personal verification. Please hold the back of your ID card facing

[0303] The camera completes the collection of identity information.

[0304] At this point, a conversation text with a timestamp can be obtained, and the conversation text includes at least one of the following: a question-answer pair with a timestamp, and a question text with a timestamp.

[0305] (VI) Speech matching module

[0306] In the actual dual recording scenario, in order to ensure the accuracy and consistency of the link division, each link corresponds to a specific speech template. Through the speech template, the speech recognition results in the dual recording process can be semantically matched to map the dialogue text to the corresponding speech template, and then determine the specific link corresponding to the dialogue text.

[0307] In order to achieve this function, the speech matching module uses the large-scale pre-trained model 3.0 (Enhanced representation through knowledge integration 3.0, Ernie3.0) as the base model, and combines the in-batch negative sampling (In-batch Negatives) method to conduct supervised training on the public literature retrieval data set (literature_search_data) to obtain a semantic index model, and then build a vector index library based on the semantic index model. Among them, the Approximate Nearest Neighbors (ANN) engine in the hierarchical navigable small world library (hnswlib) can quickly retrieve the corresponding speech template. In this way, the advanced natural language processing technology and index retrieval engine are fully utilized to provide an efficient and reliable solution for the segmentation of links in the dual recording scenario.

[0308] Specifically, as shown in FIG7(c), the specific steps of the speech matching module include:

[0309] Step a: Sort the preset speech templates, and input the sorted speech templates into the vectorization service engine based on Ernie3.0 to vectorize the speech templates, and use the vectorized speech templates and ANN engine to build a vector index library.

[0310] Step b: Integrate the conversation texts obtained in the time backtracking module, and input the integrated conversation texts into the vectorization service engine based on Ernie3.0 to obtain the vectorized conversation texts. Use the vectorized conversation texts to search in the vector index library to obtain the top K recall results with the highest semantic similarity.

[0311] Step c: Based on the first K recall results, determine the link to which each text content in the dialogue text belongs, so as to obtain the link information to which each text content in the dialogue text belongs.

[0312] (VII) Link merging module

[0313] The specific steps of the link merging module include:

[0314] Step a: merging text contents with the same link information in the dialogue text in chronological order, and merging the timestamps corresponding to the text contents with the same link information in the dialogue text;

[0315] Step b: For the links involving actions (such as identity verification, information display, qualification certificate display, signature action recognition, etc.), the timestamp needs to be extended according to the time of the adjacent links.

[0316] Step c: Get the conversation text marked with link information.

[0317] Further, Figure 8 A complete flow chart of the disclosed solution is provided, such as Figure 8 As shown, the overall process of the disclosed solution includes:

[0318] Step S801: using the FFmpeg audio and video stream processing tool, extracting the dialogue audio from the video to be inspected and performing format conversion to obtain the dialogue audio after format conversion.

[0319] Step S802: performing voice activity detection on the conversation audio after the format conversion process to obtain multiple audio segments and timestamps corresponding to each audio segment, and performing audio transcription on each audio segment to obtain a text segment of each audio segment.

[0320] Here, after obtaining each audio segment, each audio segment may be further subjected to non-human voice filtering. For the specific processing process, reference may be made to the above example, which will not be described in detail here.

[0321] Step S803: De-noising the text segment of the audio segment to remove text content that does not belong to the dual recording scenario in the text segment, thereby obtaining a de-noised text segment.

[0322] Step S804: merge the denoised text segments to obtain merged text content, and extract intent from the merged text content to obtain at least one of the following: question text related to the question, and reply text related to the answer.

[0323] Step S805: Based on the extracted text, the merged text content is divided into QA parts, and based on the timestamp, the time matching is performed on each part of the QA divided text to obtain the dialogue text with the timestamp.

[0324] Step S806: Integrate the obtained dialogue texts to obtain integrated dialogue texts; use the dialogue template to perform dialogue matching on the integrated dialogue texts to determine the link information to which each text content in the integrated dialogue text belongs.

[0325] Step S807: Based on the link information and time information to which each text content in the integrated dialogue text belongs, the text content with the same link information is merged to obtain a dialogue text marked with the link information.

[0326] In summary, the disclosed solution has the following advantages, including:

[0327] First, the quality of the conversation content is higher. The disclosed solution performs non-human voice filtering on the multiple audio segments obtained after audio segmentation to ensure the quality of audio transcription; moreover, the disclosed solution also performs conversation denoising on the multiple text segments obtained after audio transcription, for example, using text classification to classify each text segment into three categories: questions, replies, and noise, and then removes the text segments belonging to noise, thereby obtaining a relatively pure conversation content.

[0328] Second, the distinction of question and answer content is higher. The disclosed solution uses a large model to extract information from the text content to obtain question text related to the question and reply text related to the answer, effectively avoiding the problem of inaccurate segmentation of question text and reply text, thereby improving the distinction of question and answer content.

[0329] Third, the quality inspection efficiency is improved. The disclosed solution can accurately extract the conversation text related to the business content and with a timestamp from the target conversation audio, and realize the audio segmentation of the target conversation audio, thereby significantly improving the quality inspection efficiency of the audio data.

[0330] The disclosed solution also provides a data processing device based on a large model, such as Fig. 9 As shown, including:

[0331] The audio segmentation unit 901 is used to perform voice activity detection on the target conversation audio to obtain multiple target audio segments with voice activities;

[0332] An audio conversion unit 902, configured to obtain target text content based on the multiple target audio segments;

[0333] The text processing unit 903 is used to use the large model to extract the intent of the target text content and extract at least one of the following: question text related to the question, and reply text related to the answer; based on the time information in the target conversation audio and the extracted text, obtain the target conversation text with time information associated with the target conversation audio, wherein the target conversation text includes at least one of the following: a question and answer pair with time information, and a question text with time information.

[0334] In a specific example of the disclosed solution, the audio conversion unit is specifically used to:

[0335] Based on the multiple target audio segments, obtaining a text segment of each target audio segment;

[0336] De-noising the text segments of each target audio segment to obtain the de-noised text segments of each target audio segment;

[0337] The text segments of each target audio segment after denoising are concatenated to obtain the target text content.

[0338] In a specific example of the disclosed solution, the audio conversion unit is specifically used to:

[0339] Using the first model, processing the text segment of the target audio segment to determine whether there are sentences in the text segment that are irrelevant to the target business scenario; wherein the target business scenario refers to the business scenario targeted by the target conversation audio;

[0340] When it is determined that there are sentences irrelevant to the target business scenario, the sentences irrelevant to the target business scenario are deleted from the text segment of the target audio segment to obtain the denoised text segment of the target audio segment.

[0341] In a specific example of the disclosed solution, the audio conversion unit is specifically used to:

[0342] Using the first model, classify the sentences in the text segment of the target audio segment to obtain a classification result for the text segment of the target audio segment, wherein the classification result indicates that the sentence in the text segment belongs to one of the following: noise, question, and answer; noise indicates irrelevant to the target business scenario; question indicates a question related to the target business scenario; and answer indicates a reply related to the target business scenario;

[0343] Based on the classification result of the text segment of the target audio segment, it is determined whether there is noise in the text segment.

[0344] In a specific example of the disclosed solution, the text processing unit is further used to:

[0345] Determine the question-answering session involved in the text segment of each target audio segment after denoising;

[0346] The contents of the question-and-answer sessions associated with the target dialogue text are merged to obtain a target dialogue text marked with the question-and-answer session.

[0347] In a specific example of the disclosed solution, the audio segmentation unit is specifically used to:

[0348] Performing voice activity detection on the target conversation audio to obtain multiple initial audio clips with voice activities;

[0349] An initial audio segment whose frequency falls within a target frequency range is determined from the multiple initial audio segments, so that the initial audio segment falling within the target frequency range is used as a target audio segment.

[0350] In a specific example of the disclosed solution, the audio segmentation unit is further used to:

[0351] Determine the frequency spectrum information corresponding to the time frame contained in each initial audio segment;

[0352] Based on the spectrum information corresponding to the time frames included in each initial audio segment, obtaining the frequency value corresponding to the time frames included in each initial audio segment;

[0353] The target frequency range is determined based on the frequency values ​​corresponding to the time frames included in each initial audio segment.

[0354] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0355] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0356] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0357] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0358] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0359] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0360] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a data processing method based on a large model. For example, in some embodiments, a data processing method based on a large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the data processing method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the large model-based data processing method in any other appropriate manner (eg, by means of firmware).

[0361] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0362] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0363] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0364] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0365] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0366] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0367] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0368] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method based on a large model, comprising: Performing voice activity detection on the target conversation audio to obtain multiple target audio segments with voice activities; Based on the multiple target audio segments, obtaining target text content; Using the large model, extract the intent of the target text content, and extract at least one of the following: question text related to the question, and reply text related to the answer; Based on the time information in the target conversation audio and the extracted text, a target conversation text with time information associated with the target conversation audio is obtained, wherein the target conversation text includes at least one of the following: a question-answer pair with time information, and a question text with time information.

2. The method according to claim 1, wherein: The step of obtaining target text content based on the multiple target audio segments includes: Based on the multiple target audio segments, obtaining a text segment of each target audio segment; De-noising the text segments of each target audio segment to obtain the de-noised text segments of each target audio segment; The text segments of each target audio segment after denoising are concatenated to obtain the target text content.

3. The method according to claim 2, wherein: The denoising process is performed on the text segments of each target audio segment to obtain the denoised text segments of each target audio segment, including: Using the first model, processing the text segment of the target audio segment to determine whether there are sentences in the text segment that are irrelevant to the target business scenario; wherein the target business scenario refers to the business scenario targeted by the target conversation audio; When it is determined that there are sentences irrelevant to the target business scenario, the sentences irrelevant to the target business scenario are deleted from the text segment of the target audio segment to obtain the denoised text segment of the target audio segment.

4. The method according to claim 3, wherein: The method of processing the text segment of the target audio segment by using the first model to determine whether there are sentences irrelevant to the target business scenario in the text segment includes: Using the first model, classify the sentences in the text segment of the target audio segment to obtain a classification result for the text segment of the target audio segment, wherein the classification result indicates that the sentence in the text segment belongs to one of the following: noise, question, and answer; noise indicates irrelevant to the target business scenario; question indicates a question related to the target business scenario; and answer indicates a reply related to the target business scenario; Based on the classification result of the text segment of the target audio segment, it is determined whether there is noise in the text segment.

5. The method according to claim 2 or 3, further comprising: Determine the question-answering session involved in the text segment of each target audio segment after denoising; The contents of the question-and-answer sessions associated with the target dialogue text are merged to obtain a target dialogue text marked with the question-and-answer session.

6. The method according to any one of claims 1 to 5, wherein: The performing voice activity detection on the target conversation audio to obtain a plurality of target audio segments with voice activities includes: Performing voice activity detection on the target conversation audio to obtain multiple initial audio clips with voice activities; An initial audio segment whose frequency falls within a target frequency range is determined from the multiple initial audio segments, so that the initial audio segment falling within the target frequency range is used as a target audio segment.

7. The method according to claim 6, further comprising: Determine the frequency spectrum information corresponding to the time frame contained in each initial audio segment; Based on the spectrum information corresponding to the time frames included in each initial audio segment, obtaining the frequency value corresponding to the time frames included in each initial audio segment; The target frequency range is determined based on the frequency values ​​corresponding to the time frames included in each initial audio segment.

8. A data processing device based on a large model, comprising: An audio segmentation unit, used to perform voice activity detection on the target conversation audio to obtain multiple target audio segments with voice activities; An audio conversion unit, configured to obtain target text content based on the multiple target audio segments; A text processing unit is used to use a large model to extract the intent of the target text content and extract at least one of the following: question text related to the question, and reply text related to the answer; based on the time information in the target conversation audio and the extracted text, a target conversation text with time information associated with the target conversation audio is obtained, wherein the target conversation text includes at least one of the following: a question and answer pair with time information, and a question text with time information.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

11. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images