Language interaction text information calibration method based on humiture large language model Agent

Abstract

The invention discloses a language interaction text information calibration method based on a humiture large language model Agent, which relates to the field of speech recognition, solves the problem of poor calibration effect of the existing language interaction text information calibration method, and comprises the following steps: S1, acquiring real-time interaction audio and sampling to obtain a sampled audio clip, and storing the sampled audio clip in a database; the method comprises the following steps: S1, sampling an audio sample, carrying out audio analysis to obtain a speech loudness intrusive ratio and a fragment statement recognition coincidence ratio, and obtaining interactive audio collection data, S2, carrying out type division on real-time interactive audio by analyzing the interactive audio collection data, respectively carrying out audio statement segmentation to obtain real-time interactive audio segmentation data, and carrying out audio analysis on the real-time interactive audio segmentation data; and S3, performing vocabulary semantic verification on the text vocabularies according to the real-time interaction audio segmentation data, and performing vocabulary translation and outputting according to the verification result. The method can improve the pertinence and accuracy of the language interaction text information calibration method.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition and involves large language model technology, specifically a method for calibrating language interaction text information based on a temperature and humidity large language model agent. Background Technology

[0002] Existing methods for calibrating language interaction text information have the following drawbacks when performing information calibration:

[0003] 1. Existing language interaction text information calibration methods for audio interaction sentence segmentation in real-time interactive audio usually can only use fixed sentence segmentation algorithms for sentence segmentation. They do not classify interactive audio types based on the loudness intrusion analysis characteristics and sentence continuity characteristics of real-time interactive audio, and cannot match targeted audio segmentation models for different types of real-time interactive audio, resulting in a lack of targeting in the audio segmentation process.

[0004] 2. Existing language interaction text information calibration methods can only recognize audio text information based on an audio language segment. They do not divide the text words contained in the audio segment into words without alternatives and words to be selected. They do not perform direct word translation of words without alternatives to supplement the audio text context of words to be selected. They cannot perform semantic verification of words to be selected based on the supplemented audio sentences, thus resulting in inaccurate audio recognition results.

[0005] To address this, we propose a method for calibrating language interaction text information based on a large-scale language model of temperature and humidity for agents. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method for calibrating language interaction text information based on a large-scale language model agent with temperature and humidity. This invention aims to improve the accuracy and relevance of language interaction text information calibration methods.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for calibrating language interaction text information based on a large-scale temperature and humidity language model agent, comprising the following steps:

[0008] Step S1: Acquire real-time interactive audio, sample the real-time interactive audio to obtain sampled audio segments, and obtain the speech loudness intrusion ratio and segment speech recognition overlap by performing environmental noise intrusion analysis and sentence continuity analysis on the audio sample samples to obtain interactive audio acquisition data.

[0009] Step S2: By analyzing the speech loudness intrusion ratio and the overlap of segment sentence recognition, the real-time interactive audio is divided into the first type of interactive audio and the second type of interactive audio. Targeted audio sentence segmentation is then performed on the first type of interactive audio and the second type of interactive audio to obtain real-time interactive audio segmentation data.

[0010] Step S3: Based on the real-time interactive audio segmentation data, divide the text words contained in the audio segments of the sentences into words without alternatives and words to be selected. Perform direct word translation on words without alternatives, perform semantic verification on words to be selected, translate words according to the verification results, and output the translation results.

[0011] Furthermore, in step S1, the specific steps are as follows:

[0012] Step S11: Acquire terminal users who are in a language interaction state with the target terminal. Randomly select one target terminal user from the acquired terminal users, acquire the audio of the real-time interactive conversation between the target terminal user and the target terminal to obtain the initial interactive audio, perform temperature and humidity crosstalk analysis on the initial interactive audio in combination with the environmental temperature and humidity index, and perform audio index compensation on the real-time interactive audio based on the analysis results to obtain the real-time interactive audio;

[0013] Step S12: Perform audio index analysis on the real-time interactive audio, and obtain the noise intrusion duration ratio and the overlap of segment sentence recognition based on the analysis results.

[0014] Step S13: Define the speech loudness intrusion ratio and the overlap of segment sentence recognition as interactive audio acquisition data.

[0015] Furthermore, in step S12, the specific steps are as follows:

[0016] The real-time interactive audio is sampled to obtain sampled audio segments. The speech of the target terminal user in the sampled audio segments is segmented into user speech samples, and the environmental noise in the sampled audio segments is segmented into environmental audio samples.

[0017] The loudness curves of user voice samples and environmental audio samples in the sampled audio segments are obtained to obtain the first sample loudness curve and the second sample loudness curve;

[0018] Plot the first and second sampled loudness curves on the same coordinate system to obtain the audio loudness coordinate system. Set a loudness feature coordinate point in the audio loudness coordinate system, and perform coordinate analysis on the first and second sampled loudness curves to obtain the speech loudness intrusion ratio corresponding to the loudness feature coordinate point.

[0019] Furthermore, in step S12, the specific steps are as follows:

[0020] The loudness analysis line is used to traverse the sampling time interval, and the sampling time interval is filtered for noise intrusion time. The ratio of the duration of the noise intrusion time to the duration of the sampled audio segment is calculated to obtain the proportion of noise intrusion time.

[0021] In the audio loudness coordinate system, the first sampled loudness curve is divided into a speech continuous curve and a silence curve. The time range corresponding to each silence curve is obtained to obtain the silence curve duration, and an effective silence duration is set. If the silence curve duration is greater than or equal to the effective silence duration, the corresponding silence curve is divided into a speech interval curve. If the silence curve duration is less than the effective silence duration, the corresponding silence curve is re-divided into a speech continuous curve.

[0022] The sampled audio segment is divided into several sentence output segments. A sample sentence output segment is selected, and the duration of the speech continuous curve contained in the sample sentence output segment is statistically analyzed to obtain the duration of the continuous curve. The ratio of the duration of the continuous curve to the duration of the sample sentence output segment is calculated to obtain the sentence recognition overlap degree corresponding to the sample sentence output segment. The sentence recognition overlap degree corresponding to each sentence output segment is obtained, and the difference between the mean and standard deviation of multiple sentence recognition overlap degrees is calculated to obtain the sentence recognition overlap degree of the segment.

[0023] Furthermore, in step S2, the specific steps are as follows:

[0024] Set a preset range for the proportion of noise intrusion duration and a preset range for the degree of overlap in recognition. If the proportion of noise intrusion duration is within the preset range and the degree of overlap in recognition of the segment is within the preset range, then the silent detection segmentation model is used to classify the real-time interactive audio into the first type of interactive audio. If the proportion of noise intrusion duration is not within the preset range or the degree of overlap in recognition of the segment is not within the preset range, then the real-time interactive audio is classified into the second type of interactive audio, and the initial screening data of interactive audio is obtained.

[0025] The first type of interactive audio is segmented into sentence audio segments using a silence detection sentence segmentation model, and the second type of interactive audio is segmented into sentence audio segments using a deep learning sentence segmentation model, thus obtaining real-time interactive audio segmentation data.

[0026] Furthermore, in step S3, the specific steps are as follows:

[0027] Step S31: Obtain real-time interactive audio segmentation data, and obtain multiple sentence audio segments based on the initial screening data of interactive audio;

[0028] Step S32: Perform word-by-word audio recognition on each audio segment of the sentence and obtain the text words recognized by each word audio. If the text words corresponding to the word audio contain only one candidate word, then mark the corresponding audio word as having no candidate word. If the text words corresponding to the word audio contain multiple candidate words, then mark the corresponding audio word as a candidate word.

[0029] Step S33: Directly translate the audio of the word without candidate words into the corresponding audio translation word, filter the multiple candidate words corresponding to the word to be selected, and obtain the audio translation word corresponding to the word to be selected based on the filtering results.

[0030] Step S34: Use audio translation vocabulary to perform audio translation on the selected vocabulary to obtain multiple text audio sentences, and output the multiple text audio sentences obtained.

[0031] Furthermore, in step S33, the specific steps are as follows:

[0032] Step S331: Randomly select a sample candidate word from multiple candidate words, obtain the audio clause containing the sample candidate word, obtain the ratio of the number of no candidate words in the audio clause to the number of text words, obtain multiple no candidate word proportions, and set the no candidate word proportion with the largest value as the audio candidate completion degree.

[0033] Step S332: Obtain the audio candidate completion rate corresponding to each candidate word, set the candidate word corresponding to the maximum audio candidate completion rate as the first candidate word, and obtain the percentage of audio clauses with first candidate words that have no candidate words.

[0034] Step S333: If there is no maximum value among the multiple unselected words, set the audio clause corresponding to the maximum value as the first-order word audio clause;

[0035] Step S334: If the percentage of multiple no-candidate words obtained has the same maximum value, then perform word priority analysis on the audio clauses with the same maximum percentage of no-candidate words, and obtain the first-priority word audio clause based on the analysis results.

[0036] Step S335: Perform semantic matching on the first candidate words in the first candidate word audio clause, and obtain the audio translation words corresponding to the first candidate words based on the matching results;

[0037] Step S336: Translate the first candidate word using the audio translation word corresponding to the first candidate word, and set the translated first candidate word as no candidate word. Repeat the process of translating the first candidate word. Select the first candidate word again from the remaining candidate words and translate it until the audio translation word corresponding to each candidate word is obtained.

[0038] Furthermore, in step S334, the specific steps are as follows:

[0039] The audio clauses with the highest percentage of no candidate words are obtained. A sample candidate clause is randomly selected from the obtained audio clauses. The first candidate word in the sample candidate clause is obtained. The number of word intervals between each no candidate word and the nearest first candidate word is obtained. Multiple no candidate word intervals are obtained. The average of the multiple no candidate word intervals is calculated to obtain the positive average interval distance corresponding to the sample candidate clause.

[0040] The word interval between each candidate word and the nearest first candidate word is obtained, resulting in multiple word intervals. The average of these multiple word intervals is then calculated to obtain the inverse average interval distance corresponding to the candidate clause of the sample.

[0041] Label the forward average interval distance as A1 and the reverse average interval distance as A2. Calculate A1 / (A1+A2) to obtain the proportion of the forward interval distance corresponding to the candidate clauses in the sample.

[0042] Obtain the percentage of positive interval distance for each audio clause, and set the audio clause corresponding to the minimum percentage of positive interval distance as the first-order word audio clause.

[0043] Furthermore, in step S336, the specific steps are as follows:

[0044] Step S3361: Obtain the candidate words corresponding to the first candidate words, and randomly select one feature candidate word from the multiple candidate words obtained;

[0045] Step S3362: Replace the first candidate word in the first candidate word audio clause with the feature candidate word to obtain the first replaced audio clause. Perform positional fit analysis on the Chinese words in the first replaced audio clause and obtain the positional fit of the feature candidate word based on the analysis results.

[0046] Step S3363: Obtain the position fit degree corresponding to each candidate word, compare the values ​​of the multiple obtained position fit degrees, and if there is no maximum position fit degree in a tie, set the candidate word corresponding to the maximum position fit degree as the audio translation word corresponding to the first candidate word.

[0047] Step S3364: If there are ties for the maximum positional fit, replace the first-order candidate word in the audio clause of the first-order candidate word with the ties for candidate words to obtain multiple ties for audio clauses. Use the ties for audio clauses as templates to input into the large language model Agent, obtain the mask language model probability of the ties for audio clauses in the ties for audio clauses, and set the candidate word corresponding to the maximum mask language model probability as the audio translation word corresponding to the first-order candidate word.

[0048] Furthermore, in step S3362, the specific steps are as follows:

[0049] In the first replaced audio clause, Chinese words other than the first candidate words are extracted to obtain multiple clause-related words, and the obtained clause-related words are marked as G1 related words to Gc related words respectively.

[0050] Obtain the cosine similarity between the G1 associated words to the Gc associated words and the first candidate words, and obtain the cosine similarity between the G1 words and the Gc words.

[0051] If the G1 related word exists only once in the first replaced audio clause, then obtain the word interval between the G1 related word and the first candidate word to obtain the G1 word interval number. If the G1 related word exists multiple times in the first replaced audio clause, then obtain the word interval between the G1 related word and the first candidate word at each time, and set the word interval number with the smallest value as the G1 word interval number.

[0052] Obtain the number of word intervals corresponding to G2 related words to Gc related words, and get the number of G2 word intervals to Gc word intervals;

[0053] The positional fit of the candidate feature words is obtained by calculating the cosine similarity between G1 words and Gc words, as well as the number of intervals between G1 words and Gc words.

[0054] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0055] 1. This invention classifies interactive audio types based on the loudness intrusion analysis characteristics and sentence continuity characteristics of real-time interactive audio, and matches targeted audio segmentation models for different types of real-time interactive audio, which can improve the targeting of the audio segmentation process.

[0056] 2. This invention divides the text vocabulary contained in the audio segment of a sentence into no candidate words and candidate words. Direct word translation is performed on the no candidate words to supplement the audio text context of the candidate words. Based on the supplementation, the candidate words are semantically verified, thereby effectively improving the accuracy of audio recognition results. Attached Figure Description

[0057] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0058] Figure 1 This is an overall system block diagram of the present invention;

[0059] Figure 2 This is a schematic diagram of the audio loudness coordinate system of the present invention;

[0060] Figure 3 This is a schematic diagram of the speech continuity curve of the present invention. Detailed Implementation

[0061] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0062] Example 1

[0063] Please see Figure 1 This invention provides a technical solution: a method for calibrating language interaction text information based on a large-scale temperature and humidity language model agent, comprising the following steps:

[0064] Step S1: Acquire real-time interactive audio, sample audio samples from the real-time interactive audio to obtain sampled audio segments, obtain the speech loudness intrusion ratio by performing environmental noise intrusion analysis on the audio sample samples, obtain the segment speech recognition overlap by performing sentence continuity analysis on the audio sample samples, and obtain interactive audio acquisition data.

[0065] Step S1 further includes the following steps:

[0066] The system acquires terminal users who are in a language interaction state with the target terminal. Among the acquired terminal users, it randomly selects one target terminal user and acquires the audio of the real-time interactive conversation between the target terminal user and the target terminal to obtain the initial interactive audio.

[0067] Temperature and humidity crosstalk analysis was performed on the initial interactive audio based on environmental temperature and humidity indices, and audio index compensation was performed on the real-time interactive audio based on the analysis results to obtain the real-time interactive audio.

[0068] Specifically as follows:

[0069] The acquisition period corresponding to the initial interactive audio is marked as the interactive audio acquisition period, and the interactive audio acquisition period is divided into several audio acquisition time points. Then, a feature audio time point is randomly selected from the multiple acquired audio acquisition time points.

[0070] The audio generation environment corresponding to the real-time interactive audio is acquired to obtain the interactive audio generation environment. The ambient temperature corresponding to the characteristic audio time point of the interactive audio generation environment is numerically collected to obtain the characteristic ambient temperature value. The saturated vapor pressure of the water body corresponding to the characteristic ambient temperature value is acquired to obtain the characteristic saturated vapor pressure.

[0071] It should be noted here that:

[0072] In this application, the characteristic saturated vapor pressure changes with the characteristic ambient temperature. In specific implementations, there are saturated vapor pressure values ​​corresponding to the following temperature points.

[0073] Specifically as follows:

[0074] Temperature (°C) Saturated vapor pressure (kPa) 0 0.61129 10 1.2281 20 2.3388 30 4.2455 40 7.3753 50 12.3334 60 19.9320

[0075] The relative humidity of the environment at the time point corresponding to the characteristic audio generation is numerically collected to obtain the characteristic environmental humidity value.

[0076] The audio propagation speed at a specific moment is obtained by calculating the characteristic ambient temperature, characteristic saturated vapor pressure, and characteristic ambient humidity.

[0077] The audio propagation speed at a characteristic moment is calculated using the following formula:

[0078]

[0079] Where Vtz is the audio propagation speed at the characteristic time, Thw is the characteristic ambient temperature value, Stz is the characteristic ambient humidity value, Ybh is the characteristic saturated vapor pressure, and Pbz is the standard atmospheric pressure.

[0080] It should be noted here that:

[0081] In this application, the standard atmospheric pressure referred to herein is specifically 101325 Pascals.

[0082] It should be noted here that:

[0083] In this application, the empirical formula corresponding to the principle that the speed of sound in air is linearly related to temperature is: v = 331.3 + 0.606 × t, that is, for every 1 degree Celsius increase in temperature, the speed of sound increases by approximately 0.606 meters per second.

[0084] The frequencies of different audio components in the initial interactive audio at the characteristic audio time point are numerically collected, and the corresponding audio component frequencies are compared. The frequency of the audio component with the smallest value is set as the characteristic interactive frequency. The ratio of the audio propagation speed at the characteristic time point to the characteristic interactive frequency is calculated to obtain the audio limit wavelength corresponding to the characteristic audio time point.

[0085] Repeatedly obtain the audio limit wavelength corresponding to each audio acquisition time point for the characteristic audio time point, and set the audio wavelength compensation reference value. If the audio limit wavelength is greater than or equal to the audio wavelength compensation reference value, the audio acquisition time point is set as the temperature and humidity crosstalk time point. If the audio limit wavelength is less than the audio wavelength compensation reference value, the audio acquisition time point is set as the no crosstalk time point.

[0086] It should be noted here that:

[0087] The historical audio data for frequency compensation by the audio compensation algorithm is acquired, and the audio limit wavelength corresponding to the historical audio at different times is collected. The obtained audio limit wavelengths are compared numerically, and the audio limit wavelength with the smallest value is set as the audio wavelength compensation benchmark value.

[0088] If there are temperature and humidity crosstalk time points in the interactive audio acquisition cycle, then the audio compensation algorithm is used to compensate the audio indicators of the initial interactive audio. If there are no temperature and humidity crosstalk time points in the interactive audio acquisition cycle, then there is no need to use the audio compensation algorithm to compensate the audio indicators of the initial interactive audio, and the real-time interactive audio is obtained.

[0089] It should be noted here that:

[0090] The audio compensation algorithm involved here is specifically a frequency domain equalization algorithm;

[0091] In this application, the real-time interactive audio referred to herein includes environmental noise collected during the audio acquisition process;

[0092] In this application, the target terminal is equipped with a recording device and a language interaction text information calibration system.

[0093] Audio metrics analysis is performed on real-time interactive audio, and the proportion of noise intrusion duration and the overlap of segment sentence recognition are obtained based on the analysis results.

[0094] Specifically as follows:

[0095] The real-time interactive audio is sampled to obtain sampled audio segments. The speech of the target terminal user in the sampled audio segments is segmented into user speech samples, and the environmental noise in the sampled audio segments is segmented into environmental audio samples.

[0096] It should be noted here that:

[0097] In this application, the environmental audio referred to herein specifically refers to the audio content in the sampled audio segment other than the user's voice;

[0098] The audio loudness curve of the user's voice sample in the sampled audio segment is obtained to obtain the first sampled loudness curve, and the audio loudness curve of the environmental audio sample in the sampled audio segment is obtained to obtain the second sampled loudness curve.

[0099] Please see Figure 2 The first and second sampling loudness curves are plotted on the same coordinate system to obtain the audio loudness coordinate system. The time range corresponding to the sampled audio segments is marked on the x-axis as the sampling time interval.

[0100] In the audio loudness coordinate system, a loudness analysis line perpendicular to the x-axis is set. A loudness feature coordinate point is set in the x-axis region corresponding to the sampling time interval. When the loudness analysis line coincides with the loudness feature coordinate point, the ordinate of the intersection point of the loudness analysis line and the first sampling loudness curve is obtained to obtain the first sampling loudness feature value. The ordinate of the intersection point of the loudness analysis line and the second sampling loudness curve is obtained to obtain the second sampling loudness feature value. The difference between the first sampling loudness feature value and the second sampling loudness feature value is calculated, and the ratio of the obtained difference to the first sampling loudness feature value is calculated to obtain the speech loudness intrusion ratio corresponding to the loudness feature coordinate point.

[0101] The loudness analysis line is used to traverse the sampling time interval, and the speech loudness intrusion ratio corresponding to each coordinate point in the sampling time interval is obtained. A preset interval for the speech loudness intrusion ratio is set. If the speech loudness intrusion ratio is within the preset interval, the time period covered by the corresponding coordinate point in the sampling time interval is set as the non-noise intrusion time period. If the speech loudness intrusion ratio is not within the preset interval, the time period covered by the corresponding coordinate point in the sampling time interval is set as the noise intrusion time period. The ratio of the noise intrusion time period to the sampled audio segment is calculated to obtain the noise intrusion duration percentage.

[0102] It should be noted here that:

[0103] In this application, the preset range of the speech loudness intrusion ratio is 100%, that is, there is no loudness intrusion. The speech loudness intrusion ratio corresponding to the historical non-noise intrusion period is obtained, and the speech loudness intrusion ratio with the smallest value is set as the lower limit of the preset range of speech loudness intrusion ratio.

[0104] Please see Figure 3 In the audio loudness coordinate system, the part of the first sample loudness curve with a non-zero vertical coordinate is marked as a speech continuous curve, and the part of the first sample loudness curve with a zero vertical coordinate is marked as a silence curve. The time range corresponding to each silence curve is obtained to get the silence curve duration, and an effective silence duration is set. If the silence curve duration is greater than or equal to the effective silence duration, the corresponding silence curve is divided into a speech interval curve. If the silence curve duration is less than the effective silence duration, the corresponding silence curve is divided into a speech continuous curve.

[0105] It should be noted here that:

[0106] In this application, the effective silence duration referred to herein is specifically the duration of a normal human speech pause, which is specifically set to 2 seconds.

[0107] The sampled audio segment is segmented into several sentence output segments using the existing sentence segmentation model. A sample sentence output segment is selected, and the duration of the speech continuous curve contained in the sample sentence output segment is statistically analyzed to obtain the duration of the continuous curve. The ratio of the duration of the continuous curve to the duration of the sample sentence output segment is calculated to obtain the sentence recognition overlap degree corresponding to the sample sentence output segment.

[0108] It should be noted here that:

[0109] In this application, the sentence segmentation model involved is specifically a sentence segmentation model based on silence detection.

[0110] Repeat the process of obtaining the sentence recognition overlap of the sample sentence output fragments, obtain the sentence recognition overlap of each sentence output fragment, calculate the difference between the mean and standard deviation of multiple sentence recognition overlaps, and obtain the sentence recognition overlap of the fragments.

[0111] The speech loudness intrusion ratio and the overlap of segmented sentence recognition are defined as interactive audio acquisition data;

[0112] Step S2: By analyzing the speech loudness intrusion ratio and the overlap of segment sentence recognition, the real-time interactive audio is divided into the first type of interactive audio and the second type of interactive audio. Audio sentence segmentation is then performed on the first type of interactive audio and the second type of interactive audio respectively to obtain real-time interactive audio segmentation data.

[0113] Set a preset range for the proportion of noise intrusion duration and a preset range for the degree of overlap in recognition. If the proportion of noise intrusion duration is within the preset range and the degree of overlap in recognition of the segment is within the preset range, then the silent detection segmentation model is used to classify the real-time interactive audio into the first type of interactive audio. If the proportion of noise intrusion duration is not within the preset range or the degree of overlap in recognition of the segment is not within the preset range, then the real-time interactive audio is classified into the second type of interactive audio, and the initial screening data of interactive audio is obtained.

[0114] It should be noted here that:

[0115] In this application, the upper limit of the preset range for the noise intrusion duration ratio is 100%, that is, there is no noise intrusion. The historical interactive audio is obtained by the silent detection statement segmentation model, and the noise intrusion duration ratio corresponding to each historical interactive audio is obtained. The noise intrusion duration ratio with the smallest value is set as the lower limit of the preset range for the noise intrusion duration ratio.

[0116] The upper limit of the preset range for the noise intrusion duration ratio involved here is 100%, that is, the time period corresponding to the output sentence segment and the speech continuous curve completely overlaps. The historical interactive audio is obtained by the silence detection sentence segmentation model, and the noise intrusion duration ratio corresponding to each historical interactive audio is obtained. The noise intrusion duration ratio with the smallest value is preset as the lower limit of the preset range for the noise intrusion duration ratio.

[0117] The silence detection statement segmentation model is used to divide the first type of interactive audio into statement audio segments, and the deep learning statement segmentation model is used to divide the second type of interactive audio into statement audio segments, thus obtaining real-time interactive audio segmentation data.

[0118] Step S3: Based on the real-time interactive audio segmentation data, divide the text words contained in the audio segments of the sentences into words without alternatives and words to be selected. Perform direct word translation on words without alternatives, perform semantic verification on words to be selected, translate words according to the verification results, and output the translation results.

[0119] The specific steps in step S3 are as follows:

[0120] Acquire real-time interactive audio segmentation data, and obtain multiple sentence audio segments based on the initial screening data of interactive audio;

[0121] The automatic speech recognition algorithm is used to perform word-by-word audio recognition on each audio segment of the sentence, and the text words recognized for each word audio are obtained. If the text words corresponding to the word audio contain only one candidate word, the corresponding audio word is marked as having no candidate word. If the text words corresponding to the word audio contain only multiple candidate words, the corresponding audio word is marked as a candidate word.

[0122] It should be noted here that:

[0123] In this application, the term "multiple candidate words" specifically refers to two or more candidate words;

[0124] In this application, the vocabulary referred to herein includes single words such as "person", "walk", "eat", etc.

[0125] The audio of words without alternative words is directly translated into the corresponding audio translation words. Multiple alternative words corresponding to the words to be selected are filtered, and the audio translation words corresponding to the words to be selected are obtained based on the filtering results.

[0126] Specifically as follows:

[0127] Randomly select one sample word from the multiple candidate words obtained. Obtain the audio clause containing the sample word. Count the number of no-candidate words in each audio clause to obtain the first word statistical value. Count the number of text words in each audio clause to obtain the second word statistical value. Calculate the ratio of the first word statistical value to the second word statistical value to obtain the proportion of no-candidate words. Set the proportion of no-candidate words with the largest value as the audio candidate completion degree corresponding to the sample word.

[0128] Repeat the process of obtaining the audio candidate completion degree corresponding to the candidate words in the sample. Obtain the audio candidate completion degree corresponding to each candidate word. Set the candidate word corresponding to the maximum audio candidate completion degree as the first candidate word. Obtain the percentage of no candidate words for audio clauses with the first candidate word.

[0129] If the percentage of multiple unselectable words does not have a maximum value, the audio clause corresponding to the maximum value is set as the first-order word audio clause.

[0130] If multiple unselected words have the same maximum percentage, then perform word priority analysis on the audio clauses with the same maximum percentage of unselected words, and obtain the first-ranked word audio clause based on the analysis results.

[0131] Specifically as follows:

[0132] The audio clauses with the highest percentage of no candidate words are obtained. A sample candidate clause is randomly selected from the obtained audio clauses. The first candidate word in the sample candidate clause is obtained. The number of word intervals between each no candidate word and the nearest first candidate word is obtained. Multiple no candidate word intervals are obtained. The average of the multiple no candidate word intervals is calculated to obtain the positive average interval distance corresponding to the sample candidate clause.

[0133] It should be noted here that:

[0134] In this application, the unselectable words referred to herein are specifically words with definite semantics. The smaller the number of words between the first candidate word and the unselectable words, the more likely these two words are to have a direct semantic relationship in the text context, which is more conducive to word recognition of the first candidate word in the sample candidate clause.

[0135] The word interval between each candidate word and the nearest first candidate word is obtained, resulting in multiple word intervals. The average of these multiple word intervals is then calculated to obtain the inverse average interval distance corresponding to the candidate clause of the sample.

[0136] It should be noted here that:

[0137] In this application, the words to be selected are specifically words with uncertain semantics. The larger the number of word intervals between the first-order words to be selected and the words to be selected, the more likely these two words are to have a direct semantic relationship in the text context, which is more conducive to word recognition of the first-order words to be selected in the sample candidate clauses.

[0138] Label the forward average interval distance as A1 and the reverse average interval distance as A2. Calculate A1 / (A1+A2) to obtain the proportion of the forward interval distance corresponding to the candidate clauses in the sample.

[0139] It should be noted here that:

[0140] The calculation of A1 / (A1+A2) here can perform a comprehensive ratio analysis of the unselected words and the words to be selected. In an audio sentence, the smaller the proportion of the positive average interval distance, the more beneficial it is to identify the first candidate word in the sample candidate clause.

[0141] Repeat the process of obtaining the percentage of positive interval distance corresponding to the candidate clauses of the sample, obtain the percentage of positive interval distance corresponding to each audio clause, compare the numerical values ​​of the multiple positive interval distance percentages, and set the audio clause corresponding to the minimum positive interval distance percentage as the first word audio clause.

[0142] It should be noted here that:

[0143] In this application, if there is a situation where the maximum value of the ratio of the并列正向间隔距离 (the specific term needs to be further clarified according to the context) exists, the number of words in the parallel audio clauses is counted, and the audio clause with a larger number of words is set as the first-rank word audio clause, without considering the situation of equal number of words.

[0144] Semantic matching is performed on the first-rank candidate words in the first-rank word audio clause, and the corresponding audio translation words of the first-rank candidate words are obtained according to the matching results;

[0145] Specifically as follows:

[0146] The candidate words corresponding to the first-rank candidate words are obtained, and a characteristic candidate word is randomly selected from the obtained multiple candidate words;

[0147] The characteristic candidate word is used to replace the first-rank candidate word in the first-rank candidate word audio clause, obtaining the first replacement audio clause, and the position fitness analysis is performed on the characteristic candidate word and the Chinese words in the first replacement audio clause, and the corresponding position fitness of the characteristic candidate word is obtained according to the analysis results;

[0148] Specifically as follows:

[0149] In the first replacement audio clause, the Chinese words other than the first-rank candidate word are extracted, obtaining multiple clause-related words, and the obtained clause-related words are respectively marked as G1-related words to Gc-related words;

[0150] It should be noted here that:

[0151] In this application, G involved here is the symbol corresponding to the clause-related word, c involved here is the numerical value corresponding to the clause-related word, and c is an integer greater than 0;

[0152] In this application, the clause-related words involved here do not include stop words, and the stop words involved here include but are not limited to "的", "了" and "吧".

[0153] The cosine similarities between G1-related words to Gc-related words and the first-rank candidate word are obtained, obtaining G1 word cosine similarity to Gc word cosine similarity;

[0154] It should be noted that the term "并列正向间隔距离" in the original text needs to be further clarified according to the specific context for a more accurate translation.If the G1 related word exists only once in the first replaced audio clause, then obtain the word interval between the G1 related word and the first candidate word to obtain the G1 word interval number. If the G1 related word exists multiple times in the first replaced audio clause, then obtain the word interval between the G1 related word and the first candidate word at each time, and set the word interval number with the smallest value as the G1 word interval number.

[0155] Repeat the process of obtaining the number of word intervals for G1, and obtain the number of word intervals corresponding to G2 related words to Gc related words respectively, to get the number of word intervals for G2 to Gc.

[0156] The position fit of the candidate feature words is obtained by calculating the cosine similarity between G1 words and Gc words, as well as the number of intervals between G1 words and Gc words.

[0157] It should be noted here that:

[0158] Cosine similarity measures the degree of similarity between two vectors by calculating the cosine of the angle between them in vector space, with values ​​ranging from -1 to 1. In natural language processing, words are typically represented as vectors (such as word embedding vectors), which capture the semantic information of the words. When calculating the cosine similarity from G1 words to Gc words, we are actually evaluating how close these words are to the candidate feature words in semantic space.

[0159] The number of word intervals refers to the distance between the word candidate and the word in the target word set (G1 to Gc) in the text. The smaller the number of intervals, the more direct the semantic or grammatical relationship between these words in the text.

[0160] The positional fit of the candidate feature words is calculated using the following formula:

[0161]

[0162] Where Wsd is the position fit degree corresponding to the candidate feature word, Jgsi is the number of Gi word intervals, Yxsi is the cosine similarity of Gi words, and c is the number of words associated with the clause.

[0163] It should be noted here that:

[0164] In this application, the number of Gi word intervals involved here is any one of the number of G1 word intervals to the number of Gc word intervals, and the cosine similarity of Gi words involved here can be any one of the cosine similarity of G1 words to the cosine similarity of Gc words.

[0165] Repeat the process of obtaining the position fit of the candidate words for features. Obtain the position fit of each candidate word. Compare the values ​​of the multiple position fits. If there is no maximum position fit, set the candidate word corresponding to the maximum position fit as the audio translation word corresponding to the first candidate word.

[0166] If there are ties for the maximum positional fit, the first-order candidate word in the audio clause of the first-order candidate word is replaced by the ties for candidate words, resulting in multiple ties for audio clauses. The ties for audio clauses are then used as templates to input into the large language model Agent. The masked language model probabilities of the ties for audio clauses are obtained, and multiple masked language model probabilities are obtained and compared numerically. The candidate word corresponding to the maximum masked language model probability is set as the audio translation word corresponding to the first-order candidate word.

[0167] It should be noted here that:

[0168] In this application, the masked language model probability refers to the conditional probability of the model predicting the original words at the masked positions given the context of a masked (randomly masked or replaced) text sequence.

[0169] Translate the audio translation words corresponding to the first candidate words into the first candidate words, and set the first candidate words that have been translated to have no candidate words. Repeat the process of translating the first candidate words. Then, select the first candidate words again from the remaining candidate words and translate them until the audio translation words corresponding to each candidate word are obtained.

[0170] The audio translation vocabulary is used to translate the selected vocabulary into multiple text audio sentences, and then the multiple text audio sentences are output.

[0171] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A language interaction text information calibration method based on a temperature and humidity large language model agent, characterized in that, The method comprises the following steps: Step S1: obtaining real-time interactive audio, audio sample sampling is performed on the real-time interactive audio to obtain a sampled audio segment, and speech loudness interference ratio and segment sentence recognition coincidence degree are obtained by performing environmental noise interference analysis and sentence continuity analysis on the audio sample sampling to obtain interactive audio collection data; Step S2: analyzing the speech loudness interference ratio and the segment sentence recognition coincidence degree, dividing the real-time interactive audio into first type interactive audio and second type interactive audio, and performing targeted audio sentence segmentation to obtain real-time interactive audio segmentation data; Step S3: dividing text vocabulary contained in a sentence audio subsegment into no candidate vocabulary and candidate vocabulary according to the real-time interactive audio segmentation data, directly translating the no candidate vocabulary, performing vocabulary semantic verification on the candidate vocabulary, performing vocabulary translation according to the verification result, and outputting the translation result.

2. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 1, characterized in that, In step S1, the specific steps are as follows: Step S11: selecting a target terminal user from terminal users in a language interaction state with the target terminal, obtaining real-time interactive sessions of the target terminal user and the target terminal for audio acquisition to obtain initial interactive audio, performing temperature and humidity crosstalk analysis on the initial interactive audio in combination with environmental temperature and humidity indexes, and performing audio index compensation on the real-time interactive audio according to the analysis result to obtain the real-time interactive audio; Step S12: performing audio index analysis on the real-time interactive audio, and obtaining noise interference duration ratio and segment sentence recognition coincidence degree corresponding to the real-time interactive audio according to the analysis result; Step S13: defining the speech loudness interference ratio and the segment sentence recognition coincidence degree as interactive audio collection data.

3. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 2, characterized in that, In step S12, the specific steps are as follows: Audio sample sampling is performed on the real-time interactive audio to obtain a sampled audio segment, the speech of the target terminal user in the sampled audio segment is segmented into user speech sampling, and the environmental noise in the sampled audio segment is segmented into environmental audio sampling; The audio loudness curves of the user speech sampling and the environmental audio sampling in the sampled audio segment are obtained to obtain a first sampling loudness curve and a second sampling loudness curve; The first sampling loudness curve and the second sampling loudness curve are plotted in the same coordinate system to obtain an audio loudness coordinate system, a loudness feature coordinate point is set in the audio loudness coordinate system, and coordinate analysis is performed on the first sampling loudness curve and the second sampling loudness curve to obtain a speech loudness interference ratio corresponding to the loudness feature coordinate point.

4. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 3, characterized in that, In step S12, the specific steps are as follows: The sampling time interval is interval traversed using a loudness analysis straight line, the sampling time interval is noise interference time interval screened, the time length ratio of the noise interference time interval to the sampled audio segment is calculated to obtain a noise interference time length ratio; In the audio loudness coordinate system, the first sampling loudness curve is divided into a speech continuous curve and a mute curve, a time range corresponding to each mute curve segment is acquired, a mute curve duration is obtained, and an effective mute duration is set. If the mute curve duration is greater than or equal to the effective mute duration, the corresponding mute curve is divided into a speech interval curve. If the mute curve duration is less than the effective mute duration, the corresponding mute curve is re-divided into a speech continuous curve. The sampling audio segment is divided into a plurality of sentence output segments, and a sample sentence output segment is selected. The duration of the speech continuous curve contained in the sample sentence output segment is counted to obtain a continuous curve duration. The ratio of the continuous curve duration to the duration of the sample sentence output segment is calculated to obtain the sentence recognition coincidence degree corresponding to the sample sentence output segment. The sentence recognition coincidence degree corresponding to each sentence output segment is obtained. The difference between the average and the standard deviation of a plurality of sentence recognition coincidence degrees is calculated to obtain the segment sentence recognition coincidence degree.

5. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 1, characterized in that, In the step S2, the specific steps are as follows: The noise interference duration proportion preset interval and the recognition coincidence degree preset interval are set. If the noise interference duration proportion is in the noise interference duration proportion preset interval, and the segment sentence recognition coincidence degree is in the recognition coincidence degree preset interval, the real-time interactive audio is divided into a first type of interactive audio using the mute detection sentence segmentation model. If the noise interference duration proportion is not in the noise interference duration proportion preset interval or the segment sentence recognition coincidence degree is not in the recognition coincidence degree preset interval, the real-time interactive audio is divided into a second type of interactive audio, and the interactive audio preliminary screening data is obtained. The first type of interactive audio is segmented into sentence audio sub-segments using the mute detection sentence segmentation model, and the second type of interactive audio is segmented into sentence audio sub-segments using the deep learning sentence segmentation model, and the real-time interactive audio segmentation data is obtained.

6. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 1, characterized in that, In the step S3, the specific steps are as follows: Step S31: Obtain the real-time interactive audio segmentation data, and obtain a plurality of sentence audio sub-segments according to the interactive audio preliminary screening data; Step S32: Perform word-by-word audio recognition on each sentence audio sub-segment, and obtain the text word recognized by each word audio. If the text word corresponding to the word audio contains only one candidate word, the audio word corresponding to the text word is marked as a non-candidate word. If the text word corresponding to the word audio contains a plurality of candidate words, the audio word corresponding to the text word is marked as a candidate word; Step S33: The audio word corresponding to the non-candidate word is directly translated into the corresponding audio translation word. The plurality of candidate words corresponding to the candidate word are screened, and the audio translation word corresponding to the candidate word is obtained according to the screening result; Step S34: The audio translation word is used to perform audio translation on the candidate word, and a plurality of text audio sentences are obtained. The obtained plurality of text audio sentences are output.

7. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 6, characterized in that, In the step S33, the specific steps are as follows: Step S331: randomly select a sample candidate word from the plurality of candidate words, acquire the audio clause containing the sample candidate word, acquire the ratio of the number of non-candidate words to the number of text words in the audio clause, obtain a plurality of non-candidate word ratios, and set the non-candidate word ratio with the largest value as the audio candidate completion degree; Step S332: acquire the audio candidate completion degree corresponding to each candidate word, set the candidate word corresponding to the maximum audio candidate completion degree as the first-order candidate word, and acquire the non-candidate word ratio of the audio clause containing the first-order candidate word; Step S333: if there is no maximum value in the plurality of non-candidate word ratios, set the audio clause corresponding to the maximum value as the first-order word audio clause; Step S334: if there is a maximum value in the plurality of non-candidate word ratios, analyze the word priority of the audio clauses with the maximum non-candidate word ratio, and acquire the first-order word audio clause according to the analysis result; Step S335: perform semantic matching on the first-order candidate word in the first-order word audio clause, and acquire the audio translation word corresponding to the first-order candidate word according to the matching result; Step S336: translate the first-order candidate word with the audio translation word corresponding to the first-order candidate word, set the translated first-order candidate word as a non-candidate word, and repeat the translation process of the first-order candidate word, reselect the first-order candidate word from the remaining plurality of candidate words and translate it until the audio translation word corresponding to each candidate word is obtained.

8. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 7, characterized in that, In step S334, the specific steps are as follows: acquire the audio clauses with the maximum non-candidate word ratio, randomly select a sample candidate clause from the acquired audio clauses, acquire the first-order candidate word in the sample candidate clause, acquire the word interval number between each non-candidate word and the nearest first-order candidate word, obtain a plurality of non-candidate word interval numbers, and calculate the average of the obtained plurality of non-candidate word interval numbers to obtain the forward average interval distance corresponding to the sample candidate clause; acquire the word interval number between each candidate word and the nearest first-order candidate word, obtain a plurality of candidate word interval numbers, and calculate the average of the obtained plurality of candidate word interval numbers to obtain the reverse average interval distance corresponding to the sample candidate clause; label the forward average interval distance as A1 and the reverse average interval distance as A2, and calculate the forward interval distance ratio corresponding to the sample candidate clause; acquire the forward interval distance ratio corresponding to each audio clause, and set the audio clause corresponding to the minimum forward interval distance ratio as the first-order word audio clause.

9. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 7, characterized in that, In step S336, the specific steps are as follows: Step S3361: acquire the candidate word corresponding to the first-order candidate word, and randomly select a feature candidate word from the acquired plurality of candidate words; Step S3362: replacing the first-order candidate vocabulary in the first-order candidate vocabulary audio clause with the feature candidate vocabulary to obtain a first replacement audio clause, and performing position adaptation degree analysis on the feature candidate vocabulary and the Chinese vocabulary in the first replacement audio clause to obtain a position adaptation degree corresponding to the feature candidate vocabulary; Step S3363: obtaining the position adaptation degree corresponding to each candidate vocabulary, performing numerical comparison on the obtained multiple position adaptation degrees, and if there is no maximum position adaptation degree tie, setting the candidate vocabulary corresponding to the maximum position adaptation degree as the audio translation vocabulary corresponding to the first-order candidate vocabulary; Step S3364: if there is a maximum position adaptation degree tie, replacing the first-order candidate vocabulary in the first-order candidate vocabulary audio clause with the parallel candidate vocabulary to obtain multiple parallel audio clauses, inputting the parallel audio clauses into the large language model Agent as templates, obtaining the mask language model probability of the parallel audio clauses in the parallel audio clauses, and setting the candidate vocabulary corresponding to the maximum mask language model probability as the audio translation vocabulary corresponding to the first-order candidate vocabulary.

10. The language interaction text information calibration method based on the temperature and humidity large language model agent according to claim 9, characterized in that, In the step S3362, the specific steps are as follows: In the first replacement audio clause, extract the Chinese vocabulary except the first-order candidate vocabulary to obtain multiple clause-associated vocabularies, and mark the obtained clause-associated vocabularies as G1 associated vocabulary to Gc associated vocabulary respectively; Obtain the cosine similarity of G1 associated vocabulary to Gc associated vocabulary and the first-order candidate vocabulary to obtain G1 vocabulary cosine similarity to Gc vocabulary cosine similarity; If there is only one G1 associated vocabulary in the first replacement audio clause, obtain the vocabulary interval number of G1 associated vocabulary and the first-order candidate vocabulary to obtain G1 vocabulary interval number, if there are multiple G1 associated vocabularies in the first replacement audio clause, obtain the vocabulary interval number of each G1 associated vocabulary and the first-order candidate vocabulary, and set the minimum vocabulary interval number as G1 vocabulary interval number; Obtain the vocabulary interval number corresponding to G2 associated vocabulary to Gc associated vocabulary to obtain G2 vocabulary interval number to Gc vocabulary interval number; Obtain the vocabulary interval number corresponding to G2 associated vocabulary to Gc associated vocabulary to obtain G2 vocabulary interval number to Gc vocabulary interval number; Calculate G1 vocabulary cosine similarity to Gc vocabulary cosine similarity and G1 vocabulary interval number to Gc vocabulary interval number to obtain the position adaptation degree corresponding to the feature candidate vocabulary.

Images