A multi-speaker rotation offline translation direction control method

Abstract

The application relates to the technical field of speech translation processing, and discloses a multi-speaker rotation offline translation direction control method, which comprises the following steps: performing round division on a continuous interactive speech stream to obtain a sequence of speech segments; extracting speaker representations and language representations corresponding to each speech segment, and determining a response segment in combination with round position association, adjacent connection association and semantic response association; performing difference comparison on a current segment association record and a previous segment association record to generate attribution drift records and link mismatch records; disconnecting an original translation chain and rebuilding a target translation chain; performing segment translation and binding consistency verification, and outputting a target translation result according to a verification result; and the application can improve the accuracy and consistency of offline translation output in a multi-person rotation scene.

Description

Technical Field

[0001] This application relates to the field of speech translation processing technology, and more specifically, to an offline translation direction control method for multi-speaker rotation. Background Technology

[0002] Existing portable offline translation terminals typically build a processing chain around continuous voice input, encompassing speech recognition, language identification, translation direction determination, and translated text output. Existing technologies include solutions that segment multi-person audio based on speaker and language, and then perform text conversion and translation processing on the segmented speech segments; solutions that route translation results to corresponding output channels based on speaker identification and language direction recognition results; and solutions that switch the transcription and translation links during multilingual conferences when language categories change. Therefore, existing technologies already possess basic processing capabilities for multi-person speech segmentation, multilingual recognition, translation direction switching, and translation output.

[0003] However, in scenarios where multiple people take turns speaking, such as counter communication, business meetings, tour guides leading groups, and family trips, the speaker affiliation, language affiliation, translation direction, and conversational continuity in continuous interactive voice communication change continuously with each speaking turn. Although the aforementioned existing technologies disclose processing methods for segmenting audio by speaker, segmenting audio by language, determining language direction based on speaker identity, and switching translation chains when switching languages, these existing technologies mainly make local judgments about which speaker or language the current voice segment belongs to. They lack a joint judgment on the continuity between the current speech segment and the previous speech segment, and they also lack a process for further determining which translation chain the current speech segment should be assigned to based on the continuity.

[0004] Specifically, once a preceding speech segment has formed a corresponding translation direction and entered the translation processing state, if a subsequent response, inserted, or supplementary speech occurs, the terminal may still use the speaker affiliation, language affiliation, or translation direction corresponding to the previous speech segment and continue to execute the original processing path for the current speech segment; or it may directly output the translation based solely on the speaker identification result or language identification result of the current speech segment without further determining the corresponding successor object. In this way, the current speech segment may continue to be attached to the original translation chain, or it may be incorrectly merged into other session succession relationships, thereby causing a mismatch between the affiliation relationship, translation direction relationship, and translation output object of the current speech segment. This type of mismatch is not just a simple language identification deviation, but a processing chain break caused by the lack of consistent constraints between succession relationship judgment, translation chain attachment relationship, and output object binding relationship. This problem corresponds to the processing chain in the claims of this application, which includes response segment determination, affiliation drift and link mismatch identification, unattaching and rebuilding the target translation chain, and binding consistency verification control output.

[0005] For example, in an offline translation scenario where multiple speakers take turns speaking, the first speaker asks a question in the first language, the second speaker answers in the second language, and the third speaker then inserts supplementary information in the third language. If the terminal does not distinguish the connection between the third speaker and the preceding speech segment, the third speech segment may still be attached to the original translation chain of the second speech segment, or be misjudged as a continuation of the response to the first speech segment. In this case, the terminal may continue to output the translation in the wrong direction, or continue to broadcast intermediate translation results that should not be retained, thus causing the translation output object and translation output direction of the current speech segment to lose their correspondence.

[0006] Based on this, existing technologies still need to solve the following technical problems: how to effectively determine the connection between the current speech segment and the previous speech segment in a multi-person rotation scenario, and control the translation output of the current speech segment when the change in the connection relationship causes a change in the attribution relationship of the current speech segment or the connection relationship of the translation chain. Summary of the Invention

[0007] In order to overcome the above-mentioned defects of the prior art and to achieve the above objectives, this application provides the following technical solution: This application discloses an offline translation direction control method for multi-speaker rotation, including: Step S1: Obtain the continuous interactive speech stream, and perform round-by-round segmentation according to the pause position, the start and end position of the speech, and the position of the voiceprint change to obtain the speech segment sequence; Step S2: Obtain the speech segment sequence, extract the speaker representation and language representation for each speech segment to obtain the segment representation result set, and combine the round position association, adjacent connection association and semantic response association to determine the response segment corresponding to each speech segment, and obtain the segment association record sequence; Step S3: Obtain the fragment association record sequence, perform a difference comparison between the current fragment association record and the previous fragment association record, and generate the attribution drift record and the link mismatch record; Step S4: Obtain the home drift record, link mismatch record and response fragment, unattach the current speech fragment from the original translation chain, and reconstruct the target translation chain according to the speaker-language-translation direction association relationship corresponding to the response fragment to obtain the translation control record; Step S5: Obtain the translation control record and the current speech fragment, execute the segment translation corresponding to the target translation chain, and perform a binding consistency check based on the current speech fragment, response fragment, target translation chain, and target output object. Output the target translation result or cancel the intermediate translation result according to the check result.

[0008] Compared with related technologies, this application has the following advantages: This application does not merely perform isolated speaker identification, language identification, and translation output processing on the current speech segment in continuous interactive speech. Instead, it constructs a closed-loop processing method around the changing succession relationship between the current speech segment and the previous speech segment in a multi-person rotation scenario. This method involves determining the response segment, identifying attribution drift and link mismatch, unattaching and reconstructing the target translation chain, and verifying the consistency of the binding to control the output. This addresses the problem in existing technologies where the current speech segment tends to follow the original translation chain, leading to mismatches between the translation direction relationship and the output object relationship. By controlling the translation to enter the output process only when the attribution relationship, translation direction relationship, and output object relationship are consistent, the accuracy, stability, and consistency of the translation output in multi-person rotation offline translation scenarios are improved.

[0009] First, by combining round position association, adjacent connection association and semantic response association to determine the response segment for each speech segment, the processing method of performing local judgment based only on the current speech segment in the existing technology can be transformed into a processing method of determining the object to which the current speech segment belongs based on the succession relationship of the preceding and following speech segments. This makes the conversation succession relationship between the current speech segment and the preceding speech segment clear, reducing the subsequent translation chain connection deviation caused by inaccurate judgment of the response object in scenarios where multiple people take turns speaking, interrupt, or supplement speech.

[0010] Second, by performing a difference comparison between the current segment association record and the previous segment association record, affiliation drift record and link mismatch record are generated. This can further transform changes in speaker affiliation, language affiliation, and response affiliation into identifiable results such as transmission path offset, response object misconnection, or direction transmission mismatch. This improves the problem in the existing technology that only identifies changes in speaker or language but has difficulty identifying whether the translation transmission relationship still continues along the original processing path, and reduces the risk that the current speech segment will be incorrectly continued to be attached to the original translation chain.

[0011] Third, by unattaching the current speech segment from the original translation chain when a change in attribution or link mismatch is detected, and reconstructing the target translation chain based on the speaker-language-translation direction relationship corresponding to the response segment, the translation direction of the current speech segment no longer mechanically follows the existing records in the previous translation chain, but re-establishes the corresponding translation relationship based on the current recipient. This improves the targeting of translation direction switching in multi-person rotation scenarios and reduces the problem of inconsistency between the translated output direction and the actual conversation recipient direction.

[0012] Fourth, by performing segment translation corresponding to the target translation chain, and further performing segment attribution consistency checks, translation direction consistency checks, and output object consistency checks based on the current speech segment, response segment, target translation chain, and target output object, the key binding relationships before translation output can be constrained again. This prevents target translation results or intermediate translation results that do not conform to the attribution relationship, translation direction relationship, or output object relationship from entering the output process, thereby suppressing the broadcasting and retention of erroneous translations and improving the reliability of the translation output results.

[0013] Fifth, since this application organizes response relationship identification, link mismatch identification, translation chain reconstruction, and output verification into a continuous processing chain, it not only improves the processing interruption points of the existing technology in multi-person rotation scenarios, such as insufficient judgment of the succession relationship, error-prone connection of the translation chain, and lack of consistent constraints on the binding of output objects, but also helps to maintain the continuity and controllability of offline translation processing results in multi-person interaction scenarios such as counter communication, business negotiations, tour guides leading groups, and family travel. Attached Figure Description

[0014] Figure 1 A flowchart illustrating an offline translation direction control method for multi-speaker rotation provided in this application; Figure 2 A schematic diagram of the original translation chain unattaching method provided in this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] Please see Figure 1 As shown, this embodiment provides an offline translation direction control method for multi-speaker rotation, including the following steps: Step S1: Obtain the continuous interactive speech stream, and perform round-by-round segmentation according to the pause position, the start and end position of the speech, and the position of the voiceprint change to obtain the speech segment sequence.

[0017] In specific implementation, the method for obtaining the speech segment sequence includes: acquiring the continuous interactive speech stream collected and cached by the offline translation terminal during continuous interaction, and performing temporal unfolding on the continuous interactive speech stream according to a preset sampling period to obtain a speech sample segment sequence arranged in chronological order; wherein, each speech sample segment contains at least the corresponding time position and speech energy value, and the speech energy value is obtained by summing the squares of the amplitudes of each sampling point in the current speech sample segment and dividing by the number of sampling points contained in the current speech sample segment, which is used for subsequent identification of pause positions, speech start positions and speech end positions.

[0018] Secondly, based on whether the speech energy value corresponding to each speech sampling segment in the speech sampling segment sequence is continuously lower than the speech activity discrimination condition, the speech activity interruption position in the continuous interactive speech stream is identified; specifically: when the speech energy value corresponding to multiple consecutive speech sampling segments is lower than the speech activity discrimination condition, the corresponding interval is recorded as a pause interval, and the start time position and end time position corresponding to each pause interval are recorded as pause positions; the idea behind setting the speech activity discrimination condition is: to distinguish between speech intervals that carry speaking content and non-speech intervals that only contain environmental noise.

[0019] The speech start position is determined based on the time position in the speech sampling segment sequence where the speech energy value changes from below the speech activity discrimination condition to above the speech activity discrimination condition, and the speech end position is determined based on the time position where the speech energy value changes from above the speech activity discrimination condition to below the speech activity discrimination condition. The continuous speech interval between each speech start position and the corresponding speech end position is recorded as a candidate speech interval. Among them, the pause position is used to determine the time boundary between adjacent candidate speech intervals, and the speech start position and speech end position are used to determine the interval boundary of a single candidate speech interval.

[0020] The process involves acquiring candidate speech intervals, extracting voiceprint feature sequences to characterize the speaker's voice identity from each interval, and comparing the differences between adjacent voiceprint features within the same candidate speech interval in chronological order. Specifically, the candidate speech intervals are first divided into multiple candidate feature segments of equal length. Then, voiceprint feature vectors are extracted from each candidate feature segment. Next, voiceprint feature vectors that are temporally adjacent are compared item by item according to the same feature dimension. For each corresponding dimension, the absolute value of the difference between the feature value of the next candidate feature segment and the feature value of the previous candidate feature segment in the corresponding dimension is calculated to obtain the dimensional difference of each corresponding dimension. The dimensional differences of each corresponding dimension are then averaged to obtain the voiceprint difference value. When the voiceprint difference value reaches the preset segmentation condition, the corresponding time position is recorded as the voiceprint change position. The preset segmentation condition is based on distinguishing between voice identity changes of different speakers within the candidate speech interval and voiceprint fluctuations caused by changes in speech rate, tone, and short pauses of the same speaker.

[0021] By combining pause positions, speech start positions, speech end positions, and voiceprint change positions, round-by-round segmentation is performed on each candidate speech interval. Specifically, the speech start position is used as the segmentation starting point of the candidate speech interval, and the speech end position is used as the segmentation ending point. When a voiceprint change position appears within a candidate speech interval, the voiceprint change position is used as the segmentation position, thereby segmenting the continuous interactive speech stream into multiple speech segments arranged in chronological order and corresponding to a single speech round.

[0022] Each speech segment is assigned a segment number, start time position, and end time position according to its chronological order. The corresponding segment number, start time position, end time position, and source continuous interactive speech stream position for each speech segment are recorded to obtain a speech segment sequence. Through the above processing, the continuous interactive speech stream is converted into a speech segment sequence with round boundaries, aiming to provide a unified input for subsequent extraction of speaker representation and language representation for each speech segment.

[0023] To facilitate understanding of the round segmentation process in step S1, the following explanation uses a set of exemplary data. In one specific embodiment, the total duration of the continuous interactive speech stream is 18 seconds, which is expanded according to the same sampling period to obtain 180 speech sampling segments. Among them, the first second to the third second corresponds to the first candidate speech interval, the fourth second to the sixth second corresponds to the second candidate speech interval, the seventh second to the eighth second corresponds to the pause interval, the ninth second to the thirteenth second corresponds to the third candidate speech interval, and the thirteenth second to the eighteenth second corresponds to the fourth candidate speech interval. Within the third candidate speech interval, multiple candidate feature segments are extracted, and the differences between temporally adjacent voiceprint feature vectors are compared to obtain the tenth candidate speech segment. The voiceprint difference value at the 10.2-second position is significantly higher than the voiceprint difference results at other adjacent positions within the same interval. Therefore, the 10.2-second position is recorded as the voiceprint change position. Subsequently, based on the starting positions of the speech at the 1st, 4th, 9th, and 14th seconds, the ending positions of the speech at the 3rd, 6th, 13th, and 18th seconds, and the voiceprint change position at the 10.2-second position, the continuous interactive speech stream is segmented in rounds to obtain five speech segments with segment numbers P1, P2, P3, P4, and P5. Among them, segment P3 corresponds to the speech interval from the 9th to the 10.2-second position, and segment P4 corresponds to the speech interval from the 10.2-second to the 13th second position. The starting and ending time positions corresponding to segments P1 to P5 are then written into the recording positions to generate a speech segment sequence, which is used as the input for step S2.

[0024] Step S2: Obtain the speech segment sequence, extract the speaker representation and language representation for each speech segment to obtain the segment representation result set; and combine the round position association, adjacent connection association and semantic response association to determine the response segment corresponding to each speech segment to obtain the segment association record sequence.

[0025] In specific implementation, the method for extracting speaker representation and language representation for each speech segment to obtain a set of segment representation results includes: obtaining the speech segment sequence obtained in the aforementioned steps, and sequentially retrieving the speech data, start time position, and end time position corresponding to each speech segment according to the segment number to form a set of segment acquisition results; wherein, the set of segment acquisition results serves as the input object for the current step and is used to perform speaker representation extraction and language representation extraction on each speech segment respectively.

[0026] Preprocessing is performed on the speech data corresponding to each speech segment in the segment acquisition result set. The preprocessing includes silence interval removal, background noise suppression, and speech amplitude normalization to obtain purified speech segments. Among them, silence interval removal is used to remove silent intervals that do not carry speech content within the speech segment, background noise suppression is used to reduce the influence of environmental noise, and speech amplitude normalization is used to unify the speech amplitude representation between different speech segments.

[0027] The process involves acquiring purified speech segments, extracting acoustic features to characterize speaker voice differences, and combining these features in a unified order to form a segment speech feature vector. A speaker representation is then generated based on this feature vector. The speaker representation refers to the feature representation of the current speech segment that reflects the speaker's voice identity characteristics, and is used for subsequent comparisons of speaker attribution relationships between different speech segments. The process involves acquiring purified speech fragments, extracting speech and language features to represent language category differences, and merging these features according to a unified language recording standard to generate a language representation. Specifically, when the language representation uses language category labels, the first language in the ranking is determined based on the speech and language features corresponding to the purified speech fragment, and this first language is written into the language category label position corresponding to the current speech fragment. When the language representation uses candidate language ranking results, multiple candidate languages ​​and their corresponding ranking positions are formed based on the speech and language features corresponding to the purified speech fragment, and these multiple candidate languages ​​and their corresponding ranking positions are written into the language ranking record position corresponding to the current speech fragment. Here, the language representation refers to the feature representation result corresponding to the current speech fragment and used to reflect the language category affiliation, which is used for subsequent comparison of language change relationships between different speech fragments.

[0028] The segment number, start time position, end time position, speaker representation, and language representation of each speech segment are recorded to obtain a set of segment representation results. Through the above processing, each speech segment in the speech segment sequence is converted from the original speech data into a set of segment representation results that simultaneously carries speaker representation and language representation. The purpose is to provide a unified input for subsequent determination of the response segment corresponding to each speech segment by combining round position association, adjacent connection association, and semantic response association.

[0029] To facilitate understanding of the formation process of speaker representation and language representation, in a specific embodiment, preprocessing is performed sequentially on segments P1 to P5 obtained in step S1. Segment P1 has a start time of 1.0 second and an end time of 3.0 seconds, resulting in purified speech segment A1 after preprocessing. Segment P2 has a start time of 4.0 seconds and an end time of 6.0 seconds, resulting in purified speech segment A2 after preprocessing. Segment P3 has a start time of 9.0 seconds and an end time of 10.2 seconds, resulting in purified speech segment A3 after preprocessing. Acoustic features are extracted from purified speech segment A1 to generate speaker representation V1, and the language representation is determined to be Chinese based on the speech language features. Acoustic features are extracted from purified speech segment A2 to generate speaker representation V2, and the language representation is determined to be English based on the speech language features. Acoustic features are extracted from purified speech segment A3 to generate speaker representation V3, and the language representation is determined to be French based on the speech language features. Then, the segment number, start time position, end time position, speaker representation, and language representation corresponding to segment P1, segment P2, and segment P3 are recorded to obtain three segment representation records in the segment representation result set, and the segment representation result set is used as the input for subsequent determination of response segments.

[0030] In practice, methods for determining the response segments corresponding to each speech segment by combining round position association, adjacent connection association, and semantic response association, and obtaining the segment association record sequence, include: Obtain the fragment representation result set obtained from the aforementioned steps, and establish the sequential arrangement relationship between each speech fragment according to the fragment number corresponding to each speech fragment in the fragment representation result set to obtain the fragment order table; wherein, the round position association refers to the sequential position relationship between the current speech fragment and each preceding speech fragment in the speech fragment sequence. The round position association is determined according to the order of each fragment number in the fragment order table and is used to indicate the speech round position of the current speech fragment.

[0031] Based on the start and end times of each speech segment, the segment interval and sequence distance between the current speech segment and each preceding speech segment are calculated, and the segment interval and sequence distance are collectively recorded as the adjacent connection association. Among them, the segment interval refers to the time difference between the start time of the current speech segment and the end time of the candidate preceding speech segment, and the sequence distance refers to the sequence number difference between the segment number of the current speech segment and the segment number of the candidate preceding speech segment. The adjacent connection association is used to represent the association result between the current speech segment and the preceding speech segments in terms of time connection and round proximity.

[0032] The speech data corresponding to each speech segment in the segment representation result set is obtained, and speech recognition processing is performed on each speech segment to obtain the segment text content. Then, question markers, response word markers, topic word markers, pronoun markers, action word markers, and state word markers are extracted from the segment text content corresponding to the current speech segment and the segment text content corresponding to each preceding speech segment.

[0033] Based on the question-and-answer correspondence, referential succession, and semantic continuation relationships between the text content corresponding to the current speaking segment and the text content corresponding to each preceding speaking segment, a semantic response association between the current speaking segment and each preceding speaking segment is generated. Specifically: when the text content corresponding to a preceding speaking segment contains a question marker, and the text content corresponding to the current speaking segment contains an answer marker, or the topic marker corresponding to the current speaking segment is consistent with the question topic corresponding to the preceding speaking segment, it is recorded that there is a question-and-answer correspondence between the current speaking segment and the preceding speaking segment; when the text content corresponding to the current speaking segment contains a pronoun marker, and a certain When the text content of a preceding speech segment contains an object name, location name, matter name, or action content corresponding to a pronoun marker, it is considered that the current speech segment and the preceding speech segment have a referential succession relationship. When the subject word marker, action word marker, and status word marker corresponding to the current speech segment and the subject word marker, action word marker, and status word marker corresponding to a preceding speech segment continuously expand around the same matter, it is considered that the current speech segment and the preceding speech segment have a semantic continuity relationship. Among them, semantic response association refers to the association result in which the current speech segment responds to, supplements, confirms, or continues to explain a preceding speech segment in terms of semantic content.

[0034] For the current speech segment, firstly, speech segments with later segment numbers than the current speech segment are excluded based on round position association. Then, candidate preceding speech segments with segment intervals within a preset time range are selected based on adjacent connection association. Finally, the selected candidate preceding speech segments are ranked according to semantic response association, and the candidate preceding speech segment ranked first in the association ranking is determined as the response segment corresponding to the current speech segment. Among them, the association ranking is determined according to the association results formed by question-and-answer correspondence, referential succession, and semantic continuation. When candidate preceding speech segments correspond to the same association relationship, they are sorted in ascending order of segment interval. When the segment intervals are the same, they are sorted in ascending order of sequence distance. The idea behind setting the preset time range is to ensure that the response segment and the current speech segment maintain a continuous conversational succession relationship and to exclude preceding speech segments with excessively long intervals that are not suitable as direct responses.

[0035] When there is no semantic response association among the candidate preceding speech segments after filtering, the current speech segment is recorded as a non-response segment and written to the record position corresponding to the current speech segment; where, a non-response segment refers to a record result that does not inherit any preceding speech segment, which is used to indicate that the current speech segment is processed as an independent speech round.

[0036] The segment number, start time position, end time position, speaker representation, language representation, round position association, adjacent connection association, semantic response association, and response segment corresponding to each speech segment are recorded to obtain a segment association record sequence arranged in order of segment number. Through the above processing, the set of segment representation results is converted into a segment association record sequence containing response relationship information, which aims to provide a basis for subsequent comparison of differences between the current segment association record and the previous segment association record, as well as for generating attribution drift records and link mismatch records.

[0037] To facilitate understanding of the process of determining response fragments, in one specific embodiment, response fragment filtering is performed on fragment P4; wherein, the fragment text content of fragment P4 is: Do I need to check in?, the corresponding topic is marked as "check-in", and the response word is marked as empty; the fragment text content of fragment P1 is: I would like to ask about the hotel check-in time, the corresponding question topic is "check-in"; the fragment text content of fragment P2 is: check-in starts at two pm, the corresponding topic is also marked as "check-in"; the fragment text content of fragment P3 is: deux The term "personnes" corresponds to the number of people in the topic. Based on round position association, fragments P1, P2, and P3 are all preceding statements of fragment P4. Based on adjacency association, the interval between fragments P4 and P3 is 0.3 seconds and the order distance is 1; the interval between fragments P4 and P2 is 1.2 seconds and the order distance is 2; the interval between fragments P4 and P1 is 3.5 seconds and the order distance is 3. Based on semantic response association, it is determined that fragments P4 and P2 have a semantic continuation relationship, and fragments P4 and P1 have a question-and-answer correspondence relationship, but fragments P4 and P3 do not have a semantic response association. After sorting the candidate preceding statements with semantic response associations, fragment P2 is ranked first. Therefore, fragment P2 is determined as the response fragment corresponding to fragment P4, and the fragment number, round position association, adjacency association, semantic response association, and response fragment P2 corresponding to fragment P4 are written into the fragment association record sequence. This example illustrates that the response fragment is not determined solely by the principle of closest time, but rather by a combination of round position association, adjacency association, and semantic response association.

[0038] Step S3: Obtain the fragment association record sequence, perform a difference comparison between the current fragment association record and the previous fragment association record, and generate the attribution drift record and the link mismatch record.

[0039] In specific implementation, the method for generating the attribution drift record includes: obtaining the fragment association record sequence, and determining the current fragment association record and the previous fragment association record in sequence according to the fragment number corresponding to each fragment association record in the fragment association record sequence; wherein, the current fragment association record refers to the fragment association record whose fragment number is located at the current processing position, and the previous fragment association record refers to the fragment association record whose fragment number is located before the current processing position and participates in the current difference comparison.

[0040] Speaker representation, language representation, and response fragment are obtained from the current fragment association record and the previous fragment association record, respectively. The speaker representation is the feature representation result generated in the aforementioned steps to reflect the voice identity characteristics of the speaker corresponding to the speech fragment. The language representation is the feature representation result generated in the aforementioned steps to reflect the language category to which the speech fragment belongs. The response fragment is the previous speech fragment that has a response relationship with the current speech fragment, as determined in the aforementioned steps.

[0041] The speaker representation in the current segment's associated record is compared item by item with the speaker representation in the previous segment's associated record according to the same feature dimensions. The difference between each corresponding dimension is calculated, and the differences of each dimension are summarized into a representation difference value. Then, the representation difference value is compared with the preset speaker difference discrimination condition. When the representation difference value indicates that the voice identity characteristics corresponding to the current speaking segment and the voice identity characteristics corresponding to the previous speaking segment do not belong to the same speaker's voice distribution, the speaker difference result is output as "speaker changed". When the representation difference value indicates that the voice identity characteristics corresponding to the current speaking segment and the voice identity characteristics corresponding to the previous speaking segment belong to the same speaker's voice distribution, the speaker difference result is output as "speaker unchanged". The idea behind setting the preset speaker difference discrimination condition is to distinguish between the same speaker's voice representation range and different speaker's voice representation range.

[0042] The representation difference value is obtained by comparing the speaker representation in the current segment association record with the speaker representation in the previous segment association record according to the same feature dimensions. For each corresponding dimension, the absolute value of the difference between the feature value of the current segment association record in the corresponding dimension and the feature value of the previous segment association record in the corresponding dimension is calculated to obtain the dimension difference value of each corresponding dimension. Then, the dimension difference values ​​of each corresponding dimension are averaged to obtain the representation difference value. Specifically, the dimension difference values ​​of each corresponding dimension are summed and the sum is divided by the number of dimensions involved in the comparison to obtain the representation difference value.

[0043] The system retrieves the language representation from the current segment's associated record and the language representation from the preceding segment's associated record, and performs category difference judgment according to a unified language recording standard. Specifically: when the language representation is recorded in the form of language category labels, it compares whether the language category labels in the current segment's associated record are consistent with those in the preceding segment's associated record; when the language representation is recorded in the form of candidate language ranking results, it compares whether the first language in the current segment's associated record is consistent with the first language in the preceding segment's associated record; when the comparison result shows that the language representation in the current segment's associated record and the language representation in the preceding segment's associated record belong to the same language category, the output language difference result is "language unchanged"; when the comparison result shows that the language representation in the current segment's associated record and the language representation in the preceding segment's associated record belong to different language categories, the output language difference result is "language changed".

[0044] Retrieve the response fragments from the current fragment association record and the response fragments from the preceding fragment association record, and perform object difference judgment; specifically: compare whether the fragment number corresponding to the response fragment in the current fragment association record is consistent with the fragment number corresponding to the response fragment in the preceding fragment association record; when the fragment numbers are consistent, output the response difference result as no change in response; when the fragment numbers are inconsistent, output the response difference result as a change in response; when the response fragment in the current fragment association record is a non-response fragment but the response fragment in the preceding fragment association record corresponds to a specific speech fragment, or when the response fragment in the current fragment association record corresponds to a specific speech fragment but the response fragment in the preceding fragment association record is a non-response fragment, also output the response difference result as a change in response; where a non-response fragment refers to the record result recorded in the aforementioned steps as not accepting any preceding speech fragments.

[0045] Based on the speaker difference results, language difference results, and response difference results, it is determined whether the attribution of the current segment-related record has changed relative to the previous segment-related record; specifically: when the speaker difference result is a speaker change, it is recorded as a speaker attribution change; when the language difference result is a language change, it is recorded as a language attribution change; when the response difference result is a response change, it is recorded as a response attribution change.

[0046] Changes in speaker affiliation, language affiliation, and response affiliation are recorded accordingly to generate affiliation drift markers. The affiliation drift markers are used to indicate the type of affiliation change that occurs in the current speech segment relative to the previous speech segment.

[0047] The fragment number associated with the current fragment, the fragment number associated with the previous fragment, the speaker difference result, the language difference result, the response difference result, and the attribution drift marker are recorded accordingly to generate an attribution drift record. Through the above processing, the attribution changes between the preceding and following speech fragments in the fragment association record sequence are converted into attribution drift records. The purpose is to provide input for subsequent identification of whether the translation succession relationship continues along the original processing path by combining the response fragment and the attribution drift marker.

[0048] To facilitate understanding of the generation process of the attribution drift record, in a specific embodiment, the current fragment association record is determined to be the fragment association record corresponding to fragment P4, and the preceding fragment association record is determined to be the fragment association record corresponding to fragment P3; wherein, the speaker representation corresponding to fragment P4 is V4, the language representation is Chinese, and the response fragment is P2; the speaker representation corresponding to fragment P3 is V3, the language representation is French, and the response fragment is P1; after comparing speaker representation V4 and speaker representation V3 item by item according to the same feature dimension, if the representation difference value is higher than the preset speaker difference discrimination condition, the corresponding speaker difference result is output as speaker change; compare fragment P4 The corresponding language representation is Chinese, and the language representation of segment P3 is French. The output language difference result is language change. The response segment P2 corresponding to segment P4 is compared with the response segment P1 corresponding to segment P3. The output response difference result is response change. Therefore, the speaker affiliation change, language affiliation change, and response affiliation change are written together into the affiliation drift flag. The segment number corresponding to segment P4, the segment number corresponding to segment P3, the speaker difference result, the language difference result, the response difference result, and the affiliation drift flag are recorded accordingly to generate the affiliation drift record corresponding to segment P4. The affiliation drift record is used as the input to the link mismatch record generation process.

[0049] In specific implementation, the detailed implementation method for generating link mismatch records includes: obtaining the aforementioned home drift record, and locating the current segment association record and the previous segment association record corresponding to the home drift record in the segment association record sequence based on the segment number corresponding to the current segment association record and the segment number corresponding to the previous segment association record in the home drift record; wherein, the translation succession relationship refers to the succession correspondence formed when the current speech segment is mapped to the previous speech segment based on the response segment, and the translation succession relationship is jointly represented by the response segment corresponding to the current speech segment, the speaker representation corresponding to the response segment, and the language representation corresponding to the response segment.

[0050] Retrieve the response fragment corresponding to the current speech fragment from the current fragment association record, and search for the target fragment association record corresponding to the response fragment in the fragment association record sequence; when the response fragment corresponding to the current speech fragment is a non-response fragment, record the current fragment association record itself as the response reference record; when the response fragment corresponding to the current speech fragment corresponds to a specific speech fragment, record the fragment association record corresponding to the response fragment as the response reference record.

[0051] The speaker representation, language representation, and response fragment corresponding to the response reference record are obtained from the response reference record, and the speaker representation, language representation, and response fragment in the response reference record are used as reference objects for the translation succession relationship of the current speech fragment.

[0052] The process involves performing a continuity consistency comparison between the speaker representation in the current segment's associated record and the speaker representation in the response reference record. Specifically, it compares the speaker representation in the current segment's associated record and the speaker representation in the response reference record item by item according to the same feature dimensions, calculates the difference between each corresponding dimension, and summarizes the differences of each dimension into a continuity representation difference value. Then, it compares the continuity representation difference value with the preset continuity speaker comparison conditions. When the comparison result shows that the voice identity characteristics corresponding to the current speaking segment still continue the voice identity characteristics corresponding to the previous segment's associated record and have not transitioned to the voice identity characteristics corresponding to the response reference record, the speaker continuity comparison result is output as a continuation of the previous continuity. When the comparison result shows that the voice identity characteristics corresponding to the current speaking segment and the voice identity characteristics corresponding to the response reference record form a correspondence, the speaker continuity comparison result is output as a response continuity correspondence. The idea behind setting the preset continuity speaker comparison conditions is to distinguish whether the voice identity of the current speaking segment corresponds to the previous segment's associated record or the response reference record.

[0053] The reception representation difference value is obtained by comparing the speaker representation in the current segment association record with the speaker representation in the response reference record according to the same feature dimensions. For each corresponding dimension, the absolute value of the difference between the feature value of the current segment association record in the corresponding dimension and the feature value of the response reference record in the corresponding dimension is calculated to obtain the dimension difference value of each corresponding dimension. Then, the dimension difference values ​​of each corresponding dimension are averaged to obtain the reception representation difference value. Specifically, the dimension difference values ​​of each corresponding dimension are summed and the sum is divided by the number of dimensions involved in the comparison to obtain the reception representation difference value.

[0054] The language representation in the current segment association record is compared with the language representation in the previous segment association record and the language representation in the response reference record. Specifically: when the language representation in the current segment association record is consistent with the language representation in the previous segment association record, but inconsistent with the language representation in the response reference record, the output language continuation comparison result is the continuation of the previous language; when there is a continuation correspondence between the language representation in the current segment association record and the language representation in the response reference record, the output language continuation comparison result is the correspondence of the response language. Here, the language continuation correspondence means that the language classification of the current speech segment and the language classification of the response reference record together constitute a set of preceding and following translation input relationships.

[0055] Perform a response continuation comparison between the response fragment in the current fragment association record, the response fragment in the previous fragment association record, and the response fragment in the response reference record. Specifically: when the fragment number corresponding to the response fragment in the current fragment association record is the same as the fragment number corresponding to the response fragment in the previous fragment association record, but is different from the fragment number corresponding to the response fragment in the response reference record, the output response continuation comparison result is a continuation of the previous response; when the response fragment in the current fragment association record and the response reference record form a current continuation relationship, the output response continuation comparison result is a corresponding response object.

[0056] Based on the attribution drift markers in the attribution drift record, a comprehensive judgment is made on the speaker acceptance comparison results, language acceptance comparison results, and response acceptance comparison results. Specifically: when there is at least one attribution change in the attribution drift markers, and there is still one output in the speaker acceptance comparison result, language acceptance comparison result, or response acceptance comparison result that is a continuation of the previous acceptance, it is recorded as acceptance path offset; when the response segment in the current segment association record is different from the response segment in the previous segment association record, and the response acceptance comparison result output is a continuation of the previous response, it is recorded as response object misconnection; when the language difference result is a language change, and the language acceptance comparison result output is a continuation of the previous language, it is recorded as directional acceptance mismatch.

[0057] The following steps are taken to generate a link mismatch record: the segment number associated with the current segment, the segment number associated with the preceding segment, the response segment, the response reference record, the speaker's acceptance comparison result, the language acceptance comparison result, the response acceptance comparison result, and the judgment results of acceptance path offset, response object misconnection, and direction acceptance mismatch. Through the above processing, the attribution drift record is further converted into the mismatch between the current speech segment and the preceding translation acceptance relationship, aiming to provide input for subsequent unattaching of the original translation chain, reconstructing the target translation chain based on the response segment, and controlling the output of intermediate translation results.

[0058] To facilitate understanding the generation process of the link mismatch record, in one specific embodiment, the home drift record corresponding to segment P4 is read, and the response reference record is determined to be the segment association record corresponding to segment P2 based on the response segment P2 corresponding to segment P4; the speaker representation V4 corresponding to segment P4 is compared with the speaker representation V2 corresponding to the response reference record, and the speaker succession comparison result is a response succession correspondence; the language representation Chinese corresponding to segment P4 is compared with the language representation French corresponding to the preceding segment association record segment P3 and the language representation English corresponding to the response reference record segment P2, and the language succession comparison result is a preceding language continuation; the response segment P2 corresponding to segment P4 is compared with the preceding segment association record segment... The response fragment P1 corresponding to P3 and the response fragment P1 corresponding to the response reference record fragment P2 are used to obtain the response connection comparison result corresponding to the response object. Combined with the fact that the language connection change has been recorded in the attribution drift mark and the language connection comparison result is still the continuation of the previous language, the current processing result corresponding to fragment P4 is recorded as the direction connection mismatch. The fragment number corresponding to fragment P4, the fragment number corresponding to fragment P3, the response fragment P2, the response reference record P2, the speaker connection comparison result, the language connection comparison result, the response connection comparison result, and the direction connection mismatch judgment result are recorded accordingly to generate a link mismatch record. The link mismatch record is used as the input for the unattachment judgment in step S4.

[0059] Step S4: Obtain the home drift record, link mismatch record, and response fragment. Unattach the current speech fragment from the original translation chain and reconstruct the target translation chain based on the speaker-language-translation direction association corresponding to the response fragment to obtain the translation control record.

[0060] In specific implementation, the method for obtaining the home drift record, link mismatch record, and response fragment, and unattaching the current speech fragment from the original translation chain includes: obtaining the home drift record, link mismatch record, and response fragment; locating the current speech fragment and the previous speech fragment in the fragment association record sequence based on the fragment number corresponding to the current fragment association record and the fragment number corresponding to the previous fragment association record recorded in the home drift record; then locating the fragment association record corresponding to the response fragment in the fragment association record sequence based on the response fragment recorded in the fragment association record corresponding to the current speech fragment; wherein, the current speech fragment refers to the speech fragment corresponding to the current processing position, the previous speech fragment refers to the speech fragment that participated in the home change comparison before the current speech fragment, and the response fragment refers to the previous speech fragment that has a response relationship with the current speech fragment determined in the aforementioned steps; when the response fragment is a non-response fragment, the record result corresponding to the non-response fragment is also read into the current step.

[0061] like Figure 2As shown, the speaker difference result, language difference result, response difference result, and home drift flag in the home drift record corresponding to the current speech segment are obtained. The speaker acceptance comparison result, language acceptance comparison result, response acceptance comparison result, acceptance path offset judgment result, response object misconnection judgment result, and direction acceptance mismatch judgment result in the link mismatch record corresponding to the current speech segment are also obtained. The unattachment judgment result corresponding to the current speech segment is formed. Among them, the unattachment judgment result refers to the judgment result used to indicate whether the current speech segment should continue to be processed along the original translation chain record.

[0062] Based on the unattachment determination result, it is determined whether the current speech segment needs to be unattached from the original translation chain. Specifically: when the attribution drift marker records at least one of the following: speaker attribution change, language attribution change, or response attribution change, and the link mismatch record records at least one of the following: acceptance path offset, response object misconnection, or direction acceptance mismatch, the current speech segment is recorded as an unattached segment; when the attribution drift marker does not record an attribution change, or the link mismatch record does not record an acceptance path offset, response object misconnection, or direction acceptance mismatch, the current speech segment is recorded as a maintained attached segment.

[0063] Obtain the original translation chain record corresponding to the current speech segment before entering this step; wherein, the original translation chain record refers to the translation relationship record that the current speech segment has already corresponded to in the previous translation processing, and the original translation chain record includes at least the segment number in the chain, the speaker representation in the chain, the language representation in the chain, the translation direction mark in the chain, the previous connecting segment number in the chain, and the current attached segment number in the chain; the translation direction mark refers to the translation direction record result represented by two language record items, the two language record items include the translation input language record item and the translation output language record item, which are used to indicate which language to be converted from during the translation processing.

[0064] When the current speech segment is recorded as a deattached segment, the record position corresponding to the current attached segment number and the current speech segment in the original translation chain record is retrieved based on the segment number corresponding to the current speech segment. The segment number, speaker representation, language representation, translation direction marker, and the previous connecting segment number are then removed from the record position to obtain the original translation chain record after deattachment. Here, removal means deleting the record item that corresponds one-to-one with the current speech segment in the original translation chain record so that the current speech segment no longer references the connecting relationship in the original translation chain record.

[0065] When the current speech segment is recorded as a held segment, the current attached segment number, translation direction marker, and previous attached segment number in the original translation chain record are kept unchanged, and the held result is written to the processing intermediate record position corresponding to the current speech segment.

[0066] The fragment number corresponding to the current speech fragment, the unattachment judgment result, the original translation chain record position before unattachment, and the original translation chain record or maintenance result after unattachment are recorded accordingly to obtain the original translation chain adjustment result. Through the above processing, the attribution drift record, link mismatch record, and response fragment are transformed into a clear judgment result on whether the current speech fragment should continue to be processed along the original translation chain. When there is an attribution change or connection mismatch in the record, the current speech fragment is unattached from the original translation chain. The aim is to provide input for the subsequent reconstruction of the target translation chain based on the speaker-language-translation direction association relationship corresponding to the response fragment.

[0067] To facilitate understanding of the unattachment process, in one specific embodiment, before segment P4 enters step S4, the corresponding original translation chain record already records the current attached segment number P4, the previous receiving segment number P3, and the translation direction marker as French to Chinese. After reading the home drift record corresponding to segment P4, it is determined that the language home change has been recorded in the home drift marker. After reading the link mismatch record corresponding to segment P4, it is determined that the direction receiving mismatch judgment result has been recorded. Therefore, segment P4 is recorded as the unattached segment, and the current attached segment number P4, the translation direction marker as French to Chinese, and the previous receiving segment number P3 are removed from the original translation chain record to obtain the unattached original translation chain record. Then, the segment number corresponding to segment P4, the unattachment judgment result, the position of the original translation chain record before unattachment, and the original translation chain record after unattachment are recorded accordingly to generate the original translation chain adjustment result, and the original translation chain adjustment result is used as the input for the target translation chain reconstruction process.

[0068] In specific implementation, the method for reconstructing the target translation chain based on the speaker-language-translation direction association corresponding to the response fragment and obtaining the translation control record includes: obtaining the original translation chain adjustment result, and obtaining the response fragment corresponding to the current speech fragment, the fragment association record corresponding to the response fragment, and the original translation chain record corresponding to the response fragment; wherein, the speaker-language-translation direction association refers to the correspondence between the speaker representation, language representation, and translation direction marker in the same record, and the translation direction marker is the combination result of the translation input language record item and the translation output language record item defined in the aforementioned steps, which is used to indicate the translation direction that the current speech fragment should execute relative to the response fragment.

[0069] The process involves obtaining the speaker representation and language representation corresponding to the response fragment, and obtaining the speaker representation and language representation corresponding to the current speech fragment. The translation direction marker for the current speech fragment is determined based on these representations. Specifically: when the language representation of the response fragment and the language representation of the current speech fragment are in different language categories, the language representation of the response fragment is written into the translation input language record, and the language representation of the current speech fragment is written into the translation output language record, thus obtaining the translation direction marker for the current speech fragment. When the language representation of the response fragment and the language representation of the current speech fragment are in the same language category, the speaker representation of the response fragment and the speaker representation of the current speech fragment are compared. If they correspond to the same speaker affiliation, the translation direction marker in the original translation chain record corresponding to the response fragment is obtained and written into the translation direction marker position for the current speech fragment. If they correspond to different speaker affiliations, the current speech fragment is recorded as a segment to be confirmed, and the direction marker to be confirmed is written into the translation direction marker position for the current speech fragment. The criteria for determining the same speaker affiliation follow the speaker representation difference calculation and speaker difference discrimination conditions described in the previous steps.

[0070] Based on the speaker representation, language representation, and translation direction marker of the current speech segment, a speaker-language-translation direction association record is generated for the current speech segment. Specifically, the segment number corresponding to the response segment is written to the association start point record position, the segment number corresponding to the current speech segment is written to the association end point record position, the speaker representation and language representation corresponding to the response segment are written to the association source record position, the speaker representation and language representation corresponding to the current speech segment are written to the association target record position, and the translation direction marker corresponding to the current speech segment is written to the association direction record position.

[0071] The target translation chain is reconstructed based on the speaker-language-translation direction association record corresponding to the current speech fragment. Specifically, when the original translation chain adjustment result indicates that the current speech fragment has been unattached from the original translation chain, and the response fragment corresponds to a specific speech fragment, the fragment number corresponding to the response fragment is used as the previous receiving fragment number in the target translation chain, and the fragment number corresponding to the current speech fragment is used as the currently attached fragment number in the target translation chain. The translation direction mark in the association direction record position is written into the translation direction mark position in the target translation chain, and the speaker representation and language representation in the association source record position are written into the receiving source record position in the target translation chain. The speech representation in the association target record position is then written into the translation direction mark position in the target translation chain. Human representation and language representation are written into the target record position in the target translation chain to obtain the reconstructed target translation chain record. When the original translation chain adjustment result indicates that the current speech segment remains attached, the current attached segment number, translation direction marker, and previous attached segment number in the original translation chain record are directly written into the target translation chain record. When the response segment is a non-response segment, the segment number corresponding to the current speech segment is used as the current attached segment number in the target translation chain, the non-response segment marker is used as the previous attached segment record result in the target translation chain, and the translation direction marker corresponding to the current speech segment is written into the translation direction marker position in the target translation chain.

[0072] Obtain the reconstructed target translation chain record and perform record item checks on it. Specifically, check whether the target translation chain record simultaneously contains the segment number corresponding to the current speech segment, the previous successor segment number or the non-responding segment marker, the speaker representation corresponding to the current speech segment, the language representation corresponding to the current speech segment, and the translation direction marker. When all of the above record items have been written, the target translation chain record is recorded as an executable translation relationship record. When the translation direction marker position is filled with a direction marker to be confirmed, or when the translation direction marker position is not filled with the translation input language record item and the translation output language record item, the target translation chain record is recorded as a translation relationship record to be confirmed.

[0073] The following steps are taken to obtain a translation control record: First, the segment number corresponding to the current speech segment, the segment number corresponding to the response segment or the marker for a non-response segment, the home drift marker in the home drift record, the result of the accepting path offset judgment in the link mismatch record, the result of the response object misconnection judgment and the result of the direction accepting mismatch judgment, the result of the original translation chain adjustment, the speaker-language-translation direction association record corresponding to the current speech segment, the target translation chain record, and the executable translation relationship record or the translation relationship record to be confirmed. Through the above processing, the speaker-language-translation direction association corresponding to the response segment is transformed into the target translation chain corresponding to the current speech segment, forming a translation control record. This aims to provide a unified input for subsequent segment translation and output control based on the translation control record.

[0074] To facilitate understanding of the target translation chain reconstruction process, in one specific embodiment, the response fragment P2 corresponding to fragment P4 is read, and the language representation of fragment P2 is determined to be English, while the language representation of fragment P4 is Chinese. Since the language representations of fragment P2 and fragment P4 belong to different language categories, the language category of fragment P4 (Chinese) is written into the translation input language record, and the language category of fragment P2 (English) is written into the translation output language record, resulting in the translation direction marker for fragment P4 being Chinese to English. Subsequently, the fragment number corresponding to fragment P2 is written into the association start point record position, the fragment number corresponding to fragment P4 is written into the association end point record position, the speaker representation and language representation of fragment P2 are written into the association source record position, the speaker representation and language representation of fragment P4 are written into the association target record position, and the translation direction marker (Chinese to English) is written into the association direction record position, generating the speaker-language-translation direction association record for fragment P4. Then, using the segment number corresponding to segment P2 as the previous successor segment number in the target translation chain, and the segment number corresponding to segment P4 as the current attached segment number in the target translation chain, the target translation chain record corresponding to segment P4 is reconstructed, and this target translation chain record is written into the translation control record.

[0075] Step S5: Obtain the translation control record and the current speech fragment, execute the segment translation corresponding to the target translation chain, and perform a binding consistency check based on the current speech fragment, response fragment, target translation chain, and target output object. Output the target translation result or cancel the intermediate translation result according to the check result. The target output object is generated based on the previous receiving segment number, the translation output language record item, and the segment number corresponding to the current speech fragment in the target translation chain. The generated result is written to the receiving object segment number, output language record item, output segment number, and output record position.

[0076] In specific implementation, the method for obtaining the translation control record and the current speech fragment, and executing the segment translation corresponding to the target translation chain includes: obtaining the translation control record, and locating the current speech fragment and its corresponding target translation chain in the speech fragment sequence and the fragment association record sequence based on the segment number corresponding to the current speech fragment, the target translation chain record, the executable translation relationship record, or the translation relationship record to be confirmed recorded in the translation control record; wherein, the current speech fragment refers to the speech fragment that enters the translation processing in this step, and the target translation chain refers to the translation relationship record reconstructed in the aforementioned step based on the speaker-language-translation direction association relationship corresponding to the response fragment, and the target translation chain includes at least the segment number corresponding to the current speech fragment, the previous successor segment number or the non-response segment marker, the speaker representation corresponding to the current speech fragment, the language representation corresponding to the current speech fragment, and the translation direction marker.

[0077] The system acquires the speech data corresponding to the current speech segment and performs speech recognition processing on the speech data to obtain the text content of the current speech segment. The speech recognition processing includes acoustic decoding, word sequence concatenation, and text normalization of the speech data corresponding to the current speech segment. The text content of the segment is used as the input text for subsequent segment translation.

[0078] Obtain the translation direction marker in the target translation chain, and retrieve the translation input language record and translation output language record from the translation direction marker; when the translation control record records an executable translation relationship record, write the fragment text content, translation input language record, and translation output language record into the formal translation processing record position; when the translation control record records a translation relationship record to be confirmed, write the fragment text content, the language representation corresponding to the current speech fragment, the language representation corresponding to the response fragment, and the direction marker to be confirmed into the candidate translation processing record position; where the formal translation processing record position refers to the record position used to store the translation input content with the determined translation direction, and the candidate translation processing record position refers to the record position used to store the translation input content with the candidate translation directions to be compared.

[0079] The process involves performing segment translation on the text content written to the formal translation processing record. Specifically, the source language of the text content is determined based on the translation input language record, and the target language is determined based on the translation output language record. Then, language conversion is performed on each text term in the text content in sequence to obtain the translated term sequence. Finally, the translated term sequence is rearranged according to the word order rules of the target language to obtain the target translation result. The target translation result refers to the translation result generated by the current speech segment according to the translation direction corresponding to the target translation chain.

[0080] The candidate segment translation is performed on the text content of the segment written to the candidate translation processing record position. Specifically, the language representation corresponding to the current speech segment is used as the first translation input language record item, and the language representation corresponding to the response segment is used as the first translation output language record item to generate a first candidate translation direction. Then, the language representation corresponding to the response segment is used as the second translation input language record item, and the language representation corresponding to the current speech segment is used as the second translation output language record item to generate a second candidate translation direction. Subsequently, the language conversion of the text content of the segment is performed according to the first candidate translation direction to obtain a first intermediate translation result. The language conversion of the text content of the segment is performed according to the second candidate translation direction to obtain a second intermediate translation result. The first intermediate translation result, the second intermediate translation result and the segment number corresponding to the current speech segment are recorded accordingly. Among them, the intermediate translation result refers to the translation result pre-generated according to the candidate translation direction when the translation direction is not yet fixed, which is used for subsequent binding consistency verification.

[0081] The segment number, segment text content, translation direction marker in the target translation chain, and target translation result or intermediate translation result corresponding to the current speech segment are recorded to obtain the segment translation result record. Through the above processing, the translation control record and the current speech segment are converted into segment translation result records corresponding to the target translation chain, aiming to provide direct input for subsequent binding consistency checks based on the current speech segment, response segment, target translation chain, and target output object.

[0082] To facilitate understanding of the formation process of the segment translation result record, in a specific embodiment, the translation control record corresponding to segment P5 is recorded as the translation relationship record to be confirmed; wherein, the language representation of segment P5 is English, and the language representation of the response segment P4 is Chinese. After performing speech recognition processing on the speech data corresponding to segment P5, the segment text content is obtained as: please wait here; subsequently, using the language representation of segment P5 (English) as the first translation input language record item, and using the language representation of the response segment P4 (Chinese) as the first translation output language record item, a first candidate translation direction (English to Chinese) is generated; then, using the language representation of the response segment P4 (Chinese) as the second translation input language record item, and using the language representation of segment P5 (English) as the second translation output language record item, a second candidate translation direction (Chinese to English) is generated. After performing language conversion based on the first candidate translation direction, the first intermediate translation result is obtained: Please wait here; after performing language conversion based on the second candidate translation direction, the second intermediate translation result is obtained: Please wait here; then the segment number, segment text content, first intermediate translation result and second intermediate translation result corresponding to segment P5 are recorded to obtain the segment translation result record, and the segment translation result record is used as the input for binding consistency verification.

[0083] In specific implementation, the method of performing binding consistency verification based on the current speech fragment, response fragment, target translation chain, and target output object, and outputting the target translation result or revoking the intermediate translation result according to the verification result includes: obtaining the fragment translation result record, and obtaining the current speech fragment, response fragment, target translation chain, and target output object; wherein, the response fragment refers to the preceding speech fragment determined in the aforementioned steps and used to reconstruct the target translation chain, the target output object refers to the output destination record result of the translation corresponding to the current speech fragment, and the target output object includes at least the receiving object fragment number, the output language record item, the output fragment number, and the output record position, and the output record position refers to the record position used to write the target translation result.

[0084] Based on the current speech fragment, response fragment, target translation chain, and target output object, perform binding consistency checks; specifically: first perform fragment attribution consistency checks, then perform translation direction consistency checks, and finally perform output object consistency checks.

[0085] Perform a segment attribution consistency check; specifically: obtain the segment number, speaker representation, and language representation corresponding to the current speaking segment; obtain the segment number, speaker representation, and language representation corresponding to the response segment; then obtain the previous receiving segment number, receiving source record position, and receiving target record position in the target translation chain; when the previous receiving segment number in the target translation chain is consistent with the segment number corresponding to the response segment, and the speaker representation in the receiving target record position is consistent with the speaker representation corresponding to the current speaking segment, and the language representation in the receiving target record position is consistent with the language representation corresponding to the current speaking segment, the output segment attribution consistency check result is "attribution consistent"; when at least one of the above three items is inconsistent, the output segment attribution consistency check result is "attribution inconsistent"; wherein, the judgment criterion for consistent speaker representation follows the receiving representation difference value calculation and receiving speaker comparison conditions in the aforementioned steps, and the judgment criterion for consistent language representation follows the language difference judgment criterion in the aforementioned steps.

[0086] Perform a translation direction consistency check; specifically: obtain the translation direction marker from the segment translation result record, and obtain the translation input language record and translation output language record from the target translation chain, and then obtain the language representation corresponding to the current speech segment and the language representation corresponding to the response segment; when the translation input language record is consistent with the language representation record corresponding to the response segment, and the translation output language record is consistent with the language representation record corresponding to the current speech segment, the output translation direction consistency check result is "direction consistent"; when the translation input language record is inconsistent with the language representation record corresponding to the response segment, or the translation output language record is inconsistent with the language representation record corresponding to the current speech segment, the output translation direction consistency check result is "direction inconsistent".

[0087] The output object consistency check is performed as follows: The sequence number of the successor object fragment, the output language record, the output fragment sequence number, and the output record position in the target output object are obtained. The sequence number of the previous successor fragment, the translated output language record, and the fragment sequence number corresponding to the current speech fragment in the target translation chain are also obtained. The output object consistency check result is "object consistent" when the sequence number of the successor object fragment in the target output object matches the sequence number of the previous successor fragment in the target translation chain, the output language record in the target output object matches the translated output language record in the target translation chain, and the output fragment sequence number in the target output object matches the fragment sequence number corresponding to the current speech fragment. The output object consistency check result is "object inconsistent" when at least one of the above three items is inconsistent.

[0088] The binding consistency check result is generated by combining the fragment attribution consistency check result, the translation direction consistency check result, and the output object consistency check result. Specifically, when the fragment attribution consistency check result is consistent, the translation direction consistency check result is consistent, and the output object consistency check result is consistent, the binding consistency check result is recorded as passing the check. When at least one of the fragment attribution consistency check result, the translation direction consistency check result, and the output object consistency check result is inconsistent, the binding consistency check result is recorded as failing the check.

[0089] The output processing is performed according to the binding consistency check result. Specifically, when the binding consistency check result is passed and the segment translation result record contains the target translation result, the target translation result is written to the output record position in the target output object, and the segment number, target translation result, and output time position corresponding to the current speech segment are recorded accordingly. When the binding consistency check result is passed and the segment translation result record contains the first intermediate translation result and the second intermediate translation result, the translation direction mark in the target translation chain is obtained, and the first candidate translation direction and the second candidate translation direction are compared with the translation direction mark. The intermediate translation result corresponding to the consistent candidate translation direction is recorded as the target translation result, and the target translation result is written to the output record position in the target output object, and the segment number, target translation result, and output time position corresponding to the current speech segment are recorded accordingly.

[0090] When the binding consistency check result fails, the intermediate translation result cancellation process is executed. Specifically, the target translation result, the first intermediate translation result, and the record position corresponding to the second intermediate translation result corresponding to the current speech segment are obtained from the segment translation result record. The translated content in each record position is deleted, and the segment number, cancellation reason, cancellation time position, and target translation chain record position corresponding to the current speech segment are recorded accordingly to obtain the cancellation record. Among them, deletion means that the translated content corresponding to the current speech segment will no longer enter the output record position in the target output object.

[0091] The process involves recording the segment number corresponding to the current speech segment, the segment number corresponding to the response segment or the marker for a non-response segment, the target translation chain record position, the target output object, the binding consistency check result, and the target translation result or reversal record to obtain the final translation processing record corresponding to the current speech segment. Through this process, binding consistency checks are performed based on the current speech segment, response segment, target translation chain, and target output object. The target translation result is output or the intermediate translation result is revoked according to the check result. This aims to ensure that the translation only enters the output process when the current speech segment, response segment, target translation chain, and target output object are consistent.

[0092] To facilitate understanding of the binding consistency check and result output process, in a specific embodiment, the fragment translation result record corresponding to fragment P5, the response fragment P4, the target translation chain, and the target output object are read. Specifically, the previous successor fragment number recorded in the target translation chain is P4, the translated output language record is Chinese, the fragment number corresponding to the current speaking fragment is P5, and the successor object fragment number recorded in the target output object is also P4, the output language record is Chinese, and the output fragment number is P5. After performing the fragment attribution consistency check, it is determined that the previous receiving fragment number in the target translation chain is consistent with the response fragment P4, and the language representation in the receiving target record position is consistent with the language representation corresponding to the current speaking fragment. Therefore, the output fragment attribution consistency check result is "attribution consistent". After performing the translation direction consistency check, it is determined that the Chinese translation output language record item in the first candidate translation direction is consistent with the Chinese translation output language record item in the target translation chain. Therefore, the output translation direction consistency check result is "direction consistent". After performing the output object consistency check, it is determined that the receiving object fragment number, output language record item, and output fragment number are all consistent with the corresponding content in the target translation chain. Therefore, the output object consistency check result is "object consistent". Combining the above check results, the binding consistency check result is "check passed", and the first intermediate translation result, "Please wait here", is recorded as the target translation result and written to the output record position in the target output object. Conversely, when the output language record item in the output object is recorded as English, the output object consistency check result is "object inconsistent", and the first intermediate translation result and the second intermediate translation result are deleted, generating a revocation record. This example illustrates that the translation is not output directly after generation, but only enters the output process when there is consistency between the current speech fragment, the response fragment, the target translation chain, and the target output object.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the aforementioned scope.

Claims

1. A method for controlling the direction of offline translation in multi-speaker rotation, characterized in that, include: Step S1: Obtain the continuous interactive speech stream, and perform round-by-round segmentation according to the pause position, the start and end position of the speech, and the position of the voiceprint change to obtain the speech segment sequence; Step S2: Obtain the speech segment sequence, extract the speaker representation and language representation for each speech segment to obtain the segment representation result set, and combine the round position association, adjacent connection association and semantic response association to determine the response segment corresponding to each speech segment, and obtain the segment association record sequence; Step S3: Obtain the fragment association record sequence, perform a difference comparison between the current fragment association record and the previous fragment association record, and generate the attribution drift record and the link mismatch record; Step S4: Obtain the home drift record, link mismatch record and response fragment, unattach the current speech fragment from the original translation chain, and reconstruct the target translation chain according to the speaker-language-translation direction association relationship corresponding to the response fragment to obtain the translation control record; Step S5: Obtain the translation control record and the current speech fragment, execute the segment translation corresponding to the target translation chain, and perform a binding consistency check based on the current speech fragment, response fragment, target translation chain, and target output object. Output the target translation result or cancel the intermediate translation result according to the check result.

2. The offline translation direction control method for multi-speaker rotation according to claim 1, characterized in that, The method for determining the response fragments corresponding to each speech fragment includes: establishing round position association, adjacent connection association, and semantic response association for each speech fragment, and using round position association, adjacent connection association, and semantic response association together as the basis for determining the response fragment; among them, the semantic response association is determined based on the question-and-answer correspondence, referential succession relationship, and semantic continuation relationship between the text content of the fragments.

3. The offline translation direction control method for multi-speaker rotation according to claim 1, characterized in that, The method for obtaining the fragment association record sequence includes: recording the fragment number, start time position, end time position, speaker representation, language representation, round position association, adjacent connection association, semantic response association, and response fragment corresponding to each speech fragment, so as to obtain the fragment association record sequence arranged in order of fragment number.

4. The offline translation direction control method for multi-speaker rotation according to claim 3, characterized in that, The method for generating a home drift record includes: determining the current segment association record and the previous segment association record in sequence according to the segment association record sequence; extracting the speaker representation, language representation and response fragment from the current segment association record and the previous segment association record respectively; obtaining the speaker difference result, language difference result and response difference result; and generating a home drift record containing a home drift marker based on the speaker difference result, language difference result and response difference result.

5. A method for offline translation direction control in multi-speaker rotation according to claim 1 or 3, characterized in that, The method for generating link mismatch records includes: determining the response reference record based on the response fragment corresponding to the current speech fragment, and performing speaker acceptance comparison, language acceptance comparison and response acceptance comparison respectively; then combining the acceptance drift mark in the acceptance drift record for comprehensive judgment; when there is an acceptance change but the previous acceptance continues, it is recorded as acceptance path offset, response object misconnection or direction acceptance mismatch.

6. The offline translation direction control method for multi-speaker rotation according to claim 1, characterized in that, The method for unattaching the current speech segment from the original translation chain includes: obtaining the home drift record and link mismatch record corresponding to the current speech segment, and extracting the speaker difference result, language difference result, response difference result, home drift mark, as well as the acceptance path offset judgment result, response object misconnection judgment result and direction acceptance mismatch judgment result. Based on the speaker difference result, language difference result, response difference result, home drift mark, as well as the acceptance path offset judgment result, response object misconnection judgment result and direction acceptance mismatch judgment result, a deattachment determination result is formed.

7. The offline translation direction control method for multi-speaker rotation according to claim 6, characterized in that, The method for determining whether the current speech segment needs to be unattached from the original translation chain based on the unattachment determination result includes: when the attribution drift marker records at least one of the following: speaker attribution change, language attribution change, or response attribution change, and the link mismatch record records at least one of the following: acceptance path offset, response object misconnection, or direction acceptance mismatch, the current speech segment is recorded as an unattached segment; otherwise, the current speech segment is recorded as a kept attached segment.

8. A method for offline translation direction control in multi-speaker rotation according to claim 1 or 7, characterized in that, The method for reconstructing the target translation chain includes: determining the translation direction marker based on the language representation of the response fragment and the language representation of the current speech fragment; when the two belong to different language categories, the language category of the current speech fragment is recorded as the translation input language, and the language category of the response fragment is recorded as the translation output language; when the two belong to the same language category, the translation direction marker or the direction marker to be confirmed is determined by combining the corresponding speaker representation.

9. The offline translation direction control method for multi-speaker rotation according to claim 8, characterized in that, The method for performing segment translation corresponding to the target translation chain includes: determining the language category corresponding to the current speech segment as the translation input language based on the translation direction marker recorded in the translation control record, and determining the language category corresponding to the response segment as the translation output language; then performing segment translation corresponding to the target translation chain on the segment text content corresponding to the current speech segment to generate the target translation result or intermediate translation result corresponding to the target output object.

10. The offline translation direction control method for multi-speaker rotation according to claim 9, characterized in that, The method for performing binding consistency checks and controlling output includes: performing segment attribution consistency checks, translation direction consistency checks, and output object consistency checks on the target translation result or intermediate translation result based on the current speech segment, response segment, target translation chain, and target output object; when the binding consistency check result is a pass, the target translation result is output; when the binding consistency check result is a fail, the corresponding target translation result or intermediate translation result is deleted, and a revocation record is generated.

Images