Simultaneous interpretation translation method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a simultaneous interpretation translation method, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a to-be-translated audio; determining a text segmentation position of a to-be-translated text corresponding to the to-be-translated audio by using the target segmentation model; wherein the text segmentation position comprises a first text segmentation position and a second text segmentation position, the first text segmentation position is a position between a first to-be-translated sub-text and a second to-be-translated sub-text in the to-be-translated text, and the translation influence degree of the second to-be-translated sub-text on the first to-be-translated sub-text is lower than a first influence degree threshold value; the second text segmentation position is the position of a target punctuation in the to-be-translated text; and according to the text segmentation position, performing segmentation translation on the audio to be translated. Through the method, the timeliness of simultaneous interpretation can be improved, and the time delay of simultaneous interpretation can be reduced.

Description

[0001] This application claims priority to Chinese patent application No. 2024114864667, filed on October 23, 2024, entitled “Simultaneous Translation Method, Electronic Device and Computer-readable Storage Medium”, which is incorporated herein by reference in its entirety. Technical Field

[0002] The present application relates to the field of translation technology, and in particular to a simultaneous translation method, an electronic device and a computer-readable storage medium. Background Art

[0003] In the field of simultaneous translation, the cascade translation method and the end-to-end translation method are mostly used.

[0004] In the cascade translation method, the decision on when to throw out the translation result can only be made through a combination of strategies, punctuation marks and translation content of the recognition result, and the stability is too poor. In the end-to-end translation method, most of the results are currently thrown out based on the punctuation marks of the translation result, that is, the translation result will be thrown out only when a comma, period, question mark, or exclamation mark is detected. This method inevitably brings translation delays, and it is difficult to achieve real-time simultaneous translation. In addition, as the audio continues to flow in and the audio length increases, the reasoning speed will also be affected, which will cause the simultaneous translation speed to become slower and slower. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a simultaneous translation method, an electronic device and a computer-readable storage medium, which can improve the timeliness of simultaneous translation and reduce the delay of simultaneous translation.

[0006] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a simultaneous translation method, the method comprising: obtaining an audio to be translated; using a target segmentation model to determine the text segmentation position of the text to be translated corresponding to the audio to be translated; wherein the text segmentation position comprises a first text segmentation position and a second text segmentation position, the first text segmentation position is the position between a first sub-text to be translated and a second sub-text to be translated in the text to be translated, the translation influence of the second sub-text to be translated on the first sub-text to be translated is lower than a first influence threshold, and the second text segmentation position is the position of the target punctuation in the text to be translated; according to the text segmentation position, the audio to be translated is segmented and translated.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: to provide an electronic device, the electronic device includes a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the above simultaneous translation method.

[0008] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement the above simultaneous translation method.

[0009] According to the above technical scheme, after determining the text segmentation position of the text to be translated corresponding to the audio to be translated, the sub-text to be translated or the audio segment to be translated corresponding to the text segmentation position is subsequently thrown out for translation. Therefore, the simultaneous translation result can be thrown out more quickly, the timeliness of the simultaneous translation is improved, and the delay of the simultaneous translation is reduced. In addition, since the appearance of the new content after the text segmentation position will basically not affect the translation result before the text segmentation position, the translation of the sub-text to be translated or the audio segment to be translated corresponding to the text segmentation position is accurate and reliable, which improves the accuracy of the simultaneous translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of a simultaneous translation method provided by the present application; Figure 2 It is a structural schematic diagram of an embodiment of a simultaneous translation device provided by the present application; Figure 3 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application; Figure 4 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0014] See also Figure 1 , Figure 11 is a flowchart of an embodiment of a simultaneous translation method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily refer to the embodiment of the present invention. Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes: Step S11: Obtain the audio to be translated.

[0015] In this implementation, the audio to be translated is obtained. The language of the audio to be translated is not limited, for example, the language of the audio to be translated is Chinese, English, French, local dialects (such as Zhejiang dialect, Beijing dialect, Jiangsu dialect, Anhui dialect, etc.). In addition, the translation language corresponding to the audio to be translated is not limited, for example, the translation language is Chinese, English, French, local dialects (such as Zhejiang dialect, Beijing dialect, Jiangsu dialect, Anhui dialect, etc.).

[0016] In one embodiment, the audio to be translated can be obtained from local storage or cloud storage. Of course, in other embodiments, a voice acquisition device can also be used to collect audio in real time as the audio to be translated, which is not limited here.

[0017] Step S12: using the target segmentation model to determine the text segmentation position of the text to be translated corresponding to the audio to be translated.

[0018] In this embodiment, the target segmentation model is used to determine the text segmentation position of the text to be translated corresponding to the audio to be translated, wherein the text segmentation position includes a first text segmentation position and a second text segmentation position, the first text segmentation position is the position between the first sub-text to be translated and the second sub-text to be translated in the text to be translated, the translation influence of the second sub-text to be translated on the first sub-text to be translated is lower than the first influence threshold, and the second text segmentation position is the position of the target punctuation in the text to be translated.

[0019] There is no limitation on the size of the first influence threshold, and it can be set according to actual use needs.

[0020] On the one hand, the first text segmentation position of the text to be translated corresponding to the audio to be translated will be determined, the first text segmentation position is the position between the first sub-text to be translated and the second sub-text to be translated of the text to be translated, and the translation influence of the second sub-text to be translated on the first sub-text to be translated is lower than the first influence threshold. In other words, after translating the first sub-text to be translated before the first text segmentation position, the appearance of the second sub-text to be translated after the first text segmentation position, that is, the appearance of the subsequent new content, has a translation influence on the translation result of the first sub-text to be translated lower than the first influence threshold, that is, there is basically no need to rewrite the translation result of the first sub-text to be translated; that is, it indicates that after determining the first text segmentation position of the text to be translated corresponding to the audio to be translated, the sub-text to be translated or the audio segment to be translated corresponding to the first text segmentation position is thrown out for translation, and the translation result is accurate and reliable.

[0021] After determining the first text segmentation position of the text to be translated corresponding to the audio to be translated, the sub-text to be translated or the audio segment to be translated corresponding to the first text segmentation position is subsequently thrown out for translation. Therefore, the simultaneous translation result can be thrown out more quickly, which improves the timeliness of the simultaneous translation and reduces the delay of the simultaneous translation. In addition, since the appearance of new content after the first text segmentation position will basically not affect the translation result before the first text segmentation position, the translation of the sub-text to be translated or the audio segment to be translated corresponding to the first text segmentation position is accurate and reliable, which improves the accuracy of the simultaneous translation.

[0022] For example, take the text to be translated of the audio to be translated as "I am very hungry now, eat rice" as an example: the first text segmentation position is the position between the first sub-text to be translated "now" and the second sub-text to be translated "very hungry"; the translation result of the first sub-text to be translated "now" is "now", and the translation result of the second sub-text to be translated "very hungry" is "its very hungry", and the second sub-text to be translated "very hungry" has basically no effect on the translation result of the first sub-text to be translated "now"; therefore, when the streaming recognizes the first sub-text to be translated "now" of the text to be translated, the first sub-text to be translated "now" or the corresponding audio segment to be translated is thrown out for translation, and then when the streaming recognizes the second sub-text to be translated "very hungry" of the text to be translated, the second sub-text to be translated "very hungry" or the corresponding audio segment to be translated is thrown out for translation. Compared with having to recognize the complete text to be translated "now very hungry" before the comma and then throw it out for translation, the timeliness of simultaneous translation is improved and the delay of simultaneous translation is reduced.

[0023] For another example, take the text to be translated of the audio to be translated as "I watch TV": if the first text segmentation position is the position between the first sub-text to be translated "I watch" and the second sub-text to be translated "TV"; the translation result of the first sub-text to be translated "I watch" may be "I look", and the translation result of the second sub-text to be translated "TV" is "TV". The second sub-text to be translated "TV" will cause the translation result of the first sub-text to be translated "I watch" to be rewritten as "I watch". Therefore, the second sub-text to be translated has a greater impact on the translation of the first sub-text to be translated, and the position between the first sub-text to be translated "I watch" and the second sub-text to be translated "TV" cannot be used as the first text segmentation position.

[0024] On the other hand, the second text segmentation position of the text to be translated corresponding to the audio to be translated will be determined, and the second text segmentation position is the position of the target punctuation in the text to be translated. The text to be translated before the target punctuation is a semantically complete text to be translated, and the appearance of the text to be translated after the target punctuation, that is, the appearance of subsequent new content, has basically no effect on the translation result before the target punctuation; that is, it means that after determining the second text segmentation position of the text to be translated corresponding to the audio to be translated, the sub-text to be translated or the audio segment to be translated corresponding to the second text segmentation position is thrown out for translation, and the translation result is accurate and reliable.

[0025] After determining the second text segmentation position of the text to be translated corresponding to the audio to be translated, the sub-text to be translated or the audio segment to be translated corresponding to the second text segmentation position is subsequently thrown out for translation. Since the appearance of new content after the second text segmentation position will basically not affect the translation result before the second text segmentation position, the translation of the sub-text to be translated or the audio segment to be translated corresponding to the second text segmentation position is accurate and reliable, thereby improving the accuracy of simultaneous interpretation.

[0026] In one embodiment, before using the target segmentation model to determine the text segmentation position of the text to be translated corresponding to the audio to be translated, the target audio features of the audio to be translated are also obtained, wherein the target audio features are used to input the target segmentation model. That is, after obtaining the audio to be translated, feature extraction is performed on the audio to be translated to obtain the target audio features; then, the target segmentation model is used to determine the text segmentation position of the text to be translated corresponding to the audio to be translated based on the target audio features.

[0027] In a specific implementation, feature extraction is performed on the audio to be translated to obtain Fbank features; then, feature extraction is performed on the Fbank features to obtain hidden layer features as target audio features.

[0028] In one embodiment, the target segmentation model is used to determine the text segmentation position of the text to be translated corresponding to the audio to be translated, specifically: the target segmentation model is used to identify the text to be translated corresponding to the audio to be translated, and the text segmentation position of the text to be translated is obtained. In other words, the audio to be translated is identified using the target segmentation model to obtain the text to be translated and the corresponding text segmentation position.

[0029] In a specific implementation, the target segmentation model is used to identify the text to be translated including the segmentation identifier. Then, at this time, the text segmentation position of the text to be translated is obtained, specifically: the position of the segmentation identifier is used as the text segmentation position of the text to be translated.

[0030] Of course, in other specific implementations, the information obtained by using the target segmentation model recognition may also include two parts, one part is the text to be translated, and the other part is the text segmentation position, that is, the text segmentation position is not directly displayed in the text to be translated as a segmentation identifier.

[0031] In one embodiment, the target segmentation model is a CTC segmentation model.

[0032] Step S13: translating the audio to be translated in sections according to the text segmentation positions.

[0033] In this embodiment, the audio to be translated is translated in sections according to the text segmentation position. After determining the first text segmentation position of the text to be translated corresponding to the audio to be translated, the sub-text to be translated or the audio segment to be translated corresponding to the first text segmentation position is subsequently thrown out for translation, so the simultaneous translation result can be thrown out faster, the timeliness of the simultaneous translation is improved, and the delay of the simultaneous translation is reduced; in addition, due to the appearance of the new content after the first text segmentation position, it will basically not affect the translation result before the first text segmentation position, so the translation of the sub-text to be translated or the audio segment to be translated corresponding to the first text segmentation position is accurate and reliable, which improves the accuracy of the simultaneous translation. After determining the second text segmentation position of the text to be translated corresponding to the audio to be translated, the sub-text to be translated or the audio segment to be translated corresponding to the second text segmentation position is subsequently thrown out for translation, because the appearance of the new content after the second text segmentation position will basically not affect the translation result before the second text segmentation position, so the translation of the sub-text to be translated or the audio segment to be translated corresponding to the second text segmentation position is accurate and reliable, which improves the accuracy of the simultaneous translation.

[0034] Therefore, according to the text segmentation position, the audio to be translated is translated in segments. On the one hand, the simultaneous translation results can be thrown out more quickly, which improves the timeliness of simultaneous translation and reduces the delay of simultaneous translation; on the other hand, it can improve the accuracy of simultaneous translation.

[0035] It should be noted that after using the target segmentation model to determine the text segmentation position of the text to be translated corresponding to the audio to be translated, the sub-text to be translated before the text segmentation position can be directly thrown out for translation to achieve segmented translation of the audio to be translated; in the case of directly throwing out the sub-text to be translated for translation, it is explained that the present application is a cascade translation method, that is, a speech recognition + text translation method. Of course, after using the target segmentation model to determine the text segmentation position of the text to be translated corresponding to the audio to be translated, the text segmentation position can also be used to determine the thrown out audio segment, and the thrown out audio segment can be translated to achieve segmented translation of the audio to be translated; in the case of throwing out the audio segment for translation, it is explained that the present application is an end-to-end translation method, that is, translation is performed directly based on audio, rather than translation based on speech recognition text of audio.

[0036] In one embodiment, before using the target segmentation model to determine the text segmentation position of the text to be translated corresponding to the audio to be translated, the target audio features of the audio to be translated are also obtained; at this time, the audio to be translated is segmented and translated according to the text segmentation position, specifically: using the text segmentation position, the target audio features are divided into multiple feature segments, and each feature segment is translated separately using the translation model.

[0037] When the target audio features are divided into multiple feature segments using the text segmentation positions defined in the present application, the granularity of the feature segments thrown out is smaller; therefore, the feature segments can be thrown out faster, and thus the simultaneous translation results can be thrown out faster, which improves the timeliness of the simultaneous translation and reduces the delay of the simultaneous translation; in addition, due to the appearance of new content after the text segmentation position defined in the present application, it basically does not affect the translation results before the text segmentation position, so the translation of the feature segments obtained based on the text segmentation position division is accurate and reliable, which improves the accuracy of the simultaneous translation.

[0038] In a specific embodiment, the target audio features are divided into multiple feature segments using the text segmentation position, specifically: the audio segmentation position of the audio to be translated is determined using the text segmentation position; the target audio features are divided into multiple feature segments using the audio segmentation position. In other words, the audio segmentation position of the audio to be translated is determined using the text segmentation position, and the target audio features of the audio to be translated are segmented using the audio segmentation position, thereby obtaining multiple feature segments. The subsequent translation is directly based on the audio feature segments, which is an end-to-end translation, can avoid some errors caused by speech recognition, and improve the accuracy of simultaneous interpretation.

[0039] In a specific embodiment, the audio segmentation position includes the audio segmentation moment; determining the audio segmentation position of the audio to be translated by using the text segmentation position specifically includes: obtaining the target character in the text to be translated, where the target character is the character adjacent to and before the text segmentation position; and obtaining the target phoneme sequence of the audio to be translated; determining the target boundary of the target character in the target phoneme sequence; and obtaining the moment corresponding to the target boundary as the audio segmentation moment.

[0040] For example, taking the text to be translated of the audio to be translated as "Now I'm very hungry and eat rice." as an example: The text segmentation position a is between the sub-text to be translated "Now" and the sub-text to be translated "very hungry", so the character "Now" adjacent to and before the text segmentation position a is the target character α; the moment corresponding to the target boundary of the target character α in the target phoneme sequence is t1; therefore, the moment t1 is taken as an audio segmentation moment. The text segmentation position b is the position where the target punctuation ", " is located, so the character "hungry" adjacent to and before the text segmentation position b is the target character β; the moment corresponding to the target boundary of the target character β in the target phoneme sequence is t2; therefore, the moment t2 is taken as an audio segmentation moment. The text segmentation position c is between the sub-text to be translated "eat" and the sub-text to be translated "rice", so the character "eat" adjacent to and before the text segmentation position c is the target character γ; the moment corresponding to the target boundary of the target character γ in the target phoneme sequence is t3; therefore, the moment t3 is taken as an audio segmentation moment. The text segmentation position d is the position where the target punctuation ". " is located, so the character "rice" adjacent to and before the text segmentation position d is the target character δ; the moment corresponding to the target boundary of the target character δ in the target phoneme sequence is t4; therefore, the moment t4 is taken as an audio segmentation moment.

[0041] Therefore, when the audio to be translated is input in a streaming manner subsequently, the feature segment corresponding to the moment 0 - t1 will be cut out for translation, the feature segment corresponding to the moment t1 - t2 will be cut out for translation, the feature segment corresponding to the moment t2 - t3 will be cut out for translation, and the feature segment corresponding to the moment t3 - t4 will be cut out for translation.

[0042] In a specific embodiment, determining the target boundary of the target character in the target phoneme sequence specifically includes: matching the sub-phoneme sequence of the target character from the target phoneme sequence and taking the boundary of the sub-phoneme sequence as the target boundary. That is to say, first match the phoneme sequence belonging to the target character and take the boundary of the phoneme sequence belonging to the target character as the target boundary.

[0043] In one embodiment, the translation of each feature segment of the target audio feature is performed using a translation model, and the determination of the text segmentation position of the to-be-translated text corresponding to the to-be-translated audio is performed using a target segmentation model. The training steps of the translation model and the target segmentation model are specifically: extracting features from the sample audio to obtain the sample audio features; and using the sample audio features to train the target segmentation model and the translation model, respectively. In other words, the training of the translation model and the target segmentation model shares an encoder.

[0044] Specifically, at the input layer, the sample audio is feature extracted to obtain Fbank features; then the Fbank features are sent to the downsampling module and confomer neural network to downsample and encode and decode the Fbank features to obtain frame-level phoneme outputs, which are output as hidden features. Then, the hidden features of the phoneme output are sent to the target segmentation branch to train the target segmentation model, specifically: the hidden features of the phoneme output are combined with softmax through a shallow convolutional neural network to obtain the output of the target segmentation model; and the hidden features of the phoneme output are sent to the translation branch to train the translation model.

[0045] It should be noted that when training the target segmentation model and the translation model, the sample audio will be divided into sample segments according to the corresponding sample segmentation positions, and the sample segments will be stream-joined and sent to the target segmentation model and the translation model for training. In other words, one sentence will be converted into multiple sentences and multiple segmentation results and translation results will be obtained.

[0046] In one embodiment, the target segmentation model is trained using sample audio and corresponding sample audio segmentation labels, and the step of obtaining the sample audio segmentation labels is specifically as follows: obtaining a sample audio text of the sample audio; marking the first sample segmentation position and the second sample segmentation position in the sample audio text respectively, and obtaining the sample audio segmentation label of the sample audio; wherein the first sample segmentation position is the position between the first sub-sample text and the second sub-sample text in the sample audio text, the second sub-sample text has a translation influence on the first sub-sample text lower than the second influence threshold, and the second sample segmentation position is the position of the sample punctuation in the sample audio text. wherein, the size of the second influence threshold is not limited, and can be specifically set according to actual use needs.

[0047] That is to say, the sample audio segmentation labels used to train the target segmentation model are not constructed through manual labeling; the sample audio segmentation label construction method provided in the present application, compared with the method of constructing through manual labeling, on the one hand, can improve the accuracy of the constructed sample audio segmentation labels; on the other hand, it can improve the efficiency of constructing sample audio segmentation labels and avoid excessive reliance on manual labor.

[0048] In addition, the sample audio segmentation label is obtained by marking the first sample segmentation position and the second sample segmentation position in the sample audio text, and the first sample segmentation position is the position between the first sub-sample text and the second sub-sample text in the sample audio text, the second sub-sample text has a translation influence on the first sub-sample text lower than the second influence threshold, and the second sample segmentation position is the position of the sample punctuation in the sample audio text. Therefore, the target segmentation model is trained using the sample audio and the corresponding sample audio segmentation label, so that the target segmentation model with converged training can determine the text segmentation position of the text to be translated corresponding to the audio to be translated. Subsequently, the sub-text to be translated or the audio segment to be translated corresponding to the text segmentation position is thrown out for translation, so the simultaneous translation result can be thrown out faster, the timeliness of the simultaneous translation is improved, and the delay of the simultaneous translation is reduced; in addition, due to the appearance of the new content after the text segmentation position, it basically does not affect the translation result before the text segmentation position, so the translation of the sub-text to be translated or the audio segment to be translated corresponding to the text segmentation position is accurate and reliable, which improves the accuracy of the simultaneous translation.

[0049] In a specific embodiment, the sample audio segmentation label is a first sample label text, which is formed by marking a sample identifier in the sample audio text, and the sample identifier includes a first identifier added at the first sample segmentation position and a second identifier replacing the sample punctuation at the first sample segmentation position.

[0050] The specific presentation forms of the first identifier and the second identifier are not limited. For example, the first identifier is " <c>", the second identifier is " <c> ”、" <ju> ”、" <wen>"or" <gan>"wait.

[0051] For example, the sample audio text of the sample audio is "I am very hungry now", the first identifier is " <c>”For example: The translation result of the first sub-sample text "now" is "now", and the translation result of the second sub-sample text "extremely hungry" is "its very hungry". The translation result of the second sub-sample text "extremely hungry" has basically no impact on the translation result of the first sub-sample text "now". Therefore, the position between the first sub-sample text "now" and the second sub-sample text "extremely hungry" is the first sample segmentation position. Thus, mark the first identifier at the position between the first sub-sample text "now" and the second sub-sample text "extremely hungry" <c>", get the first sample label text "Now <c>Very hungry."

[0052] In a specific implementation, the sample punctuation includes several punctuation types, and at least two punctuation types have different corresponding second identifiers. The presentation form of the second identifier corresponding to each punctuation type is not limited and can be specifically set according to actual use needs.

[0053] For example, sample punctuation marks include commas, periods, question marks, and exclamation marks; the second identifier corresponding to the comma is " <dou>", the second identifier corresponding to the period is " <ju>", the second identifier corresponding to the question mark is " <wen>", the second identifier corresponding to the exclamation mark is " <gan>”.

[0054] In a specific implementation, the sample punctuation includes a comma, and the second identifier corresponding to the comma is the same as the first identifier.

[0055] For example, the first identifier is " <c>", the second identifier corresponding to the comma is " <c>”.

[0056] For example, the sample audio text of the sample audio is "Then I was very hungry and ate rice.", the first identifier and the second identifier corresponding to the comma are " <c>", the second identifier corresponding to the period is " <ju>"For example: The translation result of the first sub-sample text "then" is "afterwards", and the translation result of the second sub-sample text "extremely hungry" is "its veryhungry". The translation result of the second sub-sample text "extremely hungry" has basically no impact on the translation result of the first sub-sample text "then". Therefore, the position between the first sub-sample text "then" and the second sub-sample text "extremely hungry" is the first sample segmentation position. Thus, mark the first identifier at the position between the first sub-sample text "then" and the second sub-sample text "extremely hungry" <c>”. The translation result of the first sub-sample text "ate" is "eat", and the translation result of the second sub-sample text "rice" is "rice". The second sub-sample text "rice" has basically no influence on the translation result of the first sub-sample text "ate". Therefore, the position between the first sub-sample text "ate" and the second sub-sample text "rice" is the first sample segmentation position. Thus, the second identifier corresponding to the comma is marked at the position between the first sub-sample text "ate" and the second sub-sample text "rice" <c>". At the position of the period in the sample audio text "Then I was very hungry and ate rice." of the sample audio, mark the second identifier corresponding to the period" <ju>Therefore, the sample audio text of the sample audio "Then I was very hungry and ate rice." The corresponding sample audio segmentation label is "Then <c>Very hungry <c>have eaten <c>rice <ju>”.

[0057] In a specific implementation, the first sample segmentation position of the labeled sample audio text is performed by a segmentation model, and the training step of the segmentation model includes: using a plurality of first models to respectively label the reference segmentation position of the reference text, and obtaining the first segmentation label corresponding to each first model; wherein the reference segmentation position is the position between the first sub-reference text and the second sub-reference text in the reference text, and the translation influence of the second sub-reference text on the first sub-reference text is less than the third influence threshold; based on the first segmentation label corresponding to each first model, at least one first model whose annotation quality meets the quality requirements is selected from the plurality of first models as the second model; the first segmentation label corresponding to each second model is integrated to obtain the second segmentation label as the reference segmentation label of the reference text; and the segmentation model is trained using the reference text and the corresponding reference segmentation label. wherein, the size of the third influence threshold is not limited, and can be specifically set according to actual use needs.

[0058] That is to say, several currently open-source large models are used to annotate the reference segmentation positions of the reference text to obtain the first segmentation labels annotated by the large models; then, the first segmentation labels of the large models with high annotation quality are fused as the reference segmentation labels of the reference text. The reference segmentation labels of the reference text are obtained by fusing the first segmentation labels of the large models with high annotation quality, so the reference segmentation labels of the reference text are highly accurate. Furthermore, the large segmentation models are trained using the reference text and the corresponding reference segmentation labels, so that the large segmentation models that have converged in training can accurately annotate the first sample segmentation positions of the sample audio text of the sample audio.

[0059] In a specific embodiment, the first segmentation labels corresponding to the second large models are fused to obtain the second segmentation labels, including: extracting the first common reference segmentation position from the first segmentation labels corresponding to the second large models, and using the first common reference segmentation position as the second segmentation label. In other words, the common reference segmentation position is extracted from the reference segmentation positions annotated by the large models with high annotation quality as the second segmentation label. The reference segmentation positions annotated by the large models with high annotation quality must be accurate; using the reference segmentation positions annotated by the large models with high annotation quality as the second segmentation label can improve the accuracy of the second segmentation label.

[0060] In a specific embodiment, the first segmentation label and the second segmentation label are first label texts, and the first label texts are formed by adding the first identifier to the reference segmentation position corresponding to the annotated reference text. In other words, the first segmentation label and the second segmentation label are texts including the reference text and the first identifier formed by annotating the first identifier in the reference text.

[0061] In a specific implementation, based on the first segmentation labels corresponding to each first large model, at least one first large model whose annotation quality meets the quality requirements is selected from a number of first large models as the second large model, specifically: based on the first segmentation labels corresponding to each first large model, the annotation accuracy of each first large model is determined as the annotation quality of each first large model; and from a number of first large models, the first large model whose annotation quality meets the quality requirements is selected as the second large model. In other words, the annotation quality of each first large model is determined by the annotation accuracy of each first large model.

[0062] In a specific implementation, the first large model that meets the quality requirements is: the first large model whose annotation accuracy is greater than the annotation accuracy of a preset number of other large models. In other words, the large model with the highest annotation accuracy is regarded as a large model with high annotation quality.

[0063] There is no limitation on the size of the preset number, for example, the preset number is 2, 3, 5, etc. For example, the first largest model with the first and second largest annotation accuracy is used as the second largest model as an example: the reference segmentation label of the reference text is obtained by fusing the first segmentation label of the first largest model with the first and second largest annotation accuracy; therefore, the reference segmentation label of the reference text has high accuracy. Further, the large segmentation model is trained using the reference text and the corresponding reference segmentation label, so that the large segmentation model with converged training can accurately mark the first sample segmentation position of the sample audio text of the sample audio.

[0064] In a specific embodiment, the segmentation model is the first model with the highest annotation quality. That is, the first model with the highest annotation quality is used as the segmentation model, and the reference text and the corresponding reference segmentation label are used to fine-tune the segmentation model so that the segmentation model converges, so that the converged segmentation model can accurately mark the first sample segmentation position of the sample audio text of the sample audio.

[0065] In a specific implementation, after the large segmentation model is trained using reference text and corresponding reference segmentation labels, the large segmentation model is used to mark the reference text with reference segmentation positions to obtain predicted segmentation labels; the predicted segmentation labels and reference segmentation labels are fused to obtain new segmentation labels; the new segmentation labels are used as new reference segmentation labels, and the training of the large segmentation model using the reference text and corresponding reference segmentation labels and subsequent steps are re-executed until the training converges.

[0066] That is to say, the reference text is annotated using the segmentation model that has been fine-tuned once to obtain a predicted segmentation label; then, the reference segmentation label of the reference text and the predicted segmentation label are fused to serve as the new reference segmentation label of the reference text. The new reference segmentation label of the reference text is obtained by fusing the original reference segmentation label of the reference text and the predicted segmentation label of the segmentation model that has been fine-tuned once, so the new reference segmentation label of the reference text has high accuracy. Furthermore, the reference text and the new reference segmentation label are used to train the segmentation model, so that the segmentation model that has converged in training can accurately annotate the first sample segmentation position of the sample audio text of the sample audio.

[0067] In a specific embodiment, the predicted segmentation label and the reference segmentation label are fused to obtain a new segmentation label, specifically: extract the second common reference segmentation position from the predicted segmentation label and the reference segmentation label, and use the second common reference segmentation position as the new segmentation label. In other words, extract the common reference segmentation position in the predicted segmentation label and the original reference segmentation label of the reference text as the new reference segmentation label. The reference segmentation position that exists in both the reference segmentation label of the reference text and the predicted segmentation label must be accurate; using the reference segmentation position that exists in both the reference segmentation label of the reference text and the predicted segmentation label as the new reference segmentation label of the reference text can improve the accuracy of the obtained reference segmentation label.

[0068] In a specific implementation, the reference text acquisition step is specifically as follows: acquiring a number of sample texts; and selecting sample texts with correct reference punctuation from the sample texts as reference texts. That is, a large number of sample texts are acquired, and the sample texts with correct punctuation are selected as reference texts.

[0069] In a specific implementation, sample texts with correct reference punctuation are selected from a number of sample texts as reference texts, specifically: for each sample text, the accuracy of each reference punctuation in the sample text is scored to obtain the accuracy score of each reference punctuation in the sample text; the accuracy scores of each reference punctuation in the sample text are summed to obtain a total score; in response to the total score being greater than a first threshold, the sample text is used as a reference text. The size of the first threshold is not limited and can be specifically set according to actual use needs. In other words, the sample texts are selected by scoring the accuracy of the punctuation marks of each sample text.

[0070] In a specific implementation, sample texts with correct reference punctuation are selected from a number of sample texts as reference texts, specifically: for each sample text, a punctuation generation model is used to generate a number of predicted punctuation of the sample text; the differences between the predicted punctuation and the reference punctuation in the sample text are compared; in response to the difference between the predicted punctuation and the reference punctuation in the sample text being less than a second threshold, the sample text is used as the reference text. The size of the second threshold is not limited and can be specifically set according to actual use needs. That is, the sample text is selected by using the punctuation generation model to simulate the way of generating punctuation marks.

[0071] In one embodiment, a sample translation label corresponding to the sample audio is generated using the sample audio segmentation label, and the steps for obtaining the sample translation label are specifically as follows: obtaining sub-sample texts between each sample segmentation position; translating each sub-sample text to obtain a sample translation result of each sub-sample text; for each sample segmentation position, combining the sample translation results of the sub-sample text before the sample segmentation position to obtain a sample translation result corresponding to the sample segmentation position; wherein the sample translation results corresponding to each sample segmentation position constitute the sample translation label.

[0072] That is to say, the sample translation label corresponding to the sample audio is not constructed through manual annotation. Compared with the method of constructing through manual annotation, the sample translation label construction method provided by the present application can, on the one hand, improve the accuracy of the constructed sample translation label; on the other hand, it can improve the efficiency of constructing sample translation labels and avoid excessive reliance on manual labor.

[0073] In addition, the sample translation results corresponding to each sample segmentation position are obtained by combining the sample translation results of the subsample text before the sample segmentation position, and are incremental translation results obtained by fixing the historical translation results at the same time. Furthermore, the sample audio and the corresponding sample translation labels are used to train the translation model, so that the translation model can learn to translate the sample audio segment in combination with the historical translation results to improve the translation performance.

[0074] For example, the sample audio text of the sample audio is "Then I was particularly hungry.", and its corresponding sample audio segmentation label is "Then <c>Very hungry <ju>"For example: for the sample segmentation position" <c>”, and its corresponding sub-sample text is "然后", and the sample translation result of the sub-sample text "然后" is "afterwards"; therefore, the sample segmentation position" <c>"The corresponding sample translation result is "afterwards", and the sample translation result "afterwards" constitutes a sample translation label. For the sample segmentation position " <ju>”, and its corresponding sub-sample text is "特别饿", while the sample translation result corresponding to the sub-sample text "特别饿" is "its very hungry"; the sample segmentation position" <ju>”The corresponding sample translation result is obtained by combining the sample translation result of the sub-sample text "then", which is "afterwards", and the translation result of the sub-sample text "特别饿", which is "its very hungry". That is, the sample segmentation position <ju>"The corresponding sample translation result is "afterwards its very hungry". The sample translation result "afterwards its very hungry" constitutes a sample translation label.

[0075] See also Figure 2 , Figure 2 It is a structural diagram of an embodiment of a simultaneous translation device provided by the present application. The simultaneous translation device 20 includes an acquisition module 21, a determination module 22 and a translation module 23; the acquisition module 21 is used to acquire the audio to be translated; the determination module 22 is used to determine the text segmentation position of the text to be translated corresponding to the audio to be translated using the target segmentation model; wherein the text segmentation position includes a first text segmentation position and a second text segmentation position, the first text segmentation position is the position between the first sub-text to be translated and the second sub-text to be translated in the text to be translated, the translation influence of the second sub-text to be translated on the first sub-text to be translated is lower than the first influence threshold, and the second text segmentation position is the position of the target punctuation in the text to be translated; the translation module 23 is used to segment the audio to be translated according to the text segmentation position.

[0076] Among them, the acquisition module 21 is also used to obtain the target audio features of the audio to be translated before using the target segmentation model to determine the text segmentation position of the text to be translated corresponding to the audio to be translated; wherein the target audio features are used to input the target segmentation model; and / or, the determination module 22 is used to determine the text segmentation position of the text to be translated corresponding to the audio to be translated using the target segmentation model, including: using the target segmentation model to identify the text to be translated corresponding to the audio to be translated, and obtaining the text segmentation position of the text to be translated.

[0077] Among them, the acquisition module 21 is used to obtain the target audio features of the audio to be translated after determining the text segmentation position of the text to be translated corresponding to the audio to be translated using the target segmentation model; the translation module 23 is used to perform segmented translation of the audio to be translated according to the text segmentation position, including: using the text segmentation position to determine the audio segmentation position of the audio to be translated; using the audio segmentation position to divide the target audio features into multiple feature segments; using the translation model to translate each feature segment separately.

[0078] Among them, the above-mentioned audio segmentation position includes the audio segmentation moment; the translation module 23 is used to use the text segmentation position to determine the audio segmentation position of the audio to be translated, including: obtaining the target character in the text to be translated; wherein the target character is the character located before the text segmentation position and adjacent to the text segmentation position; and, obtaining the target phoneme sequence of the audio to be translated; determining the target boundary of the target character in the target phoneme sequence; obtaining the moment corresponding to the target boundary as the audio segmentation moment.

[0079] Among them, the above-mentioned target segmentation model is trained using sample audio and corresponding sample audio segmentation labels. The simultaneous translation device 20 also includes a labeling module 24. The labeling module 24 is used for the steps of obtaining sample audio segmentation labels, including: obtaining a sample audio text of the sample audio; marking the first sample segmentation position and the second sample segmentation position in the sample audio text respectively, and obtaining the sample audio segmentation label of the sample audio; wherein the first sample segmentation position is the position between the first sub-sample text and the second sub-sample text in the sample audio text, the second sub-sample text has an influence on the translation of the first sub-sample text lower than the second influence threshold, and the second sample segmentation position is the position of the sample punctuation in the sample audio text.

[0080] Among them, the above-mentioned sample audio segmentation label is a first sample label text, which is formed by setting a sample identifier in the sample audio text, and the sample identifier includes a first identifier added at the first sample segmentation position and a second identifier replacing the sample punctuation at the second sample segmentation position.

[0081] Among them, the first sample segmentation position of the labeled sample audio text is performed by the segmentation large model, and the simultaneous translation device 20 also includes a training module 25, which is used for training the segmentation large model, including: using a plurality of first large models to respectively mark the reference segmentation position of the reference text to obtain the first segmentation label corresponding to each first large model; wherein the reference segmentation position is the position between the first sub-reference text and the second sub-reference text in the reference text, and the translation influence of the second sub-reference text on the first sub-reference text is less than the third influence threshold; based on the first segmentation label corresponding to each first large model, at least one first large model whose annotation quality meets the quality requirements is selected from the plurality of first large models as the second large model; the first segmentation label corresponding to each second large model is integrated to obtain the second segmentation label as the reference segmentation label of the reference text; and the segmentation large model is trained using the reference text and the corresponding reference segmentation label.

[0082] Among them, the above-mentioned first segmentation label and second segmentation label are first label texts, and the first label text is formed by adding a first identifier to the reference segmentation position corresponding to the annotation in the reference text; and / or, the training module 25 is used to fuse the first segmentation labels corresponding to each second large model to obtain the second segmentation label, including: extracting the first common reference segmentation position from the first segmentation labels corresponding to each second large model, and using the first common reference segmentation position as the second segmentation label.

[0083] Among them, the training module 25 is used to select at least one first large model whose annotation quality meets the quality requirements from a number of first large models based on the first segmentation labels corresponding to each first large model, as the second large model, including: determining the annotation accuracy of each first large model based on the first segmentation labels corresponding to each first large model, as the annotation quality of each first large model; selecting a first large model whose annotation quality meets the quality requirements from a number of first large models as the second large model; wherein the first large model that meets the quality requirements is: a first large model whose annotation accuracy is greater than the annotation accuracy of a preset number of other large models; and / or a segmented large model is a first large model with the highest annotation quality.

[0084] Among them, the training module 25 is also used to obtain reference texts, including: obtaining a number of sample texts; for each sample text, scoring the accuracy of each reference punctuation in the sample text to obtain the accuracy score of each reference punctuation in the sample text; summing the accuracy scores of each reference punctuation in the sample text to obtain a total score; in response to the total score being greater than a first threshold, using the sample text as a reference text; or, for each sample text, using a punctuation generation model to generate a number of predicted punctuation of the sample text; comparing the differences between the number of predicted punctuation and the number of reference punctuation in the sample text; in response to the degree of difference between the number of predicted punctuation and the number of reference punctuation in the sample text being less than a second threshold, using the sample text as a reference text.

[0085] Among them, the training module 25 is used to use the reference text and the corresponding reference segmentation label to train the large segmentation model, and then use the large segmentation model to mark the reference text reference segmentation position to obtain a predicted segmentation label; fuse the predicted segmentation label and the reference segmentation label to obtain a new segmentation label; use the new segmentation label as a new reference segmentation label, and re-execute the training of the large segmentation model using the reference text and the corresponding reference segmentation label and its subsequent steps until the training converges.

[0086] See also Figure 3 , Figure 3 3 is a schematic diagram of the structure of an embodiment of an electronic device provided by the present application. The electronic device 30 includes a memory 31 and a processor 32 coupled to each other, and the processor 32 is used to execute program instructions stored in the memory 31 to implement the steps of any of the above-mentioned simultaneous interpretation method embodiments. In a specific implementation scenario, the electronic device 30 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 30 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0087] Specifically, the processor 32 is used to control itself and the memory 31 to implement the steps of any of the above-mentioned simultaneous interpretation method embodiments. The processor 32 can also be called a CPU (Central Processing Unit). The processor 32 may be an integrated circuit chip with signal processing capabilities. The processor 32 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 32 can be implemented by an integrated circuit chip.

[0088] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of a computer-readable storage medium provided by the present application. The computer-readable storage medium 40 of the embodiment of the present application stores a program instruction 41, and when the program instruction 41 is executed, the method provided by any embodiment of the simultaneous interpretation method of the present application and any non-conflicting combination is implemented. Among them, the program instruction 41 can form a program file and be stored in the above-mentioned computer-readable storage medium 40 in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the methods of each implementation method of the present application. The aforementioned computer-readable storage medium 40 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0089] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, set clear and prominent signs to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to collect his or her personal information; or on the device that processes personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0090] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.< / ju> < / ju> < / ju> < / c> < / c> < / ju> < / c> < / ju> < / c> < / c> < / c> < / ju> < / c> < / c> < / ju> < / c> < / c> < / c> < / gan> < / wen> < / ju> < / dou> < / c> < / c> < / c> < / gan> < / wen> < / ju> < / c> < / c>

Claims

1. A simultaneous translation method, characterized in that: The method comprises: Get the audio to be translated; Determine the text segmentation position of the text to be translated corresponding to the audio to be translated by using the target segmentation model; wherein the text segmentation position includes a first text segmentation position and a second text segmentation position, the first text segmentation position is the position between a first sub-text to be translated and a second sub-text to be translated in the text to be translated, the translation influence of the second sub-text to be translated on the first sub-text to be translated is lower than a first influence threshold, and the second text segmentation position is the position of the target punctuation in the text to be translated; The audio to be translated is translated in sections according to the text segmentation positions.

2. The method according to claim 1, characterized in that Before determining the text segmentation position of the text to be translated corresponding to the audio to be translated by using the target segmentation model, the method further includes: Acquire target audio features of the audio to be translated; wherein the target audio features are used to input the target segmentation model; And / or, the determining the text segmentation position of the text to be translated corresponding to the audio to be translated by using the target segmentation model includes: The target segmentation model is used to identify the text to be translated corresponding to the audio to be translated, and the text segmentation position of the text to be translated is obtained.

3. The method according to claim 1, characterized in that After determining the text segmentation position of the text to be translated corresponding to the audio to be translated by using the target segmentation model, the method further includes: Acquire target audio features of the audio to be translated; The step of translating the audio to be translated in sections according to the text segmentation positions includes: Determining the audio segmentation position of the audio to be translated by using the text segmentation position; Using the audio segmentation position, dividing the target audio feature into a plurality of feature segments; The translation model is used to translate each of the characteristic segments.

4. The method according to claim 3, characterized in that: The audio segmentation position includes the audio segmentation time; and the step of using the text segmentation position to determine the audio segmentation position of the audio to be translated includes: Acquire target characters in the text to be translated; wherein the target characters are characters located before the text segmentation position and adjacent to the text segmentation position; And, obtaining a target phoneme sequence of the audio to be translated; Determining a target boundary of the target character in the target phoneme sequence; The time corresponding to the target boundary is obtained as the audio segmentation time.

5. The method according to claim 1, characterized in that The target segmentation model is trained using sample audio and corresponding sample audio segmentation labels, and the steps of obtaining the sample audio segmentation labels include: Get the sample audio text of the sample audio; The first sample segmentation position and the second sample segmentation position in the sample audio text are respectively marked to obtain a sample audio segmentation label of the sample audio; wherein the first sample segmentation position is a position between a first sub-sample text and a second sub-sample text in the sample audio text, the translation influence of the second sub-sample text on the first sub-sample text is lower than a second influence threshold, and the second sample segmentation position is a position where a sample punctuation in the sample audio text is located.

6. The method according to claim 5, characterized in that The sample audio segmentation label is a first sample label text, which is formed by setting a sample identifier in the sample audio text, and the sample identifier includes a first identifier added at the first sample segmentation position and a second identifier replacing the sample punctuation at the second sample segmentation position.

7. The method according to claim 5, characterized in that The first sample segmentation position of the sample audio text is marked by a segmentation model, and the training steps of the segmentation model include: Using a plurality of first large models to respectively mark reference segmentation positions of the reference text, and obtain first segmentation labels corresponding to the first large models; wherein the reference segmentation position is a position between a first sub-reference text and a second sub-reference text in the reference text, and the translation influence of the second sub-reference text on the first sub-reference text is less than a third influence threshold; Based on the first segmentation labels corresponding to the first large models, at least one first large model whose annotation quality meets the quality requirement is selected from the plurality of first large models as the second large model; The first segmentation labels corresponding to the second largest models are fused to obtain second segmentation labels as reference segmentation labels of the reference text; The segmentation model is trained using the reference text and the corresponding reference segmentation labels.

8. The method according to claim 7, characterized in that The first segmentation label and the second segmentation label are first label texts, and the first label texts are formed by adding a first identifier to the reference segmentation position corresponding to the annotation in the reference text; And / or, the fusing of the first segmentation labels corresponding to each of the second largest models to obtain the second segmentation labels comprises: A first common reference segmentation position is extracted from the first segmentation labels corresponding to each of the second large models, and the first common reference segmentation position is used as the second segmentation label.

9. The method according to claim 7, characterized in that: The selecting, based on the first segmentation labels corresponding to the first large models, at least one first large model whose annotation quality meets the quality requirement from the plurality of first large models as the second large model comprises: Determining, based on the first segmentation labels corresponding to the first large models, the labeling accuracy of the first large models as the labeling quality of the first large models; From the plurality of first large models, a first large model whose annotation quality meets the quality requirements is selected as the second large model; The first large model that meets the quality requirements is: the first large model whose labeling accuracy is greater than the labeling accuracy of a preset number of other large models; and / or the segmented large model is the first large model with the highest labeling quality.

10. The method according to claim 7, characterized in that The step of obtaining the reference text comprises: Get some sample texts; For each of the sample texts, scoring the accuracy of each reference punctuation in the sample text to obtain an accuracy score of each reference punctuation in the sample text; Sum up the accuracy scores of the reference punctuation points of the sample text to obtain a total score; In response to the total score being greater than a first threshold, using the sample text as the reference text; Alternatively, for each of the sample texts, a punctuation generation model is used to generate a number of predicted punctuations of the sample text; comparing differences between the plurality of predicted punctuation marks and a plurality of reference punctuation marks in the sample text; In response to the difference between the plurality of predicted punctuation marks and a plurality of reference punctuation marks in the sample text being less than a second threshold, the sample text is used as the reference text.

11. The method according to claim 7, characterized in that After the large segmentation model is trained using the reference text and the corresponding reference segmentation label, the method further includes: Using the large segmentation model, marking the reference segmentation position of the reference text to obtain a predicted segmentation label; Fusion of the predicted segmentation label and the reference segmentation label to obtain a new segmentation label; The new segmentation label is used as the new reference segmentation label, and the training of the large segmentation model using the reference text and the corresponding reference segmentation label and subsequent steps are re-executed until the training converges.

12. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the method according to any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the method according to any one of claims 1-11.

Images