Multimodal video captioning and audio collaborative translation and dynamic shunting method and system

Abstract

The application relates to the field of audio-video collaborative translation, and discloses a multi-modal video subtitle and audio collaborative translation and dynamic shunting method and system, which comprises the following steps: performing feature extraction on a pre-acquired multi-modal video, simulating semantic attraction and semantic repulsion between different modal feature points, and constructing a multi-modal semantic field; calculating a semantic flow energy difference value in a first flow path, screening out a first translation area and a second translation area; performing collaborative translation on subtitles and audio to obtain a first translation result; screening out a translation difficulty point in combination with a second translation path and the corresponding first flow path, performing semantic decoding and semantic stitching on the translation difficulty point, and obtaining a second translation result; and combining the first translation result and the second translation result to obtain a collaborative translation result; through collaborative translation, the application can effectively integrate subtitle and audio information, and improve the accuracy and consistency of a translation result.

Description

Technical Field

[0001] This application relates to the field of audio and video collaborative translation technology, and more specifically to a method and system for multimodal video subtitle and audio collaborative translation and dynamic splitting. Background Technology

[0002] Multimodal video translation technology aims to automatically translate subtitles and audio content in videos into the target language to support cross-language access and global dissemination. Existing technologies are primarily based on machine translation models, but are limited to single-modal processing or simple multimodal fusion, translating only text subtitles while ignoring audio information and lacking deep semantic analysis, resulting in low translation quality. They also cannot effectively handle semantically complex regions, such as culture-specific expressions, logical jumps, or modal conflicts, leading to unnatural or inaccurate translation results.

[0003] Existing technologies suffer from the following problems: processing subtitles or audio separately ignores the semantic relationships between modalities, leading to inconsistent translation results or information loss; translating based on surface features of data fails to deeply understand the profound relationships of multimodal semantics, such as semantic attraction, repulsion, or logical conflict, resulting in translation errors in complex scenarios; processing all data based on a unified model fails to dynamically adjust translation strategies according to semantic complexity, leading to over-processing of simple regions or under-processing of complex regions, affecting overall translation efficiency and quality. To address at least one of the above problems, this application proposes a method and system for multimodal video subtitle and audio collaborative translation and dynamic splitting. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this application is to provide a method and system for multimodal video subtitle and audio collaborative translation and dynamic streaming, which can effectively solve the problems in the background technology. The specific technical solution of this application is as follows:

[0005] Multimodal video subtitle and audio collaborative translation and dynamic splitting methods include:

[0006] Feature extraction is performed on the pre-acquired multimodal video, and the features are mapped to a high-dimensional semantic space to simulate the semantic attraction and semantic repulsion between feature points of different modalities, thereby constructing a multimodal semantic field;

[0007] In the multimodal semantic field, a first flow path with semantic flow is identified, the semantic flow energy difference in the first flow path is calculated, and regions with semantic flow energy differences less than or equal to a preset energy difference threshold are selected as the first translation region, while regions with semantic flow energy differences greater than the preset energy difference threshold are selected as the second translation region.

[0008] The first translation path of the first translation region is identified by a preset first translation model, and the subtitles and audio are translated collaboratively to obtain the first translation result.

[0009] The second translation path of the second translation region is identified by a preset second translation model. The translation difficulties are screened by combining the second translation path and the corresponding first flow path. The translation difficulties are then semantically decoded and semantically stitched to obtain the second translation result.

[0010] By combining the first translation result and the second translation result, a collaborative translation result is obtained.

[0011] Specifically, the step of extracting features from the pre-acquired multimodal video, mapping the features to a high-dimensional semantic space, simulating semantic attraction and repulsion between feature points of different modalities, and constructing a multimodal semantic field includes:

[0012] Feature extraction is performed on the pre-acquired multimodal video to obtain text feature sequences and audio feature sequences. The text feature sequences and audio feature sequences are then mapped and projected into a high-dimensional semantic space to construct a feature point cloud.

[0013] Calculate the semantic recognition confidence and cross-modal semantic consistency of each feature point in the feature point cloud to obtain the semantic energy value and construct the semantic point cloud;

[0014] In semantic point clouds, we simulate the semantic attraction and repulsion between feature points of different modalities to construct a multimodal semantic field.

[0015] Specifically, in the semantic point cloud, simulating semantic attraction and repulsion between feature points of different modalities to construct a multimodal semantic field includes:

[0016] In semantic point clouds, the similarity between semantic features of semantic points is calculated, and a similarity matrix is ​​constructed.

[0017] In the feature point cloud, the co-occurrence frequency of features of different modalities is counted and the corresponding semantic association degree is calculated to construct a semantic association matrix;

[0018] By combining the similarity matrix and semantic association matrix, and using a preset attraction analysis model, the semantic attraction between feature points of different modalities is simulated and calculated to construct an attraction distribution map;

[0019] Feature point pairs that are less than a preset distance threshold and have a semantic relevance less than a preset relevance threshold in the attraction distribution map are selected as semantic conflict point pairs, and the semantic repulsion of the semantic conflict point pairs is calculated.

[0020] We calculate the combined effect of semantic attraction and semantic repulsion to construct a multimodal semantic field.

[0021] Specifically, identifying a first flow path with semantic flow in the multimodal semantic field and calculating the semantic flow energy difference in the first flow path includes:

[0022] In a multimodal semantic field, analyze the direction of semantic propagation and identify the first flow path where semantic flow exists;

[0023] Extract the energy value sequence along the first flow path, select the energy maxima as semantic barrier points, calculate the energy consumption of each semantic barrier point, and sum them to obtain the semantic flow energy difference of the first flow path.

[0024] Specifically, the step of analyzing the semantic propagation direction in a multimodal semantic field and identifying the first flow path where semantic flow exists includes:

[0025] By using a pre-defined semantic node recognition model, the direction of semantic propagation is analyzed in a multimodal semantic field, and semantic start nodes, semantic turning points, and semantic termination nodes are identified to construct the first set of nodes.

[0026] Starting from each semantic start node in the first node set, the semantic propagation direction is analyzed, the corresponding semantic turning point nodes are selected, and the semantic termination nodes are selected according to the semantic turning point nodes. Each path connecting the semantic start node, semantic turning point node and semantic termination node is taken as the first flow path.

[0027] Specifically, the step of identifying the first translation path of the first translation region through a preset first translation model, and performing collaborative translation of subtitles and audio to obtain a first translation result includes:

[0028] The first translation path with the highest semantic relevance among nodes in the first translation region is identified using a pre-defined first translation model.

[0029] Following the first translation path, the subtitles and audio are translated collaboratively to obtain the first translation result.

[0030] Specifically, the second translation path of the second translation region is identified through a preset second translation model, and translation difficulties are screened by combining the second translation path with the corresponding first flow path. Semantic decoding and semantic stitching are then performed on these translation difficulties to obtain the second translation result, including:

[0031] The second translation path of the second translation region is identified by a preset second translation model. The semantic breakpoints and logical conflict points are identified by combining the second translation path with the corresponding first flow path, and a set of translation difficulties is constructed.

[0032] Semantic decoding is performed on each translation difficulty to generate a set of translation fragments. The semantically complete and logically reasonable translation fragments are then selected for semantic stitching to obtain the second translation result.

[0033] Specifically, the second translation path of the second translation region is identified through a preset second translation model, and semantic breakpoints and logical conflict points are identified by combining the second translation path with the corresponding first flow path, thus constructing a set of translation difficulties, including:

[0034] The second translation path with the highest semantic relevance among nodes in the second translation region is identified using a pre-defined second translation model.

[0035] By combining the second translation path and the corresponding first flow path, the path curvature is calculated to filter out regions that are greater than the preset path curvature threshold, which are used as logical jump regions.

[0036] The semantic interruption detection model identifies the semantic interruption locations in the logical jump region as semantic breakpoints.

[0037] Based on the semantic breakpoints, the feature dispersion is calculated and positions with feature dispersion greater than a preset dispersion threshold are selected as logical conflict points.

[0038] By combining the semantic breakpoints and logical conflict points, a set of translation difficulties is constructed.

[0039] Specifically, the process involves semantic decoding of each translation difficulty to generate a set of translation fragments. Semantically complete and logically sound translation fragments are then selected for semantic stitching to obtain a second translation result, including:

[0040] For each translation challenge, a pre-set causal intervention model is used to analyze semantic logical relationships and generate interference blocking factors.

[0041] Based on the interference blocking factor, a set of translated segments is generated using a preset semantic decoder model;

[0042] From the set of translated segments, semantically complete and logically sound translated segments are selected, and semantic stitching is performed to obtain the second translation result.

[0043] A multimodal video subtitle and audio collaborative translation and dynamic splitting system is used to implement the aforementioned multimodal video subtitle and audio collaborative translation and dynamic splitting method, including:

[0044] The multimodal semantic field construction module extracts features from the pre-acquired multimodal video, maps the features to a high-dimensional semantic space, simulates the semantic attraction and repulsion between feature points of different modalities, and constructs a multimodal semantic field.

[0045] The dynamic diversion module identifies a first flow path with semantic flow in the multimodal semantic field, calculates the semantic flow energy difference in the first flow path, and filters out regions with semantic flow energy differences less than or equal to a preset energy difference threshold as first translation regions, and regions with semantic flow energy differences greater than the preset energy difference threshold as second translation regions.

[0046] The first translation module identifies the first translation path of the first translation region through a preset first translation model, and performs collaborative translation of subtitles and audio to obtain the first translation result.

[0047] The second translation module identifies the second translation path of the second translation region through a preset second translation model, combines the second translation path with the corresponding first flow path to filter out translation difficulties, performs semantic decoding and semantic stitching on the translation difficulties, and obtains the second translation result.

[0048] The translation result integration module combines the first translation result and the second translation result to obtain a collaborative translation result.

[0049] The beneficial effects of this application are as follows: By mapping multimodal features to a high-dimensional semantic space and simulating semantic attraction and repulsion to construct a dynamic semantic field, deep semantic fusion and relation modeling of multimodal features can be performed. Semantic flow paths can be identified within the multimodal semantic field, energy differences can be calculated, and the translation region can be divided into a first translation region and a second translation region. Data can be dynamically split based on semantic complexity. Different translation models are used for different regions; a simple translation model is used for the first translation region, while semantic decoding and stitching can handle complex translation scenarios for the second translation region, ensuring the integrity and logical rationality of the translation results. Through multimodal semantic fields and collaborative translation, subtitle and audio information can be effectively integrated, improving the accuracy and consistency of the translation results. Dynamic splitting and handling of difficulties can ensure the naturalness of translation in complex regions, improving the logical consistency and efficiency of the translation results. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the multimodal video subtitle and audio collaborative translation and dynamic splitting method in the embodiments of this application;

[0051] Figure 2 This is a schematic diagram of the attraction distribution map in the embodiments of this application;

[0052] Figure 3 This is a schematic diagram of the first flow path in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the structure of the multimodal video subtitle and audio collaborative translation and dynamic splitting system in the embodiments of this application. Detailed Implementation

[0054] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0055] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] Hereinafter, the terms "first," "second," and other generic terms are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0057] refer to Figure 1 The diagram illustrates a specific implementation of the multimodal video subtitle and audio collaborative translation and dynamic splitting method of this application, including:

[0058] S101. Extract features from the pre-acquired multimodal video, map the features to a high-dimensional semantic space, simulate the semantic attraction and semantic repulsion between feature points of different modalities, and construct a multimodal semantic field.

[0059] S102. Identify a first flow path with semantic flow in the multimodal semantic field, calculate the semantic flow energy difference in the first flow path, and select regions with semantic flow energy difference less than or equal to a preset energy difference threshold as the first translation region, and regions with semantic flow energy difference greater than the preset energy difference threshold as the second translation region.

[0060] S103. Identify the first translation path of the first translation region through the preset first translation model, and perform collaborative translation of subtitles and audio to obtain the first translation result;

[0061] S104. Identify the second translation path of the second translation region through the preset second translation model, combine the second translation path and the corresponding first flow path to screen out translation difficulties, perform semantic decoding and semantic stitching on the translation difficulties, and obtain the second translation result.

[0062] S105. Combining the first translation result and the second translation result, a collaborative translation result is obtained.

[0063] In the multimedia era, video has become the core carrier of information dissemination, typically containing two key modalities: subtitles and audio. Existing video translation technologies mainly rely on machine translation of text subtitles or speech recognition followed by translation of audio. This approach severs the inherent connections between modalities and, when integrating multimodal information, often simply splices together features. It cannot handle complex video content with varying context, cultural background, or inconsistencies, resulting in low-quality translations and logical inconsistencies.

[0064] In this embodiment, feature extraction is performed on the pre-acquired multimodal video. Feature sequences are extracted from the subtitle text and audio waveform of the video, respectively. The features are mapped into a high-dimensional semantic space to simulate the semantic attraction and repulsion between feature points of different modalities, thus constructing a multimodal semantic field. By constructing a multimodal semantic field, combining the semantic relationship between subtitles and audio, and quantifying the mutual reinforcement or contradiction between modalities, a unified and accurate semantic basis is provided for the translation process. This avoids the shortcomings of existing technologies that simply splice features while ignoring deep semantic interactions. It provides a data foundation for accurately distinguishing translation difficulty areas and adopting differentiated translation strategies, thereby improving the accuracy of the translation results.

[0065] Specifically, the gradient distribution of the semantic field is analyzed in the multimodal semantic field to identify the first flow path with semantic flow, the semantic flow energy difference in the first flow path is calculated, the overall difficulty of understanding and translating the content of the path is quantified, an energy difference threshold is set according to the translation accuracy requirements, and regions with semantic flow energy differences less than or equal to the preset energy difference threshold are selected as the first translation region, i.e., simple region, and regions with semantic flow energy differences greater than the preset energy difference threshold are selected as the second translation region, i.e. complex region.

[0066] It should be noted that by calculating the semantic flow energy difference and performing dynamic region splitting, translation resources can be dynamically configured. Based on the semantic flow energy difference, translation difficulties in video content can be automatically identified, and computational resources can be concentrated on processing complex and error-prone second translation regions, avoiding over-processing of the first translation region, thus improving overall translation efficiency. Targeted translation methods can be adopted for different regions to improve the quality of translation results.

[0067] For the first translation region, a pre-set first translation model is used to identify the first translation path based on the semantic correlation between nodes. Subtitles and audio are then translated collaboratively to obtain the first translation result. The first translation model can quickly process simple regions and achieve collaboration between subtitles and audio during the translation process, which can speed up the translation and improve the real-time performance of the translation results. Collaboration between subtitles and audio can also improve the accuracy and naturalness of the translation in simple regions. It can more accurately translate pronouns and judge the tone of sentences, avoiding ambiguity or incoherence caused by relying solely on text, thus improving the accuracy and quality of the first translation result.

[0068] For the second translation region, the second translation path of the second translation region is identified by the preset second translation model. Combining the second translation path and the corresponding first flow path, the path curvature and regional feature dispersion are calculated. The severity of logical transitions and the degree of semantic confusion are analyzed. Semantic breakpoints and logical conflict points are selected as translation difficulties. Semantic decoding and semantic stitching are performed on the translation difficulties to obtain the second translation result.

[0069] It is important to emphasize that employing targeted difficulty identification, decoding, and stitching mechanisms for complex regions can improve the robustness and translation quality when handling complex content. By locating semantic breakpoints and logical conflict points, translation difficulties can be addressed in a targeted manner. Through causal analysis and deep decoding, difficult content can be analyzed to optimize translation results and generate translation fragments that are closer to the original meaning. Semantic stitching can effectively solve the problems of awkward translations and illogical statements in complex translations, and improve the fluency and accuracy of translation results in extreme scenarios with cultural barriers, logical jumps, or severe modal information conflicts.

[0070] Specifically, combining the first and second translation results, the results are arranged according to their temporal position in the original video. A neural network model pre-trained with a large amount of historical translation data is used to optimize the language at the splicing point. This includes, but is not limited to, checking and ensuring clear pronoun references between adjacent sentences, appropriate use of conjunctions, consistency of tense, and uniformity of overall tone and style. This eliminates logical discontinuities caused by separate processing, resulting in a collaborative translation. Through the integration and optimization of the translation results, it is ensured that the two separately processed parts form a logically coherent, linguistically fluent, and stylistically consistent whole after merging, improving the logical consistency and quality of the collaborative translation result.

[0071] This application maps multimodal features to a high-dimensional semantic space, simulating semantic attraction and repulsion to construct a dynamic semantic field. This allows for deep semantic fusion and relation modeling of multimodal features, identifying semantic flow paths within the multimodal semantic field, calculating energy differences, and dividing the translation region into a first and second translation region. Data can be dynamically split based on semantic complexity. Different translation models are employed for different regions: a simple model for the first translation region, and semantic decoding and stitching for the second translation region to handle complex translation scenarios, ensuring the integrity and logical rationality of the translation results. Through the multimodal semantic field and collaborative translation, subtitle and audio information can be effectively integrated, improving the accuracy and consistency of the translation results. Dynamic splitting and handling of difficult points ensure the naturalness of translations in complex regions, enhancing the logical consistency and efficiency of the translation results.

[0072] Furthermore, feature extraction is performed on the pre-acquired multimodal videos, and the features are mapped into a high-dimensional semantic space to simulate the semantic attraction and repulsion between feature points of different modalities, thus constructing a multimodal semantic field, including:

[0073] S201. Extract features from the pre-acquired multimodal video to obtain text feature sequences and audio feature sequences. Map and project the text feature sequences and audio feature sequences into a high-dimensional semantic space to construct a feature point cloud.

[0074] S202. Calculate the semantic recognition confidence and cross-modal semantic consistency of each feature point in the feature point cloud to obtain the semantic energy value and construct the semantic point cloud.

[0075] S203. In the semantic point cloud, simulate the semantic attraction and semantic repulsion between feature points of different modalities to construct a multimodal semantic field.

[0076] In this embodiment, feature extraction is performed on the pre-acquired multimodal video. For the text modality, a pre-trained BERT encoder model is used to process the subtitle text, obtaining the context-related feature vectors of each word or sentence to form a text feature sequence. For the audio modality, a pre-trained audio neural network model, including but not limited to the Wav2Vec 2.0 model, is used to extract acoustic feature vectors from the audio waveform to form an audio feature sequence. The text feature sequence and the audio feature sequence are linearly transformed and projected into a shared, higher-dimensional semantic space. In the higher-dimensional semantic space, semantically similar concepts, whether from text or audio, have similar feature vectors. The projected feature vectors are used as feature point clouds.

[0077] It should be noted that by mapping text and audio features to a unified high-dimensional semantic space and constructing a feature point cloud, multimodal information can be initially aligned and fused in geometric space. This can break down the data barriers between text and audio modalities, provide accurate data support for analyzing cross-modal semantic relationships, quantify the distance and direction information between features, and provide a data foundation for simulating interactions between points.

[0078] Specifically, the semantic recognition confidence and cross-modal semantic consistency of each feature point in the feature point cloud are calculated. A linear regression model pre-trained using a large amount of historical feature point cloud data is used to analyze the degree of certainty in the semantic content judgment corresponding to the feature point, and the semantic recognition confidence is calculated. For each feature point, the feature points corresponding to the time are selected from another modality sequence, and the cosine similarity between the feature vectors is calculated as the cross-modal semantic consistency. The mean of the semantic recognition confidence and the cross-modal semantic consistency is calculated to obtain the semantic energy value. Based on the semantic energy value of each feature point, the semantic point cloud is constructed.

[0079] It is important to emphasize that by calculating the semantic recognition confidence and cross-modal semantic consistency of each feature point, a semantic energy value is obtained through fusion. The feature point cloud is then assigned a corresponding energy value to obtain a semantic point cloud. This allows for the quantitative analysis of the importance and reliability of different feature points. High-energy points reflect clear semantics that are mutually corroborated by multiple modalities, while low-energy points reflect semantic ambiguity or intermodal conflicts. This provides accurate data support for distinguishing key semantic regions and potential conflict regions, thereby improving the ability to perceive the quality of multimodal information.

[0080] Specifically, in semantic point clouds, the semantic attraction and repulsion between feature points of different modalities are simulated, and the combined effect of semantic attraction and repulsion is calculated to construct a multimodal semantic field. By constructing a multimodal semantic field, the cooperative and conflict relationships between modalities can be dynamically reflected, and potential paths and semantic obstacle regions of semantic flow can be reflected. This provides accurate data support for path recognition and region segmentation, enhances the understanding of the deep semantic structure of video content, effectively handles complex scenarios such as inconsistencies in modal information and logical transitions, and improves the accuracy of semantic analysis results.

[0081] Furthermore, in the semantic point cloud, the semantic attraction and repulsion between feature points of different modalities are simulated to construct a multimodal semantic field, including:

[0082] S301. In the semantic point cloud, calculate the similarity between the semantic features of semantic points and construct a similarity matrix;

[0083] S302. In the feature point cloud, count the co-occurrence frequency of features of different modalities and calculate the corresponding semantic association degree to construct a semantic association matrix;

[0084] S303. Combining the similarity matrix and semantic association matrix, and using a preset attraction analysis model, simulate and calculate the semantic attraction between feature points of different modalities, and construct an attraction distribution map;

[0085] S304. Select feature point pairs whose distance is less than a preset distance threshold and whose semantic relevance is less than a preset relevance threshold in the attraction distribution map, and use them as semantic conflict point pairs. Calculate the semantic repulsion of the semantic conflict point pairs.

[0086] S305. Calculate the combined effect of semantic attraction and semantic repulsion to construct a multimodal semantic field.

[0087] In this embodiment, each point in the semantic point cloud is traversed, and each point is compared pairwise with all other points in the point cloud. The cosine similarity between the two feature vectors is calculated to obtain the similarity between the semantic features of the semantic points. The points are then arranged according to the calculated similarity to construct a similarity matrix. By calculating the cosine similarity between all pairs of semantic points and constructing a similarity matrix, the semantic relationships within the point cloud can be quantified, providing accurate data support for analyzing the interactions between feature points.

[0088] Specifically, in the feature point cloud, the co-occurrence frequency of different modal features is statistically analyzed and the corresponding semantic correlation is calculated. A time window is set according to the accuracy requirements of semantic analysis and data translation. Based on the original video timestamps corresponding to each point in the feature point cloud, the frequency of co-occurrence of a text modal feature point and an audio modal feature point is statistically analyzed within the time window. The higher the frequency, the closer the correlation between the two points in the original data. By using a neural network model pre-trained with a large amount of historical feature point cloud data, each pair of text and audio feature vectors is analyzed. The model learns the deep correlation between text and audio feature vectors, calculates the correlation score, and reflects the degree of matching between the co-occurrence frequency and the feature itself. The semantic correlation matrix is ​​constructed according to the calculated correlation scores.

[0089] It should be noted that by combining co-occurrence frequency statistics and semantic relevance to construct a semantic association matrix, the temporal co-occurrence and semantic dependency relationships of multimodal data can be effectively analyzed. This can make up for the shortcomings of simple semantic similarity calculation, and can identify feature point pairs that are not highly similar in semantic content but have strong correlation in specific scenarios. This provides accurate data support for analyzing the interaction between modalities and avoids the situation where they are incorrectly judged as unrelated simply because of low semantic similarity.

[0090] like Figure 2As shown in the figure, the target point represents each feature point in the feature point cloud, and the value of the connecting edge represents the semantic attraction of the target point to other feature points. Combining the similarity matrix and semantic association matrix, the semantic attraction between feature points of different modalities is simulated and calculated through a preset attraction analysis model. The attraction analysis model includes, but is not limited to, a learnable weighted fusion model. The model is trained and optimized based on the input semantic similarity and semantic association, and the semantic similarity and semantic association are weighted and fused according to the optimized weights to calculate the corresponding semantic attraction. For each pair of feature points in the feature point cloud, the semantic attraction is calculated, and the semantic attraction value is added to the corresponding point pair to obtain an attraction distribution map, reflecting the intensity distribution of the attraction of any point in space to other points.

[0091] It is important to emphasize that by integrating similarity and relevance information to calculate semantic attractiveness and construct an attractiveness distribution map, we can avoid the limitations of a single indicator, improve the accuracy of attractiveness calculation results, reflect the complexity of multimodal information interaction, improve the accuracy and rationality of the constructed attractiveness field, and truly reflect the synergistic effect of mutual reinforcement and complementarity between modalities.

[0092] Specifically, in the semantic space, if two feature points from different modalities are close to each other, it reflects that they are expected to express consistent semantics. If their calculated semantic correlation is lower than another preset correlation threshold, it reflects semantic inconsistency, indicating a semantic conflict between the two feature points. Distance and correlation thresholds are set according to the requirements of semantic analysis accuracy and translation accuracy. All cross-modal point pairs are traversed, and feature point pairs with a distance less than the preset distance threshold and a semantic correlation less than the preset correlation threshold in the attraction distribution map are selected as semantic conflict point pairs. The semantic repulsion is obtained by calculating the reciprocal of the sum of semantic correlation and distance. The lower the correlation and the closer the distance, the greater the corresponding repulsion force.

[0093] It should be noted that by filtering semantic conflict point pairs through distance and relevance and calculating the corresponding semantic repulsion, inconsistencies in multimodal information can be quantified, and semantic conflicts that are easily overlooked but affect translation quality can be identified. By accurately identifying semantic conflict points and calculating the corresponding repulsion, accurate data support is provided for analyzing adversarial relationships between modalities in a multimodal semantic field. This ensures that the constructed multimodal semantic field can comprehensively reflect semantic collaboration and conflict, and provides accurate data support for identifying translation difficulties.

[0094] For any target point in the semantic point cloud, based on the attraction distribution map, the attraction forces of points that attract the target point are vector-synthesized to obtain the total attraction force; the repulsive forces of semantic conflict point pairs that repel the target point are vector-synthesized to obtain the total repulsive force; the total attraction force and the total repulsive force are vector-summed to obtain the resultant force on the target point at that location; by traversing every point in the point cloud, the resultant force of each target point is calculated, and a multimodal semantic field is constructed based on the resultant force. Constructing a multimodal semantic field by vector-synthesizing the resultant force of semantic attraction and repulsion can accurately reflect the complex interactions between multiple modalities, the potential direction of semantic information flow and evolution in space, and the areas of obstruction. This provides accurate data support for identifying semantic flow paths and dividing translation regions, improving the accuracy and quality of translation results.

[0095] Furthermore, identifying a first flow path with semantic flow in the multimodal semantic field and calculating the semantic flow energy difference in the first flow path includes:

[0096] S401. Analyze the direction of semantic propagation in a multimodal semantic field and identify the first flow path where semantic flow exists;

[0097] S402. Extract the energy value sequence along the first flow path, select the energy maxima as semantic obstacle points, calculate the energy consumption of each semantic obstacle point, and sum them to obtain the semantic flow energy difference of the first flow path.

[0098] In this embodiment, the direction of semantic propagation is analyzed in the multimodal semantic field to identify semantic start nodes, semantic turning points, and semantic termination nodes, and to filter out the first flow path with semantic flow. Through semantic node identification and the first flow path filtering, the core semantic context in the video content can be automatically extracted. By analyzing the semantic flow in the multimodal semantic field, the smoothness of semantic flow can be evaluated based on semantic logic, and the filtered first flow path can accurately reflect the true semantic structure of the video content.

[0099] Specifically, based on each identified first flow path, the semantic energy value corresponding to each location point along the path is extracted to obtain an energy value sequence distributed along the path. Local energy maxima points in the energy value sequence are identified as semantic obstacle points, reflecting areas where there are potential conflicts in technical terms, puns, or modal information. The energy difference between adjacent energy points before and after the semantic obstacle point is calculated to obtain the energy consumption for crossing the corresponding semantic obstacle point. The energy consumption of all semantic obstacle points on a first flow path is summed to obtain the semantic flow energy difference of the path, reflecting the total resistance that needs to be overcome to understand and translate the content of the entire path.

[0100] It should be noted that by extracting the path energy sequence, identifying semantic obstacle points, and calculating the cumulative energy consumption to obtain the semantic flow energy difference, the translation complexity of different semantic segments in the video can be quantitatively analyzed. The semantic flow energy difference reflects the difficulty of translating the content of that path. The higher the difference, the greater the difficulty. This provides accurate data support for dynamic data distribution, and allows for on-demand allocation of translation resources and matching of corresponding translation strategies, thereby improving translation efficiency and resource utilization.

[0101] Furthermore, the semantic propagation direction is analyzed in the multimodal semantic field to identify the first flow path where semantic flow exists, including:

[0102] S501. By using a pre-defined semantic node recognition model, analyze the direction of semantic propagation in the multimodal semantic field, identify semantic start nodes, semantic turning points and semantic termination nodes, and construct the first node set.

[0103] S502. Starting from each semantic start node in the first node set, analyze the semantic propagation direction, select the corresponding semantic turning point nodes, select semantic termination nodes according to the semantic turning point nodes, and take each path connecting the semantic start node, semantic turning point node and semantic termination node as the first flow path.

[0104] In this embodiment, a pre-defined semantic node recognition model is used to analyze the direction of semantic propagation in a multimodal semantic field, identifying semantic start nodes, semantic transition nodes, and semantic termination nodes. The semantic node recognition model includes, but is not limited to, a graph neural network model pre-trained using a large amount of historical data. The model analyzes the vector features of the multimodal semantic field, identifying the source point of the multimodal semantic field vector as the semantic start node. The field vector direction of the area surrounding the semantic start node points towards or away from that point, indicating the beginning of a new semantic unit. The location where the field vector direction changes drastically is identified as the semantic transition node. The curvature of the semantic transition node exhibits a local extremum, corresponding to the transformation of logical relationships, the switching of topics, or the sudden change in emotion. The divergence point of the field vector is identified as the semantic termination node, reflecting that the vector field tends to be static or spreads out in all directions at this point, indicating the end of a semantic unit. The model identifies semantic start nodes, semantic transition nodes, and semantic termination nodes in the multimodal semantic field, constructing a first node set.

[0105] It should be noted that by using a semantic node recognition model to extract and classify key features of the multimodal semantic field, the global vector field can be transformed into a specific set of nodes. This simplifies the path generation process, improves the robustness and depth of the semantic analysis process, and can filter out deeper semantic structure change points determined by multimodal interactions, thereby improving the accuracy of the semantic analysis process.

[0106] like Figure 3As shown, using a path tracing algorithm, each semantic starting node in the first node set is used as the starting point. Integration is performed along the direction of the resultant force vector of the semantic field at that point to generate a streamline. When the streamline reaches the vicinity of a semantic turning node, the semantic turning node is included in the current path. The vicinity can be set according to the semantic analysis accuracy. The streamline is then used as the new starting point and integration continues along the direction of the field force. This process of integration and inclusion of semantic turning nodes continues until the streamline reaches the vicinity of a semantic ending node. This results in a first flow path that starts from the starting node, passes through one or more turning nodes, and finally reaches the ending node. By traversing all starting nodes and performing the path generation process, a set of first flow paths covering the main semantic flow is obtained.

[0107] It is important to emphasize that by connecting key semantic nodes through a path tracing algorithm based on field force direction integration to form the first flow path, the generated first flow path is not a simple linear connection. It conforms to the semantic laws of the semantic field, improves the logic and rationality of the first flow path, and can accurately reflect the actual propagation trajectory of semantic information in multimodal environments. This provides an accurate semantic structure basis for the semantic analysis process and improves the accuracy and efficiency of the translation processing.

[0108] Furthermore, by identifying the first translation path of the first translation region through a preset first translation model, the subtitles and audio are collaboratively translated to obtain the first translation result, including:

[0109] S601. Identify the first translation path with the highest semantic correlation among nodes in the first translation region using a preset first translation model;

[0110] S602. Following the first translation path, perform collaborative translation of the subtitles and audio to obtain the first translation result.

[0111] In this embodiment, for the semantic nodes and their connections within the first translation region, the edge weights reflect the semantic correlation between nodes. A first translation model is used to identify the first translation path with the highest semantic correlation between nodes in the first translation region. The first translation model includes, but is not limited to, a pre-trained dynamic programming model. The semantic start node in the region is taken as the path start point, the semantic end node is taken as the path end point, and the reciprocal of the semantic correlation between nodes is taken as the path cost. The path with the lowest cumulative cost from the start point to the end point is searched and taken as the first translation path with the highest semantic correlation between nodes.

[0112] It should be noted that by searching for the path with the highest semantic relevance of nodes, the optimal sequence of information processing is provided for the collaborative translation process. This ensures that when the first translation model is processing simple regions, it can always follow the path with the most natural semantic connection and the fewest logical jumps. This can reduce the probability of ambiguity or errors caused by incoherent context information during the translation process. By prioritizing the translation process of the core semantic flow, the accuracy and fluency of the translation results can be improved.

[0113] Specifically, following the first translation path, a pre-trained encoder-decoder translation model collaboratively translates the subtitles and audio. The model sequentially reads the text and audio features associated with each node according to the node order of the first translation path. In the encoding stage, the model adopts a cross-modal attention mechanism, using text features as queries and corresponding audio features as keys and values. By calculating attention weights, the model dynamically and selectively extracts the sentiment, tone, or supplementary information most relevant to the current text from the audio stream and integrates this information into the encoded representation of the text features to obtain a context vector that includes multimodal information. In the decoding stage, the decoder generates a word sequence of the target language based on the context vector. This process is repeated until the entire path is processed. The translation results of all nodes are concatenated according to the path order to obtain the first translation result.

[0114] It is important to emphasize that by using a cross-modal attention mechanism to co-encode and translate subtitles and audio under the constraints of the first translation path, the accuracy and coherence of translation results in simple regions can be improved while ensuring processing speed. The translation results can be combined with the literal meaning of the text and the paralinguistic information of the audio modality to improve the accuracy and quality of the translation results.

[0115] Furthermore, a second translation path is identified in the second translation region using a pre-defined second translation model. The translation difficulties are then identified by combining the second translation path with the corresponding first flow path. Semantic decoding and semantic stitching are performed on these translation difficulties to obtain the second translation result, including:

[0116] S701. Identify the second translation path of the second translation region through the preset second translation model, and identify semantic breakpoints and logical conflict points by combining the second translation path and the corresponding first flow path, and construct a set of translation difficulties.

[0117] S702. Semantically decode each translation difficulty, generate a set of translation segments, select semantically complete and logically reasonable translation segments for semantic stitching, and obtain the second translation result.

[0118] In this embodiment, a second translation path for the second translation region is identified using a preset second translation model. Semantic breakpoints and logical conflict points are identified by combining the second translation path with the corresponding first flow path. The contextual coherence at the location of the semantic breakpoint drops sharply, while the location of the logical conflict point suffers from problems such as multimodal information confusion and inconsistent logical orientation, thus constructing a set of translation difficulties. By comparing the second translation path with the first flow path and filtering semantic breakpoints and logical conflict points, potential fault points within the complex translation region can be located, providing accurate location references for the repair process. This avoids resource waste and poor results caused by uniform processing of complex regions, thereby improving the efficiency and accuracy of the translation process.

[0119] For each difficulty in the set of translation challenges, the local semantic logic relationship of the difficulty is analyzed, the confusion factors that lead to semantic confusion or logical interruption are identified, and corresponding interference blocking factors are generated. Based on the interference blocking factors, semantic decoding is performed to generate a set of translation fragments. The semantically complete and logically reasonable translation fragments are selected and semantically stitched together to obtain the second translation result. Through semantic decoding, fragment selection, and semantic stitching under causal intervention, the most appropriate and fluent translation solution can be generated and selected for the specific problems of each difficulty. This improves the translation quality when dealing with complex scenarios such as culturally loaded words, complex logical long sentences, and multimodal information conflicts, thereby improving the accuracy, coherence, and logic of the translation results.

[0120] Furthermore, a second translation path is identified in the second translation region using a pre-defined second translation model. Semantic breakpoints and logical conflict points are identified by combining the second translation path with the corresponding first flow path, thus constructing a set of translation difficulties, including:

[0121] S801. Identify the second translation path with the highest semantic correlation among nodes in the second translation region using a preset second translation model;

[0122] S802. Combining the second translation path and the corresponding first flow path, calculate the path curvature and filter out the region that is greater than the preset path curvature threshold as the logical jump region.

[0123] S803. Using a preset semantic interruption detection model, identify the semantic interruption location in the logical jump region as the semantic breakpoint.

[0124] S804. Based on the semantic breakpoint, calculate the feature dispersion and select the positions where the feature dispersion is greater than a preset dispersion threshold as logical conflict points.

[0125] S805. Combining the semantic breakpoints and logical conflict points, construct a set of translation difficulties.

[0126] In this embodiment, a pre-defined second translation model is used to identify the second translation path with the highest semantic relevance among nodes in the second translation region. The second translation model includes, but is not limited to, the A* algorithm. It uses the semantic start node within the second translation region as the starting point and the semantic end node as the ending point, and uses the reciprocal of the semantic relevance between nodes as the edge weights to search for the path with the lowest cumulative weight from the start point to the end point. This path is then considered the second translation path with the highest semantic relevance. By searching for the second translation path within complex regions, an accurate range is provided for difficult point identification, improving the efficiency and accuracy of the difficult point identification process.

[0127] Specifically, combining the second translation path and the corresponding first flow path, for each point on the path, the path curvature is obtained by calculating the rate of change of the tangent direction at that point. A path curvature threshold is set based on semantic analysis and translation accuracy requirements. The calculated curvature value is compared with the preset path curvature threshold, and nodes with curvature values ​​greater than the threshold are selected. The region formed by these nodes is designated as a logical jump region. By calculating and comparing path curvature to identify logical jump regions, logically incoherent regions in the semantic flow can be quickly filtered out. This allows for rapid processing of logically jump regions with abrupt structural changes, improving the efficiency and accuracy of the translation process.

[0128] Furthermore, a pre-defined semantic interruption detection model is used to identify semantic interruption locations within logical jump regions. This model includes, but is not limited to, recurrent neural network models pre-trained using extensive historical data. Based on learned semantic coherence patterns, the model analyzes the information flow and dependencies between the preceding and following contexts at each location within the logical jump region, calculates the probability of a semantic interruption at each location, and identifies semantic breakpoints. By identifying and locating semantic breakpoints within logical jump regions, corresponding semantic repair can be performed, improving the efficiency and accuracy of the semantic decoding and stitching process, enhancing the targeting of semantic repair, and improving the quality of the translation results.

[0129] Specifically, for each semantic breakpoint, a temporal neighborhood is set according to the semantic analysis accuracy requirements. Feature vectors of different modalities within the temporal neighborhood are calculated. The feature dispersion is obtained by calculating the average cosine distance of the feature vectors relative to the center point. A dispersion threshold is set according to the semantic analysis accuracy requirements, and positions with feature dispersion greater than the preset threshold are selected as logical conflict points. At these logical conflict points, the semantic information conveyed by different modalities is significantly inconsistent or contradictory. The selected semantic breakpoints and logical conflict points are combined to form a set of translation difficulties. By calculating feature dispersion to identify logical conflict points, logical conflict points of information conflict between modalities in multimodal translation can be effectively screened. Correcting logical conflict points ensures that the translated sentences are fluent and accurately reflect the semantic intent after multimodal information integration, improving the accuracy and rationality of the translation results.

[0130] Furthermore, semantic decoding is performed on each translation difficulty to generate a set of translation fragments. Semantically complete and logically sound translation fragments are then selected for semantic stitching to obtain the second translation result, including:

[0131] S901. For each translation difficulty, the semantic logic relationship is analyzed through a pre-set causal intervention model to generate interference blocking factors;

[0132] S902. Based on the interference blocking factor, a set of translated segments is generated using a preset semantic decoder model;

[0133] S903. Select semantically complete and logically reasonable translation segments from the set of translation segments, perform semantic stitching, and obtain the second translation result.

[0134] In this embodiment, for each translation difficulty, a pre-defined causal intervention model analyzes the semantic logic relationship. This model performs causal analysis on the local context of the difficulty, identifying potential causal relationships between key semantic variables. These key semantic variables include, but are not limited to, subject, action, object, and emotional polarity. A causal graph is constructed, and confounding variables causing multimodal information conflict or logical breakage are located. The model fixes the value of these confounding variables, cutting off interference paths to external semantic reasoning and calculating interference blocking factors. By identifying confounding variables and generating interference blocking factors, noise signals causing problems can be quickly extracted, improving the accuracy of the generated translation segments.

[0135] Specifically, based on interference blocking factors, a set of translated segments is generated through a pre-defined semantic decoder model. This semantic decoder model includes, but is not limited to, a large model based on the Transformer architecture pre-trained using extensive historical data. The model blocks interference sources according to the interference blocking factors, generating multiple candidate translated segments to obtain the translated segment set. Generating translated segments through interference blocking can improve the accuracy, semantic integrity, and logical coherence of the translated segments, thereby enhancing the quality of the translation results.

[0136] Specifically, semantically complete and logically sound translation segments are selected from the set of translated segments. A quality evaluation function scores each candidate segment, analyzing semantic completeness, logical coherence, and grammatical correctness, selecting the segment with the highest total score. A pre-trained text repair model based on a cross-attention mechanism is then used to semantically stitch the selected segments. This model learns and mimics natural language transition patterns, fine-tuning the boundaries of the translated segments and rewriting a few adjacent words and phrases to ensure the segments seamlessly integrate into the existing translation, resulting in a second translation. This semantic stitching avoids the problem of disfluency in the repaired translation, improving the overall quality of the translation.

[0137] like Figure 4 As shown, a multimodal video subtitle and audio collaborative translation and dynamic splitting system is used to implement a multimodal video subtitle and audio collaborative translation and dynamic splitting method, including:

[0138] The multimodal semantic field construction module extracts features from the pre-acquired multimodal video, maps the features to a high-dimensional semantic space, simulates the semantic attraction and repulsion between feature points of different modalities, and constructs a multimodal semantic field.

[0139] The dynamic diversion module identifies a first flow path with semantic flow in the multimodal semantic field, calculates the semantic flow energy difference in the first flow path, and filters out regions with semantic flow energy differences less than or equal to a preset energy difference threshold as first translation regions, and regions with semantic flow energy differences greater than the preset energy difference threshold as second translation regions.

[0140] The first translation module identifies the first translation path of the first translation region through a preset first translation model, and performs collaborative translation of subtitles and audio to obtain the first translation result.

[0141] The second translation module identifies the second translation path of the second translation region through a preset second translation model, combines the second translation path with the corresponding first flow path to filter out translation difficulties, performs semantic decoding and semantic stitching on the translation difficulties, and obtains the second translation result.

[0142] The translation result integration module combines the first translation result and the second translation result to obtain a collaborative translation result.

[0143] In this embodiment, the multimodal semantic field construction module maps text and audio features to a shared high-dimensional semantic space, simulating the attraction and repulsion of features between modalities to construct a vector semantic field. This, combined with the inherent correlations of multimodal information, builds a unified computational basis, avoiding the defects of isolated or simply spliced ​​modal information, and providing accurate data support for the translation process. The dynamic diversion module identifies the energy consumption differences along the semantic flow path and computational path, dividing the video content into a first translation region and a second translation region. This intelligently allocates translation resources on demand, avoiding uniform data processing and focusing computational resources on semantic difficulties, thus improving the processing efficiency and accuracy of the translation results.

[0144] Specifically, the first translation module quickly determines the optimal translation path and collaboratively integrates subtitles and audio information to generate a preliminary translation result for that region. While ensuring translation speed, the cross-modal collaborative mechanism effectively improves the accuracy and naturalness of the basic translation, thus enhancing the quality of the final translation. The second translation module identifies translation difficulties such as semantic breaks and logical conflicts, and repairs and reconstructs these difficulties through causal intervention and deep decoding, generating corresponding translation results. This improves the quality of the translation results and avoids performance degradation when dealing with logical jumps and culturally loaded words. The translation result integration module seamlessly integrates and optimizes the coherence of the first and second translation results, ensuring a complete, fluent, and stylistically consistent overall translation. This eliminates the discontinuity issues caused by segmented processing and further improves the quality of the final translation.

[0145] The above description is merely a preferred embodiment of this application. The scope of protection of this application is not limited to the above embodiments. All technical solutions falling within the scope of this application's concept are within the scope of protection of this application. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of this application should also be considered within the scope of protection of this application.

Claims

1. A multi-modal video subtitle and audio collaborative translation and dynamic offloading method, characterized in that, The method comprises the following steps: feature extraction is performed on the pre-acquired multi-modal video, the features are mapped into a high-dimensional semantic space, semantic attraction and semantic repulsion between different modal feature points are simulated, and a multi-modal semantic field is constructed; a first flow path with semantic flow in the multi-modal semantic field is identified, a semantic flow energy difference value in the first flow path is calculated, a region with a semantic flow energy difference value less than or equal to a preset energy difference threshold is selected as a first translation region, and a region with a semantic flow energy difference value greater than the preset energy difference threshold is selected as a second translation region; a first translation path of the first translation region is identified through a preset first translation model, and the subtitle and the audio are collaboratively translated to obtain a first translation result; a second translation path of the second translation region is identified through a preset second translation model, translation difficulties are screened out in combination of the second translation path and the corresponding first flow path, semantic decoding and semantic stitching are performed on the translation difficulties, and a second translation result is obtained; the collaborative translation result is obtained in combination of the first translation result and the second translation result; a first flow path with semantic flow in the multi-modal semantic field is identified, a semantic flow energy difference value in the first flow path is calculated, and the method comprises the following steps: a semantic starting node, a semantic turning node and a semantic termination node are identified by analyzing a semantic propagation direction in the multi-modal semantic field through a preset semantic node identification model, and a first node set is constructed; the semantic propagation direction is analyzed from each semantic starting node in the first node set respectively, a corresponding semantic turning node is screened out, a semantic termination node is screened out according to the semantic turning node, and each path connecting the semantic starting node, the semantic turning node and the semantic termination node is taken as a first flow path; an energy value sequence passed by the first flow path is extracted, an energy maximum value point is selected as a semantic obstacle point, an energy consumption of each semantic obstacle point is calculated, and a semantic flow energy difference value of the first flow path is obtained by superposition.

2. The multimodal video captioning and audio co-translation and dynamic offloading method of claim 1, wherein, The method comprises the following steps: feature extraction is performed on the pre-acquired multi-modal video, the features are mapped into a high-dimensional semantic space, semantic attraction and semantic repulsion between different modal feature points are simulated, and a multi-modal semantic field is constructed; text feature sequences and audio feature sequences are obtained through feature extraction on the pre-acquired multi-modal video, the text feature sequences and the audio feature sequences are mapped and projected into a high-dimensional semantic space, and a feature point cloud is constructed; semantic recognition confidence and cross-modal semantic consistency of each feature point in the feature point cloud are calculated to obtain a semantic energy value, and a semantic point cloud is constructed; 3. The multi-modal video captioning and audio co-translation and dynamic offloading method of claim 2, wherein, semantic attraction and semantic repulsion between different modal feature points in the semantic point cloud are simulated, and a multi-modal semantic field is constructed. The method comprises the following steps: in the semantic point cloud, the similarity between semantic features of semantic points is calculated to construct a similarity matrix; in the feature point cloud, the co-occurrence frequency of different modal features is counted and the corresponding semantic correlation degree is calculated to construct a semantic correlation matrix; The semantic attraction degree between different modal feature points is simulated and calculated by a preset attraction analysis model, and an attraction distribution map is constructed in combination with the similarity matrix and the semantic correlation matrix; Feature point pairs with a distance less than a preset distance threshold and a semantic correlation degree less than a preset correlation degree threshold in the attraction distribution map are screened out as semantic conflict point pairs, and the semantic repulsion degree of the semantic conflict point pairs is calculated; The resultant force of the semantic attraction degree and the semantic repulsion degree is calculated to construct a multi-modal semantic field.

4. The multi-modal video subtitle and audio co-translation and dynamic offloading method of claim 1, wherein, The first translation path of the first translation region is identified by a preset first translation model, and the subtitle and the audio are collaboratively translated to obtain a first translation result, including: The first translation path with the highest node semantic correlation degree in the first translation region is identified by a preset first translation model; The subtitle and the audio are collaboratively translated according to the first translation path to obtain a first translation result.

5. The multi-modal video subtitle and audio co-translation and dynamic offloading method of claim 1, wherein, The second translation path of the second translation region is identified by a preset second translation model, and the translation difficulty is screened out in combination with the second translation path and the corresponding first flow path, and the second translation result is obtained by semantic decoding and semantic stitching, including: The second translation path of the second translation region is identified by a preset second translation model, and the semantic break point and the logical conflict point are identified in combination with the second translation path and the corresponding first flow path to construct a translation difficulty set; Each translation difficulty is decoded respectively to generate a translation fragment set, and the translation fragment with complete semantics and reasonable logic is screened out for semantic stitching to obtain a second translation result.

6. The multi-modal video captioning and audio co-translation and dynamic offloading method of claim 5, wherein, The second translation path of the second translation region is identified by a preset second translation model, and the semantic break point and the logical conflict point are identified in combination with the second translation path and the corresponding first flow path to construct a translation difficulty set, including: The second translation path with the highest node semantic correlation degree in the second translation region is identified by a preset second translation model; In combination with the second translation path and the corresponding first flow path, the region with a path curvature greater than a preset path curvature threshold is screened out as a logical jump region; The semantic break point is identified as a semantic break point by a preset semantic interruption detection model in the logical jump region; According to the semantic break point, the feature dispersion degree is calculated and the position with a feature dispersion degree greater than a preset dispersion degree threshold is screened out as a logical conflict point; In combination with the semantic break point and the logical conflict point, a translation difficulty set is constructed.

7. The multi-modal video captioning and audio co-translation and dynamic offloading method of claim 5, wherein, Each translation difficulty is decoded respectively to generate a translation fragment set, and the translation fragment with complete semantics and reasonable logic is screened out for semantic stitching to obtain a second translation result, including: For each translation difficulty, the semantic logic relationship is analyzed by a preset causal intervention model to generate an interference blocking factor; Based on the interference blocking factor, a translation fragment set is generated by a preset semantic decoder model; The translation fragment with complete semantics and reasonable logic is screened out from the translation fragment set for semantic stitching to obtain a second translation result.

8. A multi-modal video caption and audio co-translation and dynamic offloading system, characterized in that, A multi-modal video subtitle and audio collaborative translation and dynamic shunting method is implemented, including: The multi-modal semantic field construction module extracts features from the pre-acquired multi-modal video, maps the features to a high-dimensional semantic space, simulates semantic attraction and semantic repulsion between different modal feature points, and constructs a multi-modal semantic field; The dynamic shunting module identifies a first flow path with semantic flow in the multi-modal semantic field, calculates a semantic flow energy difference value in the first flow path, screens out a region with a semantic flow energy difference value less than or equal to a preset energy difference threshold as a first translation region, and screens out a region with a semantic flow energy difference value greater than the preset energy difference threshold as a second translation region; The first translation module identifies a first translation path of the first translation region through a preset first translation model, cooperatively translates the subtitles and the audio, and obtains a first translation result; The second translation module identifies a second translation path of the second translation region through a preset second translation model, screens out a translation difficulty in combination with the second translation path and the corresponding first flow path, performs semantic decoding and semantic stitching on the translation difficulty, and obtains a second translation result; The translation result integration module combines the first translation result and the second translation result to obtain a cooperative translation result.

Images