Video processing method and system based on audio and picture synchronization, and electronic equipment

Abstract

The invention relates to a video processing method based on audio and picture synchronization, and the method comprises the steps: screening candidate segments from a video material library according to the content inputted by a user, determining a target segment from the candidate segments based on a deep interaction algorithm and semantic entity correlation analysis, determining the video time length of the target segment, and outputting the video time length of the target segment. Determining the line number of the target segment based on the large language model, calculating a target playing duration according to the line number and a preset dubbing speed, judging whether the target segment has a duration deviation based on the target playing duration and the video duration, and if the target segment has the duration deviation, playing the target segment based on the target playing duration and the video duration. And performing adaptive speed change processing on the target clip to generate a final video clip. According to the method and the device, the problem that the audio and the picture of the video are asynchronous is solved, the speed change algorithm is dynamically optimized based on the deviation degree of the target playing duration and the video duration, and the integrity of the picture and the audio is kept while the audio and picture synchronization is ensured.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to video processing methods, systems and electronic devices based on audio-visual synchronization. Background Technology

[0002] In fields such as short video production, AI video generation, and intelligent dubbing, achieving precise synchronization between audio and video is crucial for ensuring content quality.

[0003] In the process of material retrieval and matching, existing technologies mostly focus on the semantic relevance of the content, usually based on keywords or vector similarity, to retrieve video clips from the material library that semantically match a given line of dialogue. The retrieved clips often have a significant deviation in length from the target audio. At the temporal alignment level, existing solutions often simply change the overall speed of the video, such as using the regular speed-up function of tools like FFmpeg. This simple speed-up processing will result in a loss of video quality and affect the viewing experience. Summary of the Invention

[0004] This application provides a video processing method, system, electronic device, and storage medium based on audio-visual synchronization, to at least solve the problem of audio-visual asynchrony in related technologies.

[0005] In a first aspect, embodiments of this application provide a video processing method based on audio-visual synchronization, the method comprising: Based on user input, candidate segments are selected from the video material library, and the target segment is determined from the candidate segments based on deep interaction algorithms and semantic entity relevance analysis. The video length of the target segment is determined, and the number of words in the dialogue of the target segment is determined based on a large language model. The target playback length is calculated based on the number of words in the dialogue and a preset dubbing speed. Based on the target playback length and the video length, it is determined whether there is a duration deviation in the target segment. If the target segment has a duration deviation, the target segment is subjected to adaptive speed adjustment based on the target playback duration and the video duration to generate the final video segment.

[0006] In some embodiments, the step of performing adaptive speed-changing processing on the target segment based on the target playback duration and the video duration to generate the final video segment includes: Calculate the speed factor based on the video duration and the target playback duration; Based on the speed change factor and the preset speed change tolerance threshold, it is determined whether speed change is needed. If so, the target segment is speed-changed based on the speed change factor to obtain the final video segment.

[0007] In some embodiments, the step of performing speed-changing processing on the target segment based on the speed-changing factor to obtain the final video segment includes: If the speed factor does not exceed the preset speed range, the target segment is time-scaled according to the speed factor to obtain the final video segment. If the speed factor exceeds the preset speed range, the target segment is subjected to frame extraction or frame interpolation based on motion compensation or frame interpolation strategies to obtain the final video segment.

[0008] In some embodiments, the method further includes: Determine whether the absolute difference between the target segment duration after adaptive speed adjustment and the target playback duration is greater than a secondary verification threshold. If so, fine-tune the speed adjustment factor and, based on the fine-tuned speed adjustment factor, re-process the original target segment using speed adjustment; and / or Record the video duration, target playback duration, speed factor, speed processing time, and output path for each target segment.

[0009] In some embodiments, the step of filtering candidate segments from the video material library based on user input includes: The system parses and encodes user input to generate semantic vectors for the input screen and input text. Based on the semantic vector of the input image, the semantic vector of the input text, and the semantic vector library, a first candidate segment is determined from the video material library, wherein the semantic vector library is constructed based on the video material library; Perform multi-dimensional fusion scoring on the first candidate segment, and select a preset number of segments from the first candidate segment as the second candidate segment based on the multi-dimensional fusion scoring results.

[0010] In some embodiments, the semantic vector library includes a visual semantic vector library and a text semantic vector library; determining the first candidate segment from the video material library based on the input visual semantic vector, the input text semantic vector, and the semantic vector library includes: A structured query language is generated based on user input, and the structured query language is matched against an SQLite database to obtain a preliminary set of candidate fragments. The preliminary candidate fragment set is matched with the image semantic vector library and the text semantic vector library respectively to obtain image vector subsets and text vector subsets; Using FAISS, k-nearest neighbor search is performed on the subset of the input image vector and the subset of the input text vector based on the input image semantic vector and the input text semantic vector, and the first candidate segment is determined based on the search results.

[0011] In some embodiments, the multidimensional fusion scoring of the first candidate fragment includes: Based on the input image semantic vector and the image semantic vector of the first candidate segment, determine the image semantic score; Based on the input text semantic vector and the text semantic vector of the first candidate segment, the dialogue semantic score is determined; Based on a preset tag system, the tag similarity between the user input content and the first candidate segment is calculated to obtain a dynamic tag matching score; The image semantic score, dialogue semantic score, and dynamic tag matching score are linearly weighted and fused to obtain a multi-dimensional weighted score.

[0012] In some embodiments, determining the target segment from the candidate segments based on deep interaction algorithms and semantic entity relevance analysis includes: Based on the interactive encoder, the scene description and dialogue of the second candidate segment are deeply interacted with the user input content to obtain the fusion score of each second candidate segment; Extract the first core entity from the user input content and the second core entity from the second candidate fragment, calculate the semantic similarity between the first core entity and the second core entity, and obtain the object overlap score; Based on the fusion score and the object overlap score, a comprehensive score is determined for each of the second candidate segments, and a target segment is determined from the second candidate segments based on the comprehensive score.

[0013] Secondly, embodiments of this application provide a video processing system based on audio-visual synchronization, the system comprising: The segment selection module is used to filter candidate segments from the video material library based on user input, and determine the target segment from the candidate segments based on deep interaction algorithms and semantic entity relevance analysis. The deviation judgment module is used to determine the video duration of the target segment, determine the number of words in the dialogue of the target segment based on the large language model, calculate the target playback duration based on the number of words in the dialogue and the preset dubbing speed, and determine whether there is a duration deviation in the target segment based on the target playback duration and the video duration. The audio-visual alignment module is used to perform adaptive speed adjustment on the target segment based on the target playback duration and the video duration when the target segment has a duration deviation, so as to generate the final video segment.

[0014] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the video processing method based on audio-visual synchronization as described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video processing method based on audio-visual synchronization as described in the first aspect above.

[0016] Compared to related technologies, the video processing method based on audio-visual synchronization provided in this application, through deep interaction algorithms and semantic entity relevance analysis, not only matches surface keywords but also mines the potential needs of user input and the semantic entities of candidate segments, achieving deep alignment between needs and materials and reducing screening errors. By predicting the number of words in dialogue using a large language model and calculating the target playback duration using a preset speech rate, and comparing the deviation with the video duration, the method achieves accurate quantification of duration deviation, providing data support for subsequent processing. Based on the degree of deviation between the target playback duration and the video duration, the method dynamically optimizes the speed-changing algorithm, ensuring audio-visual synchronization while maximizing the preservation of the integrity of the video and audio, thus solving the problem of audio-visual asynchrony in video. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a video processing method based on audio-visual synchronization according to an embodiment of this application; Figure 2 This is a structural block diagram of a video processing system based on audio-visual synchronization according to an embodiment of this application; Figure 3 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0019] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0020] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0021] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] This embodiment provides a video processing method based on audio-visual synchronization. Figure 1 This is a flowchart of a video processing method based on audio-visual synchronization according to an embodiment of this application, such as... Figure 1As shown, the process includes the following steps: Step S101: Based on the user input, candidate segments are selected from the video material library, and the target segment is determined from the candidate segments based on deep interaction algorithms and semantic entity correlation analysis.

[0023] A hybrid retrieval architecture combining structured initial screening and multimodal semantic recall is adopted: First, based on user-input keywords, classification labels, and time range, keyword matching is used to perform efficient filtering on a pre-built SQLite database to generate an initial candidate set; then, a dual-channel vector index of image semantics and text semantics is constructed, and a pre-trained Bi-Encoder model is used to map natural language queries to the corresponding semantic spaces, and k-nearest neighbor recall is performed in parallel in FAISS to achieve high recall matching and obtain candidate segments.

[0024] For candidate segments in the initial screening and recall stages, a re-ranking score is generated through deep interaction of the cross encoder architecture model, taking into account both the semantic similarity of the scene and the semantic similarity of the dialogue. A semantic entity enhancement module is also introduced to perform fine-grained re-ranking and improve the relevance of the results.

[0025] In some embodiments, step S101, which involves filtering candidate segments from the video material library based on user input, includes: Step S1011: Parse and encode the user input content to generate the semantic vector of the input screen and the semantic vector of the input text.

[0026] The image semantic vector is generated by encoding the video's shot descriptions, image types, and various visual tags, while the text semantic vector comes from the video's dialogue or script. User input is parsed into two parts: image description query Q. v With dialogue search Q d A pre-trained bidirectional encoder model (Bi-Encoder), such as BAAI / bge-large-zh-v1.5, is used to encode the Q... v With Q d Encoded separately as normalized high-dimensional vectors: Input image semantic vector v q With the input text semantic vector d q .

[0027] Step S1012: Based on the semantic vector of the input image, the semantic vector of the input text, and the semantic vector library, determine the first candidate segment from the video material library, wherein the semantic vector library is constructed based on the video material library.

[0028] In some embodiments, the semantic vector library includes a screen semantic vector library and a text semantic vector library; step S1012 specifically includes: Step S201: Generate a structured query language based on user input, and match the structured query language against the SQLite database to obtain a preliminary candidate fragment set.

[0029] The system receives deterministic filtering conditions input by the user. In this embodiment, the user input includes, but is not limited to, keywords, duration ranges, and preset category tags. The category tags cover semantic dimensions such as image type and emotional atmosphere. A pre-query is performed on the backend metadata database (SQLite database) using Structured Query Language (SQL). This fully utilizes the database indexing mechanism to quickly filter out video segments that do not meet the basic conditions, generating a preliminary candidate segment set. This stage significantly narrows the candidate range for subsequent semantic calculations, providing a fundamental guarantee for overall system efficiency.

[0030] It should be noted that the keywords support Boolean logical expressions, such as "(A|B) AND C AND NOT D", which can be achieved by combining SQL operators such as LIKE, NOT LIKE, and OR, thereby providing users with flexible and efficient hard filtering capabilities.

[0031] Step S202: Match the preliminary candidate fragment set with the image semantic vector library and the text semantic vector library respectively to obtain the image vector subset and the text vector subset.

[0032] This embodiment maintains two independent high-dimensional vector index libraries, corresponding to the image semantics and text semantics of video clips, respectively.

[0033] Step S203: Using FAISS, k-nearest neighbor search is performed in the subset of image vectors and the subset of text vectors based on the input image semantic vector and the input text semantic vector, and the first candidate segment is determined based on the search results.

[0034] Semantic matching is performed on the initially screened candidate segments. FAISS (Facebook AISimilarity Search) is used to perform a k-nearest neighbor search in the corresponding vector index, efficiently recalling the top-K semantically most similar candidate segments based on Euclidean distance.

[0035] Traditional video filtering often relies on single-dimensional retrieval (such as based solely on text tags or solely on image features), which can easily lead to problems such as text matching but image mismatch, or similar images but semantically disconnected text. This embodiment constructs a dual-library architecture of "image semantic vector library + text semantic vector library," simultaneously matching from both visual and semantic dimensions to achieve audio-visual semantic alignment, thereby reducing matching bias at its source.

[0036] In existing technologies, structured queries can only achieve precise keyword matching and cannot handle fuzzy semantics or latent needs (e.g., if a user inputs "romantic seaside atmosphere," keyword matching alone cannot capture the emotional semantics of "romantic"); while simple semantic vector matching lacks the efficiency of structured filtering and is difficult to quickly narrow down the search scope. This solution uses a hierarchical logic of "structured query → preliminary filtering → dual-database semantic matching → precise search." First, it uses SQLite structured queries to quickly filter irrelevant fragments (reducing the amount of data for subsequent semantic matching), and then uses a semantic vector database to mine deep relationships, achieving a combination of "precise filtering + fuzzy semantic matching."

[0037] Furthermore, this embodiment introduces the FAISS (Facebook AI Similarity Search) toolkit to perform k-nearest neighbor search on the "image vector subset + text vector subset". By utilizing FAISS's efficient indexing mechanism (such as IVF and HNSW indexes), the complexity of vector similarity calculation is greatly reduced, enabling fast and accurate retrieval under large data volumes and resolving the contradiction between semantic matching accuracy and retrieval efficiency.

[0038] Step S1013: Perform multi-dimensional fusion scoring on the first candidate segment, and select a preset number of segments from the first candidate segment as the second candidate segment based on the multi-dimensional fusion scoring results.

[0039] In some embodiments, step S1013 of performing multidimensional fusion scoring on the first candidate fragment includes: Step S301: Determine the semantic score of the image based on the input image semantic vector and the image semantic vector of the first candidate segment.

[0040] Step S302: Determine the dialogue semantic score based on the input text semantic vector and the text semantic vector of the first candidate segment.

[0041] Step S303: Based on the preset tag system, calculate the tag similarity between the user input content and the first candidate segment to obtain the dynamic tag matching score.

[0042] Step 304: Perform linear weighted fusion of the image semantic score, dialogue semantic score, and dynamic tag matching score to obtain a multi-dimensional weighted score.

[0043] The system performs an initial sorting of the candidate segments recalled by dual-channel retrieval, and integrates three types of correlation signals from image semantics, dialogue semantics, and tag system.

[0044] For any candidate segment C i Let its image vector be v. i The text vector is l iGiven that all vectors have been normalized, its semantic segmentation S... sem (C i This can be represented as:

[0045] Semantic Segmentation of Dialogue dia (C i This can be represented as:

[0046] S sem (C i ) and S dia (C i Mathematically, it is equivalent to cosine similarity, which is used to measure the degree of matching between input content and candidate judgments in their respective semantic spaces.

[0047] Calculate the screen description query Q using the Bi-Encoder model v With each tag T in the tag system j (e.g., "warmth" or "achievement display") semantic similarity Sim(Q) v ,T j The tagging system is divided into several semantic dimensions D. k For example, the emotional atmosphere dimension D1, the content function dimension D2, etc. For candidate segment C... i In dimension D k The set of tags owned by the entity is denoted as Tags(C). i D k The label matching score for semantic dimension k is S. tag (C i D k The similarity between the tags contained in the current segment and the query is defined as the average value multiplied by the preset weight w of that dimension. k ,Right now:

[0048] Final dynamic tag matching score S tag (C i This is obtained by normalizing and weighting the scores across all dimensions to ensure that its value range is comparable to other semantic scores.

[0049] Based on this, candidate fragment C is generated by linearly weighting and fusing the three sub-scores. i Multidimensional weighted score S total (C i ): S total (C i )=α·S sem(C i ) +β·S dia (C i ) +γ·S tag (C i ) Where α, β, and γ are configurable non-negative weight parameters, satisfying α + β + γ = 1. Based on S... total (C i The candidate set is sorted, and the top-N segments (e.g., the top 20) are selected as the second candidate segments and enter the subsequent deep re-sorting stage.

[0050] Optionally, the initial tag score weight γ is set to 0, at which point the tag score enhancement function is not enabled; upon receiving a tag instruction from the user (e.g., "more emphasis on emotion", "more emphasis on content function", or "more emphasis on presentation techniques"), the tag score enhancement function is enabled in response to the tag instruction, γ is adjusted to a preset non-zero value, and a multi-dimensional weighted score S is calculated. total (C i ).

[0051] In some embodiments, step S101, which determines the target segment from the candidate segments based on deep interaction algorithms and semantic entity relevance analysis, includes: Step S1014: Based on the interactive encoder, the scene description and dialogue of the second candidate segment are deeply interacted with the user input content to obtain the fusion score of each second candidate segment.

[0052] An interactive encoder model, such as cross-encoder / mmarco-mMiniLMv2-L12-H384-v1, can be used, and its function can be abstracted as a mapping function f. CE Text pairs A ,text B The mapping is transformed into a real-valued relevance score. For each candidate fragment C i Construct separate screen input pairs and dialogue input pairs: screen input pairs are generated by the user querying Q on the screen. v Image description D of the candidate clip i Composed of multiple parts, represented as [CLS]Q v [SEP]D i [SEP]; Dialogue input is used by user dialogue query Q d The dialogue text L of the candidate segment i Composed of multiple parts, represented as [CLS]Q d [SEP]L i [SEP]. The two pairs of inputs mentioned above are fed into the Cross-Encoder model to obtain the original image resolution R.v (i)=f CE (P v (i) and the original score of the dialogue R d (i)=f CE (P d (i)). The two are weighted by a preset internal weight λ. v With λ d (For example, 0.8 and 0.2) are merged into a unified fraction R. fused (i): R fused (i) = λ v ·R v (i)+λ d ·R d (i) Since the logits score output by the Cross-Encoder has an unpredictable range, which is detrimental to subsequent processing and sorting, this embodiment uses a hyperbolic tangent function for nonlinear mapping and introduces a temperature coefficient τ to adjust the sensitivity of the mapping.

[0053] Mapped fusion score R final (i) is restricted to the interval (-1, 1).

[0054] Step S1015: Extract the first core entity from the user input content and the second core entity from the second candidate fragment, calculate the semantic similarity between the first core entity and the second core entity, and obtain the object overlap score.

[0055] Using natural language processing tools (such as spaCy) to query Q from the user screen v Automatic extraction of the core noun entity set E from the Chinese q (The first core entity), such as "dog," "cell phone," "desk," etc. For each candidate fragment C i Obtain its pre-extracted second core entity set E i Unlike simple keyword matching, this embodiment maps all entities to semantic vectors using a Bi-Encoder model and calculates the semantic similarity between the query entity set and the candidate entity set. Specifically, for each query entity e... j ∈E q Calculate its relationship with the second core entity set E i The cosine similarity of all entities in the query entity is calculated, and the maximum value is taken as the best matching score for that entity. Then, the arithmetic mean of the best matching scores for all query entities is calculated to obtain the object similarity score (Object_Similarity(C)). i ):

[0056] Step S1016: Determine the comprehensive score of each second candidate segment based on the fusion score and the object overlap score, and determine the target segment from the second candidate segments based on the comprehensive score.

[0057] The fusion score is used as the core score for deep interaction re-ranking, and the object overlap score is used as a multiplicative enhancement factor, applied to the aforementioned fine-ranking score, to form the final score (C). i ) is represented as: Final_Score(C i ) =R fused (i)·(1+w obj ·Object_Similarity(C i )) Among them, w obj This is a non-negative adjustable parameter used to control the strength of the influence of entity matching on the final ranking.

[0058] This mechanism ensures that, given overall semantic fit, explicitly mentioned key objects can significantly improve the final ranking of candidate segments, thereby enhancing the relevance of the results and user satisfaction.

[0059] This embodiment uses an interactive encoder to perform two-way deep interaction between the visual description and dialogue of the second candidate segment and the user's input content. This can uncover the semantic relationship, emotional fit, and logical consistency between the input and the segment, fully restore the user's real needs, and avoid the screening bias caused by misjudgment of the interaction intent.

[0060] By using the logic of "extracting core entities + calculating semantic similarity", the core entities (such as key objects / scenes like "child", "kite", and "seaside") are first extracted from the user input content and candidate fragments. Then, the degree of matching is quantified through semantic similarity algorithm to obtain the object overlap score, thereby achieving accurate and quantifiable entity matching.

[0061] By using a two-dimensional decision-making process of "fusion score + object overlap score", the system considers both the overall interaction fit between the input and the fragment (fusion score) and emphasizes the accurate matching of core entities (object overlap score), forming a dual verification of "overall fit + core focus", which reduces the probability of misselection from the root.

[0062] Continue to refer to Figure 1 After the target segment is determined, step S102 is executed.

[0063] Step S102: Determine the video length of the target segment, and determine the number of words in the dialogue of the target segment based on the large language model. Calculate the target playback length based on the number of words in the dialogue and the preset dubbing speed. Based on the target playback length and the video length, determine whether there is a duration deviation in the target segment.

[0064] The target playback duration is automatically calculated based on the number of words in the dialogue generated by the large language model and the preset dubbing speed (e.g., 4-5 words per second).

[0065] By introducing a large language model, natural dialogue that fits the scene can be generated based on the video clip's visual content and core entities. The model can also accurately count the number of words to ensure the adaptability of the dialogue to the video scene. At the same time, it provides accurate input for duration calculation, avoiding duration judgment failures caused by word count estimation errors.

[0066] Step S103: If there is a duration deviation in the target segment, adaptive speed adjustment is performed on the target segment based on the target playback duration and the video duration to generate the final video segment.

[0067] Adaptive speed adjustment is performed on video clips that are semantically matched but have different durations. When the clip is too long, frame skipping is used to reduce the frame rate. When the clip is too short, an AI frame interpolation model is called to generate intermediate frames, thereby achieving precise synchronization between video duration and audio rhythm while maintaining visual smoothness.

[0068] By precisely aligning the video segment (target clip) corresponding to each scene with the corresponding audio duration using the video length after speed adjustment, audio-visual synchronization at the scene level is achieved.

[0069] It should be noted that this speed adjustment only times the video track and does not change the audio (the audio is used as the alignment reference). The audio and video tracks will be merged during subsequent compositing.

[0070] In some embodiments, step S103, which involves adaptively adjusting the speed of the target segment based on the target playback duration and the video duration to generate the final video segment, includes: Step S1031: Calculate the speed-up factor based on the video duration and the target playback duration.

[0071] Step S1032: Based on the speed change factor and the preset speed change tolerance threshold, determine whether speed change is required. If so, perform speed change processing on the target segment based on the speed change factor to obtain the final video segment.

[0072] The formula for calculating the speed factor scale is: scale = v_dur / a_dur (when a_dur>0). Where v_dur is the video duration of the target segment, and a_dur is the target playback duration (audio duration).

[0073] Set a speed tolerance threshold epsilon (e.g., 2%). When |1-scale| ≤ epsilon, it is considered that no speed adjustment is needed, and the original video can be reused directly to avoid unnecessary recoding and quality loss. When |1-scale| > epsilon, optionally, a "time scaling" method such as MoviePy / FFmpeg can be used to make the duration after speed adjustment v_dur' = v_dur / scale = a_dur to achieve strict alignment.

[0074] This embodiment can also perform boundary and robustness processing: if a_dur<= 0, it falls back to scale =1.0 and issues an alarm.

[0075] By quantifying the speed adjustment factor, we ensure that the speed adjustment perfectly matches the duration requirements. A preset speed tolerance threshold is introduced, and speed adjustment only occurs when the deviation corresponding to the speed factor exceeds the threshold, avoiding "over-optimization." It should be noted that the speed tolerance threshold is set according to the tolerance for speed changes in different scenarios (e.g., for advertising videos with high duration accuracy requirements, the threshold is set at ±1%; for Vlog scenarios, it can be set more leniently at ±5%).

[0076] In some embodiments, step S1032, which involves speed-changing the target segment based on a speed-changing factor to obtain the final video segment, includes: Step S401: If the speed factor does not exceed the preset speed range, perform time scaling on the target segment according to the speed factor to obtain the final video segment.

[0077] In step S402, if the speed factor exceeds the preset speed range, the target segment is processed by frame extraction or frame interpolation based on motion compensation or frame interpolation strategies to obtain the final video segment.

[0078] Scene-specific processing: Set upper and lower limits for scale, i.e. preset speed range (e.g., [0.2, 4.0]). Within this range, time scaling is used (slight speed change, distortion can be ignored). Outside this range, motion compensation / frame interpolation is used (intermediate frames are generated by algorithm or intelligent frame extraction to maintain the smoothness of the picture) to solve the distortion problem of excessive speed change.

[0079] By employing a "frame extraction / interpolation + motion compensation" strategy, extreme duration deviations can be addressed: if the speed factor is too small, intelligent frame extraction based on motion compensation is used (key action frames are retained, redundant frames are deleted, and screen jumps are avoided); if the speed factor is too large, intermediate frames are generated based on frame interpolation (such as optical flow interpolation to make the action transition natural), ensuring that the target duration can still be accurately adapted even under extreme deviations.

[0080] In some embodiments, step S103 further includes: Step S1033: Determine whether the absolute difference between the target segment duration after adaptive speed processing and the target playback duration is greater than the secondary verification threshold. If so, fine-tune the speed factor and, based on the fine-tuned speed factor, re-process the original target segment with speed.

[0081] Optionally, after the speed change is completed, a second verification is performed to determine whether |v_dur' - a_dur| exceeds the millisecond threshold. If so, the scale can be fine-tuned and the test can be repeated.

[0082] The residual deviation is quantitatively judged based on the standard of "absolute difference ≤ secondary verification threshold (e.g., ±0.1 seconds)"; if the deviation exceeds the standard, the speed factor is finely adjusted (e.g., if the factor is 0.8 and the deviation is 0.2 seconds, it is finely adjusted to 0.78) and the process is repeated.

[0083] Step S1034: Record the video duration, target playback duration, speed factor, speed processing time, and output path for each target segment.

[0084] Record the original video duration v_dur, target playback duration (audio duration) a_dur, speed factor scale, processing time, and output path for each target segment; this facilitates review, adjustment, and problem backtracking.

[0085] Optionally, a fallback strategy can be set when the speed change fails or exceeds the limit: select a candidate video of approximately the same length, perform a slight crop / leave black text, or allow a slight desynchronization of ±δ in local segments to ensure that the task can be completed.

[0086] Through the above steps, deep interaction algorithms and semantic entity relevance analysis not only match surface keywords but also uncover the potential needs of user input and the semantic entities of candidate segments, achieving deep alignment between needs and materials and reducing screening errors. By predicting the number of words in dialogue using a large language model and calculating the target playback duration using a preset speech rate, and comparing the deviation with the video duration, precise quantification of duration deviation is achieved, providing data support for subsequent processing. Based on the degree of deviation between the target playback duration and the video duration, the speed-changing algorithm is dynamically optimized to ensure audio-visual synchronization while maximizing the preservation of the integrity of the visuals and audio, thus solving the problem of audio-visual asynchrony in video.

[0087] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0088] This embodiment also provides a video processing system based on audio-visual synchronization, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0089] Figure 2 This is a structural block diagram of a video processing system based on audio-visual synchronization according to an embodiment of this application, such as... Figure 2 As shown, the system includes: The segment selection module 51 is used to filter candidate segments from the video material library based on user input, and determine the target segment from the candidate segments based on deep interaction algorithms and semantic entity relevance analysis.

[0090] The deviation judgment module 52 is used to determine the video duration of the target segment, determine the number of words in the dialogue of the target segment based on the large language model, calculate the target playback duration based on the number of words in the dialogue and the preset dubbing speed, and judge whether there is a duration deviation in the target segment based on the target playback duration and the video duration.

[0091] The audio-visual alignment module 53 is used to perform adaptive speed adjustment on the target segment based on the target playback duration and the video duration when there is a duration deviation in the target segment, so as to generate the final video segment.

[0092] In some embodiments, the audio-visual alignment module includes: The speed factor calculation module is used to calculate the speed factor based on the video duration and the target playback duration.

[0093] The segment processing module is used to determine whether speed adjustment is needed based on the speed adjustment factor and the preset speed adjustment tolerance threshold. If so, the target segment is processed based on the speed adjustment factor to obtain the final video segment.

[0094] In some embodiments, the fragment processing module includes: The first processing module is used to perform time scaling on the target segment according to the speed factor, so as to obtain the final video segment, provided that the speed factor does not exceed the preset speed range.

[0095] The second processing module is used to perform frame extraction or frame interpolation on the target segment based on motion compensation or frame interpolation strategies when the speed factor exceeds the preset speed range, so as to obtain the final video segment.

[0096] In some embodiments, the fragment processing module further includes: The secondary verification module is used to determine whether the absolute difference between the target segment duration after adaptive speed processing and the target playback duration is greater than the secondary verification threshold. If so, the speed factor is fine-tuned, and the original target segment is re-processed based on the fine-tuned speed factor.

[0097] The recording module is used to record the video duration, target playback duration, speed factor, speed processing time, and output path for each target segment.

[0098] In some embodiments, the fragment selection module includes: The encoding module is used to parse and encode user input, generating semantic vectors for the input screen and input text.

[0099] The first filtering module is used to determine the first candidate segment from the video material library based on the semantic vector of the input image, the semantic vector of the input text, and the semantic vector library. The semantic vector library is constructed based on the video material library.

[0100] The second screening module is used to perform multi-dimensional fusion scoring on the first candidate segments, and select a preset number of segments from the first candidate segments as second candidate segments based on the multi-dimensional fusion scoring results.

[0101] In some embodiments, the semantic vector library includes a visual semantic vector library and a text semantic vector library; the first filtering module includes: The initial screening module is used to generate a structured query language based on user input, and then match the structured query language against the SQLite database to obtain a preliminary set of candidate fragments.

[0102] The vector matching module is used to match the initial candidate fragment set with the image semantic vector library and the text semantic vector library respectively, to obtain image vector subsets and text vector subsets.

[0103] The selection module is used to perform k-nearest neighbor search in the subsets of image vectors and text vectors based on the input image semantic vector and the input text semantic vector using FAISS, and determine the first candidate segment based on the search results.

[0104] In some embodiments, the second screening module includes: The image scoring module is used to determine the image semantic score based on the input image semantic vector and the image semantic vector of the first candidate segment.

[0105] The text scoring module is used to determine the semantic score of the dialogue based on the semantic vector of the input text and the semantic vector of the first candidate segment.

[0106] The tag scoring module is used to calculate the tag similarity between the user input content and the first candidate segment based on a preset tag system, and obtain a dynamic tag matching score.

[0107] The comprehensive scoring module is used to linearly weight and fuse the image semantic score, dialogue semantic score, and dynamic tag matching score to obtain a multi-dimensional weighted score.

[0108] In some embodiments, the fragment selection module includes: The deep interaction analysis module is used to perform deep interaction between the visual description and dialogue of the second candidate segment and the user input content based on the interactive encoder, so as to obtain the fusion score of each second candidate segment.

[0109] The semantic entity analysis module is used to extract the first core entity from the user input content and the second core entity from the second candidate fragment, calculate the semantic similarity between the first core entity and the second core entity, and obtain the object overlap score.

[0110] The target segment determination module is used to determine the comprehensive score of each second candidate segment based on the fusion score and the object overlap score, and to determine the target segment from the second candidate segments based on the comprehensive score.

[0111] Through the aforementioned system, deep interaction algorithms and semantic entity relevance analysis not only match surface keywords but also uncover potential user input needs and semantic entities of candidate segments, achieving deep alignment between needs and materials and reducing screening errors. By predicting the number of words in dialogue using a large language model and calculating the target playback duration using a preset speech rate, and comparing the deviation with the video duration, precise quantification of duration deviation is achieved, providing data support for subsequent processing. Based on the degree of deviation between the target playback duration and the video duration, the speed-adjusting algorithm is dynamically optimized to ensure audio-visual synchronization while maximizing the preservation of the integrity of the visuals and audio, thus solving the problem of audio-visual asynchrony in video.

[0112] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0113] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0114] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0115] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1: Based on user input, candidate segments are selected from the video material library. Based on deep interaction algorithms and semantic entity correlation analysis, the target segment is determined from the candidate segments.

[0116] S2, determine the video length of the target segment, and determine the number of words in the dialogue of the target segment based on the large language model. Calculate the target playback length based on the number of words in the dialogue and the preset dubbing speed. Based on the target playback length and the video length, determine whether there is a duration deviation in the target segment.

[0117] S3, when there is a duration deviation in the target segment, performs adaptive speed adjustment on the target segment based on the target playback duration and the video duration to generate the final video segment.

[0118] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0119] In one embodiment, Figure 3 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 3 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 3 As shown, this electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a video processing method based on audio-visual synchronization.

[0120] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0122] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A video processing method based on audio-visual synchronization, characterized in that, The method includes: Based on user input, candidate segments are selected from the video material library, and the target segment is determined from the candidate segments based on deep interaction algorithms and semantic entity relevance analysis. The video length of the target segment is determined, and the number of words in the dialogue of the target segment is determined based on a large language model. The target playback length is calculated based on the number of words in the dialogue and a preset dubbing speed. Based on the target playback length and the video length, it is determined whether there is a duration deviation in the target segment. If the target segment has a duration deviation, the target segment is subjected to adaptive speed adjustment based on the target playback duration and the video duration to generate the final video segment.

2. The method according to claim 1, characterized in that, The step of performing adaptive speed-changing processing on the target segment based on the target playback duration and the video duration to generate the final video segment includes: Calculate the speed factor based on the video duration and the target playback duration; Based on the speed change factor and the preset speed change tolerance threshold, it is determined whether speed change is needed. If so, the target segment is speed-changed based on the speed change factor to obtain the final video segment.

3. The method according to claim 2, characterized in that, The process of speed-changing the target segment based on the speed-changing factor to obtain the final video segment includes: If the speed factor does not exceed the preset speed range, the target segment is time-scaled according to the speed factor to obtain the final video segment. If the speed factor exceeds the preset speed range, the target segment is subjected to frame extraction or frame interpolation based on motion compensation or frame interpolation strategies to obtain the final video segment.

4. The method according to claim 2, characterized in that, The method further includes: Determine whether the absolute difference between the target segment duration after adaptive speed adjustment and the target playback duration is greater than a secondary verification threshold. If so, fine-tune the speed adjustment factor and, based on the fine-tuned speed adjustment factor, re-process the original target segment using speed adjustment; and / or Record the video duration, target playback duration, speed factor, speed processing time, and output path for each target segment.

5. The method according to claim 1, characterized in that, The step of filtering candidate segments from the video material library based on user input includes: The system parses and encodes user input to generate semantic vectors for the input screen and input text. Based on the semantic vector of the input image, the semantic vector of the input text, and the semantic vector library, a first candidate segment is determined from the video material library, wherein the semantic vector library is constructed based on the video material library; Perform multi-dimensional fusion scoring on the first candidate segment, and select a preset number of segments from the first candidate segment as the second candidate segment based on the multi-dimensional fusion scoring results.

6. The method according to claim 5, characterized in that, The semantic vector library includes a visual semantic vector library and a text semantic vector library; the step of determining the first candidate segment from the video material library based on the input visual semantic vector, the input text semantic vector, and the semantic vector library includes: A structured query language is generated based on user input, and the structured query language is matched against an SQLite database to obtain a preliminary set of candidate fragments. The preliminary candidate fragment set is matched with the image semantic vector library and the text semantic vector library respectively to obtain image vector subsets and text vector subsets; Using FAISS, k-nearest neighbor search is performed on the subset of the input image vector and the subset of the input text vector based on the input image semantic vector and the input text semantic vector, and the first candidate segment is determined based on the search results.

7. The method according to claim 5, characterized in that, The multidimensional fusion scoring of the first candidate fragment includes: Based on the input image semantic vector and the image semantic vector of the first candidate segment, determine the image semantic score; Based on the input text semantic vector and the text semantic vector of the first candidate segment, the dialogue semantic score is determined; Based on a preset tag system, the tag similarity between the user input content and the first candidate segment is calculated to obtain a dynamic tag matching score; The image semantic score, dialogue semantic score, and dynamic tag matching score are linearly weighted and fused to obtain a multi-dimensional weighted score.

8. The method according to claim 1, characterized in that, The determination of the target segment from the candidate segments based on deep interaction algorithms and semantic entity relevance analysis includes: Based on the interactive encoder, the scene description and dialogue of the second candidate segment are deeply interacted with the user input content to obtain the fusion score of each second candidate segment; Extract the first core entity from the user input content and the second core entity from the second candidate fragment, calculate the semantic similarity between the first core entity and the second core entity, and obtain the object overlap score; Based on the fusion score and the object overlap score, a comprehensive score is determined for each of the second candidate segments, and a target segment is determined from the second candidate segments based on the comprehensive score.

9. A video processing system based on audio-visual synchronization, characterized in that, The system includes: The segment selection module is used to filter candidate segments from the video material library based on user input, and determine the target segment from the candidate segments based on deep interaction algorithms and semantic entity relevance analysis. The deviation judgment module is used to determine the video duration of the target segment, determine the number of words in the dialogue of the target segment based on the large language model, calculate the target playback duration based on the number of words in the dialogue and the preset dubbing speed, and determine whether there is a duration deviation in the target segment based on the target playback duration and the video duration. The audio-visual alignment module is used to perform adaptive speed adjustment on the target segment based on the target playback duration and the video duration when the target segment has a duration deviation, so as to generate the final video segment.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video processing method based on audio-visual synchronization as described in any one of claims 1 to 8.

Images