A method, system, and readable storage medium for video production

Abstract

This application relates to a video production method, system, and readable storage medium. The video production method includes: acquiring a set of synchronous audio-visual footage and a set of establishing shot footage captured by a terminal device; preprocessing the audio signals in each synchronous audio-visual footage to obtain multiple structured segments; preprocessing each establishing shot footage to obtain multiple valid establishing shot segments; determining a fusion score for each structured segment based on a scoring rule; determining whether a valid establishing shot segment needs to be inserted into the synchronous audio-visual footage corresponding to each structured segment and the corresponding editing strategy based on the fusion score; if insertion is required, determining the most matching valid establishing shot segment based on a matching rule; inserting the most matching valid establishing shot segment into the corresponding synchronous audio-visual footage segment based on the editing strategy; and splicing and rendering each synchronous audio-visual footage to obtain a finished video, thereby improving the accuracy and timeliness of video production.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video production method, system, and readable storage medium. Background Technology

[0002] Currently, in the field of news and short video production, there are related technologies such as "automatic editing" or "AI one-click video production". However, their underlying algorithms are mainly based on template matching or low-level visual feature analysis. They lack intelligent understanding of the semantic level of interview content and the ability to match semantics at the shot level. They cannot retrieve the semantically corresponding empty shot at the visual level, resulting in the generated result being inconsistent with the semantics of the speech in terms of content logic, which makes it difficult to meet the accuracy requirements of video production. Summary of the Invention

[0003] Therefore, it is necessary to provide a video production method, system, and readable storage medium to address the aforementioned technical problems.

[0004] In a first aspect, embodiments of this application provide a method for producing video clips, the method comprising:

[0005] Acquire a set of synchronous audio and video footage captured by a terminal device and a set of establishing shot footage; wherein: the set of synchronous audio and video footage includes multiple synchronous audio and video clips, and the set of establishing shot footage includes multiple establishing shot footage.

[0006] The audio signals in each of the synchronous sound and video recordings are preprocessed to obtain multiple structured sentence segments; and the empty shot footage is preprocessed to obtain multiple effective empty shot segments.

[0007] Based on the scoring rules, the fusion score of each structured segment is determined; and based on the fusion score, it is determined whether the synchronous audio-visual segment corresponding to each structured segment needs to be inserted with an effective empty shot and the corresponding editing strategy.

[0008] If insertion is required, the effective empty scene segment that best matches the structured sentence segment is determined based on the matching rules.

[0009] Based on the editing strategy, the most matching valid empty shot is inserted into the corresponding synchronous sound and video clip; and the synchronous sound and video clips are spliced ​​and rendered to obtain the final video.

[0010] In one embodiment, the audio signals in each of the synchronous audio-visual recordings are preprocessed to obtain multiple structured segments, including:

[0011] Speech recognition is performed on the audio signals in the synchronous sound and video to obtain the corresponding text information;

[0012] The text information is segmented into sentences based on speech features and language structure to obtain multiple sentence segments; and corresponding video start time and speech quality parameters are added to each sentence segment to obtain multiple structured sentence segments.

[0013] In one embodiment, the preprocessing of each of the empty shot materials to obtain multiple valid empty shot segments includes:

[0014] Calculate the inter-frame visual differences of each of the aforementioned establishing shot materials, and determine whether the corresponding establishing shot material has a shot segmentation point based on the inter-frame visual differences; if so, segment the corresponding establishing shot material based on the segmentation point.

[0015] Statistically analyze the average optical flow and duration of each empty shot image.

[0016] Based on preset optical flow thresholds and preset duration thresholds, multiple empty shot materials are filtered to obtain multiple valid empty shot segments.

[0017] In one embodiment, determining the fusion score of each structured segment based on the scoring rules includes:

[0018] Based on each of the structured sentence segments, determine the corresponding semantic importance and keyword density;

[0019] Based on the synchronous audio-visual segments corresponding to each of the structured sentence segments, the corresponding character's emotional intensity, face stability score, and speech signal-to-noise ratio are determined.

[0020] The fusion score of the structured sentence segment is obtained by weighting the semantic importance, keyword density, emotional intensity, facial stability score, and speech signal-to-noise ratio.

[0021] In one embodiment, determining whether valid empty shot segments need to be inserted into the synchronous audio-visual segments corresponding to each of the structured segments based on the fusion score and the corresponding editing strategy includes:

[0022] Determine whether the fusion score of the structured sentence segment is greater than or equal to a first preset value; if yes, then no valid empty shot segment needs to be inserted; if no, then a valid empty shot segment needs to be inserted.

[0023] If it is necessary to insert effective empty shot footage, and the fusion score of the structured segment is between the second preset value and the first preset value, then the first part of the synchronous audio and video clip is retained, and the remaining part of the synchronous audio and video clip is replaced with the most matching effective empty shot clip.

[0024] If the fusion score of the structured segment is less than the second preset value, then all synchronous audio and video segments are replaced with the most matching effective empty shot segments.

[0025] In one embodiment, determining the effective empty shot fragment that best matches the structured sentence segment based on matching rules includes:

[0026] Calculate the semantic similarity and entity similarity between the structured sentence segment and each of the effective empty scene segments;

[0027] Based on the semantic similarity and the entity similarity, the effective empty scene segment that best matches the structured sentence segment is selected from the multiple effective empty scene segments.

[0028] In one embodiment, inserting the most matching valid establishing shot into the corresponding synchronous sound video based on the editing strategy includes:

[0029] Determine the entry point of the most matching valid empty shot segment; the entry point includes one of the following: sentence end frame, tone pause frame, and mouth closing frame;

[0030] Based on the stated entry point and editing strategy, the most matching and effective empty shot is inserted into the corresponding synchronous sound and video.

[0031] In one embodiment, before preprocessing each of the empty shot footage to obtain multiple valid empty shot segments, the method further includes:

[0032] The aforementioned empty shot footage was subjected to compliance and stability checks, and blurry frames, sensitive information frames, and frames featuring illegal characters were deleted.

[0033] Secondly, embodiments of this application also provide a video editing system, the system comprising:

[0034] The acquisition module is used to acquire a set of synchronous audio and video footage and a set of establishing shot footage captured by the terminal device; wherein: the set of synchronous audio and video footage includes multiple synchronous audio and video clips, and the set of establishing shot footage includes multiple establishing shot footage.

[0035] The preprocessing module is used to preprocess the audio signals in each of the synchronous audio and video recordings to obtain multiple structured segments; and to preprocess the empty shot footage to obtain multiple effective empty shot clips.

[0036] The scoring module is used to determine the fusion score of each structured segment based on the scoring rules; and to determine whether the synchronous audio-visual segment corresponding to each structured segment needs to be inserted with an effective empty shot and the editing strategy based on the fusion score.

[0037] The matching module is used to determine, if so, the effective empty shot fragment that best matches each of the structured sentence segments based on the matching rules;

[0038] The video editing module is used to insert the most matching valid empty shot segment into the corresponding synchronous sound and video segment based on the editing strategy; and to splice and render each synchronous sound and video segment to obtain the final video.

[0039] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect above.

[0040] The aforementioned video production method, system, and readable storage medium, after capturing synchronous sound video and related scene footage using a mobile terminal device, allow the terminal device to automatically analyze interview semantics, understand the content of the speech and the context of the event, and retrieve semantically consistent empty shots. This achieves an end-to-end process from interview shooting to semantic understanding, shot matching, intelligent editing, and instant production, improving the accuracy and timeliness of video production.

[0041] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0043] Figure 1 This is a hardware structure block diagram of a terminal device for a video editing method in one embodiment;

[0044] Figure 2 This is a flowchart illustrating a video editing method in one embodiment;

[0045] Figure 3 This is a flowchart illustrating the editing strategy in one embodiment;

[0046] Figure 4 This is a flowchart illustrating the process of removing redundant empty shot segments in one embodiment;

[0047] Figure 5 This is a schematic diagram of the structure of a visual entity detection model in one embodiment;

[0048] Figure 6 This is a flowchart illustrating a preferred embodiment of a video editing method;

[0049] Figure 7 This is a structural block diagram of a video production system in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0051] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the mobile terminal for the video production method of this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0052] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the video processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0053] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a radio frequency (RF) module used for wireless communication with the Internet.

[0054] This application provides a method for video editing, which can be applied to... Figure 1 Taking the terminal in the example of this, for example... Figure 2 As shown, the method includes the following steps:

[0055] Step S201: Obtain a set of synchronous audio and video footage and a set of establishing shot footage captured by the terminal device; wherein: the set of synchronous audio and video footage includes multiple synchronous audio and video clips, and the set of establishing shot footage includes multiple establishing shot footage.

[0056] In the news and short video production process, personnel typically use mobile devices such as smartphones and cameras to collect original interview footage, including several synchronous audio-visual recordings and establishing shots. Synchronous audio-visual recordings consist of the interviewee's speech and constitute the main information carrier of the news report or short video. Establishing shots are used to record the interview or filming environment, urban architecture, natural scenes, or supplementary footage, primarily for enriching the visuals, adjusting the pacing, and facilitating transitions.

[0057] Step S202 involves preprocessing the audio signals in each of the synchronous audio and video recordings to obtain multiple structured segments; and preprocessing the empty shot footage to obtain multiple effective empty shot clips.

[0058] This step involves basic audio-visual processing, structured expression extraction, and quality analysis of each synchronous audio-visual and each establishing shot in the original interview footage, providing crucial input for subsequent editing.

[0059] Step S203: Based on the scoring rules, determine the fusion score of each structured segment; and based on the fusion score, determine whether the synchronous audio-visual segment corresponding to each structured segment needs to be inserted with an effective empty shot and the corresponding editing strategy.

[0060] This step calculates a fusion score for each structured statement according to the scoring rules. The fusion score is used to evaluate the exposure value of the corresponding synchronous audio-visual segment. Then, based on the fusion score, it is determined whether an empty shot needs to be inserted into the synchronous audio-visual segment corresponding to the structured statement. If an empty shot needs to be inserted, the most matching valid empty shot is selected from multiple valid empty shot segments for insertion. At the same time, the corresponding editing strategy is determined based on the fusion score. The editing strategy includes replacing all or part of the synchronous audio-visual segment corresponding to the structured statement with the most matching valid empty shot.

[0061] In step S204, if insertion is required, the effective empty scene segment that best matches the structured sentence segment is determined based on the matching rules.

[0062] Step S205: Based on the editing strategy, insert the most matching valid empty shot segment into the corresponding synchronous sound and video segment; and splice and render each synchronous sound and video segment to obtain the final video.

[0063] Automatically adds subtitles and background music, organizes audio transcription text of synchronous video and audio, inserts effective empty shot clips, subtitles, transitions and other editing elements into a standard timeline, drives subsequent video rendering, subtitle alignment and multi-scale export, outputs video clips that meet the expected style, and can be directly published on terminal devices.

[0064] In the above-mentioned video production method, after shooting synchronous sound video of a person's interview and related scene footage using a mobile terminal device, the terminal device can automatically analyze the interview semantics, understand the content of the speech and the context of the event, and retrieve semantically consistent empty shots. This realizes an end-to-end process from interview shooting to semantic understanding, shot matching, intelligent editing, and instant production, improving the accuracy and timeliness of video production.

[0065] In one embodiment, the audio signals in each of the synchronous audio-visual recordings are preprocessed to obtain multiple structured sentence segments, including the following: performing speech recognition on the audio signals in the synchronous audio-visual recordings to obtain corresponding text information; segmenting the text information into sentences based on speech features and language structure to obtain multiple sentence segments; and adding corresponding video start time and speech quality parameters to each sentence segment to obtain multiple structured sentence segments.

[0066] In one specific embodiment, preprocessing the audio signals in each synchronous audio-visual recording includes the following steps:

[0067] Step S301, Audio signal preprocessing: For the audio signal of the original synchronous audio-visual recording, noise suppression is performed using a real-time audio noise suppression library (RNNoise) based on a recurrent neural network, and the loudness is normalized to... LUFS The scope is expanded to improve speech recognition accuracy and the robustness of subsequent sentence segmentation.

[0068] Step S302, Automatic Speech Recognition and Timestamp Alignment: Based on Automatic Speech Recognition (ASR) technology, the audio signals of the interview video are aligned and recognized, and the corresponding text information and frame-level timestamps are output. At the same time, the signal-to-noise ratio (SNR) and clarity of each sentence are calculated.

[0069] Step S303, speech-text cleaning: unify and clarify interjections and filler words (such as "that," "that is to say," "actually," "um"), merge obviously repetitive sentence structures, and correct double sentences in automatic speech recognition. In addition, the GPT-bsedrewriter algorithm is used to extract core expressions and generate simplified main sentences.

[0070] Step S304, Sentence-level semantic segmentation logic: Combining speech features and language structure, the continuous text is segmented into sentences. Sentences are segmented according to the following rules: if the speech pause is >300ms, or the speech rate drops suddenly, it is used as a potential sentence break point; if the sentence length is >10s, BERT syntax structure is used to assist in segmentation; if the sentence length is <2s and it is a "catchphrase / greeting", it is merged with adjacent sentences. For example, one structured sentence segment is: {"sentence_id":3,"start_time":12.38,"end_time":17.42,"text":"Our first batch of implementations is the government affairs hall area.","signal-to-noise ratio":22.6,"clarity":0.91}. This structured sentence segment serves as the input for subsequent face analysis and face recognition determination, the anchor point for effective empty shot insertion and semantic matching, and the basis for generating the subtitle timeline.

[0071] In one embodiment, the preprocessing of each of the establishing shot materials to obtain multiple valid establishing shot segments includes the following: calculating the inter-frame visual differences of each of the establishing shot materials; determining whether the corresponding establishing shot material has a shot segmentation point based on the inter-frame visual differences; if so, segmenting the corresponding establishing shot material based on the segmentation point; calculating the average optical flow and duration of each of the establishing shot materials; and filtering the multiple establishing shot materials based on a preset optical flow threshold and a preset duration threshold to obtain multiple valid establishing shot segments.

[0072] In one specific embodiment, preprocessing the various empty shot footage includes the following steps:

[0073] Step S401, Video Preprocessing: Perform unified preprocessing on the establishing shot footage, removing all audio and retaining only the video. Simultaneously, use FFmpeg to de-jitter and stabilize the image.

[0074] Step S402, Shot Segmentation and Still Frame Extraction of Establishing Shot Footage: To extract representative establishing shot footage segments, shot boundary detection and still frame segment selection are required. First, calculate the inter-frame visual differences (RGB histogram differences or optical flow abrupt changes), mark the shot segmentation points, and segment the establishing shot footage based on these points. Next, calculate the inter-frame motion vectors and select sub-segments (represented as relatively static) with an average optical flow less than a threshold. Then, select valid establishing shot footage segments by limiting the effective candidate segments to 2s~5s.

[0075] Motion vectors are vectors used in video coding to identify block-level pixel offsets between frames. Optical flow represents the direction and velocity of movement of each pixel in an image over time, typically a displacement field between consecutive image frames. In video coding, motion vectors are discrete, block-level approximations of optical flow; optical flow is a continuous, dense flow estimate of all pixels or keypoints. Average optical flow is a more refined expression of motion vectors, used to measure the degree of motion in the image, and is suitable for selecting effective establishing shot clips.

[0076] After filtering the empty shot footage using the average optical flow threshold, the segmentation algorithm further divides it into several effective empty shot footage segments between 2s and 5s based on semantics.

[0077] Step S403, Content Structure Tag Generation: To support subsequent tasks such as semantic matching and entity consistency judgment, each valid empty shot clip needs to generate structure tags, including {Clip ID, Description Text, Clip Duration, Image Style Embedding, Scene Classification};

[0078] Step S404, Content Compliance Check and Filtering: Each valid empty shot clip must pass a security check to ensure that the final cut is compliant and legal.

[0079] Through steps S401 to S404 above, each original empty shot footage is standardized and structured with tags, providing a basis for candidate footage for subsequent semantic matching, style consistency judgment, and final editing.

[0080] In one embodiment, determining the fusion score of each structured sentence segment based on the scoring rules includes the following: determining the corresponding semantic importance and keyword density based on each structured sentence segment; determining the corresponding character emotion intensity, face stability score, and speech signal-to-noise ratio based on the synchronous audio-visual segment corresponding to each structured sentence segment; and obtaining the fusion score of the structured sentence segment based on the weighted sum of the semantic importance, keyword density, character emotion intensity, face stability score, and speech signal-to-noise ratio.

[0081] The scoring metrics for this application include semantic importance, keyword density, character emotion intensity, face stability score, and speech signal-to-noise ratio, as detailed in Table 1.

[0082] Table 1

[0083]

[0084] Among them, semantic importance and keyword density are determined based on structured sentence segments, emotion intensity and signal-to-noise ratio are determined based on the audio signals of the synchronous audio-visual segments corresponding to the structured sentence segments, and face stability score is determined based on the video footage of the synchronous audio-visual segments corresponding to the structured sentence segments.

[0085] This application embodiment constructs a multimodal editing scoring system to intelligently determine whether and for how long a character's face is shown. It integrates indicators such as semantic importance, keyword density, emotional intensity, facial stability, and signal-to-noise ratio to output a fusion score as the core basis for editing, which can accurately identify and retain content.

[0086] In a specific embodiment, the ShowScore for each structured segment is calculated using the following formula:

[0087] ;

[0088] Wherein, P1 is semantic importance, P2 is keyword density, P3 is emotion intensity, P4 is face stability score, and P5 is signal-to-noise ratio.

[0089] In one embodiment, determining whether to insert valid empty shot clips and the corresponding editing strategy for each structured segment based on the fusion score includes the following: determining whether the fusion score of the structured segment is greater than or equal to a first preset value; if yes, then no valid empty shot clip needs to be inserted; if no, then a valid empty shot clip needs to be inserted. If valid empty shot clips need to be inserted, and the fusion score of the structured segment is between a second preset value and the first preset value, then the first part of the synchronous audio-visual segment is retained, and the remaining synchronous audio-visual segments are replaced with the most matching valid empty shot clip. If the fusion score of the structured segment is less than the second preset value, then all synchronous audio-visual segments are replaced with the most matching valid empty shot clip.

[0090] In one specific embodiment, a first preset value is set to 0.65, and a second preset value is set to 0.45. If the ShowScore is ≥ 0.65, no valid empty shot is inserted, no editing is performed, and the complete face shot of the character is retained. The duration of the synchronous audio-visual segment is controlled between 2 and 4 seconds. Otherwise, the corresponding editing strategy is as follows: Figure 3 As shown: If 0.45≤ShowScore<0.65, the first 1~1.5s of the scene showing the person's face will be retained, and the end will automatically transition to the most matching effective empty shot. If ShowScore<0.45, the person will be completely hidden, and only the audio track and subtitles of the synchronous audio-visual clip will be retained. The entire scene will be replaced with the most matching effective empty shot.

[0091] Furthermore, if there are multiple consecutive high-scoring segments (ShowScore≥0.65), by default only the synchronous audio-visual segment of the first segment is retained, and subsequent segments with semantically consistent and valid empty shots are cut in to ensure the rhythm of the characters' exposure and avoid fatigue.

[0092] If there are multiple consecutive low-scoring segments (ShowScore≤0.45), multiple effective empty shot segments are used throughout the entire process. The empty shot segments are spliced ​​together to form visual continuity blocks using the method of "semantic theme + style clustering" to ensure that the expression of the picture does not lose semantic coherence.

[0093] Current technology has not yet established a unified speech-text-visual semantic space, making it impossible to map interview audio content to a unified semantic embedding space that can be measured with visual features. Therefore, the retrieval of establishing shots remains at the level of keyword matching or visual similarity, lacking semantic-level relevance measurement, and the selected shots often become semantically disconnected from the interview content. Therefore, in one embodiment, determining the most matching valid establishing shot fragment based on matching rules includes the following: calculating the semantic similarity and entity similarity between the structured sentence segment and each valid establishing shot fragment; and based on the semantic similarity and entity similarity, selecting the most matching valid establishing shot fragment from among the multiple valid establishing shot fragments.

[0094] First, the semantic similarity between the structured sentence segment and each of the effective empty scene segments is calculated, and the first round of matching and filtering is performed based on the semantic similarity.

[0095] This application establishes a unified multimodal semantic space to achieve accurate semantic matching between synchronous interview sound content and establishing shot footage, ensuring that the generated video maintains consistency in content logic, visual expression, and emotional rhythm.

[0096] Semantically align the interview's synchronous sound content (structured segments) with candidate valid empty shot segments (videos) to improve content relevance. For each structured segment in the set S={s1,s2,...,sn}, the following steps are performed:

[0097] Step S501, Semantic Vector Extraction: The sentence segment si∈S is converted into a semantic vector v using CLIP or VideoCLIP text encoder. t That is, v t =ftext(si).

[0098] Step S502: Input the candidate valid empty shot segment keyframes / intermediate frames into the visual encoder to obtain the image vector v i .

[0099] Step S503: Calculate the cosine similarity as the semantic similarity, and select the top-k semantic vectors with a semantic similarity ≥ 0.65 as the set of candidate matching valid empty scene segments V={v1,v2,...,vm}.

[0100] Semantic similarity:

[0101]

[0102] Step S504: Calculate MMR (Maximal Marginal Relevance), and remove redundant empty shot segments from the candidate matching valid empty shot segment set V. The specific process is as follows: Figure 4 As shown, the empty shot fragment set V is used as input to initialize the selected set S. The best fragment is selected from the empty shot fragment set V in a loop. The MMR score of each candidate empty shot fragment is calculated. The best fragment is selected based on the MMR score and updated to set S.

[0103] For each candidate empty shot segment v i ∈V, the MMR score is:

[0104]

[0105] λ∈[0,1] is a moderating coefficient that balances correlation and diversity. For example, λ=0.6~0.8.

[0106] Secondly, the entity similarity between the structured sentence segment and each of the effective empty scene segments is calculated, and a second round of matching and filtering is performed based on the entity similarity.

[0107] This application establishes consistency constraints on key entities (people, places, objects, institutions, etc.) between the interview semantic understanding results and visual materials. This ensures that during automatic editing, the objects, scenes, or environments appearing in the footage are semantically consistent with the entities mentioned in the interviewee's speech, thus guaranteeing consistency between what is "seen" and what is "described." For example, when the interviewee says, "This is the first laryngectomy surgery in Zhejiang," effective empty shot clips containing "hospital" and "operating room" are prioritized, including the following steps:

[0108] Step S601: Perform entity recognition on each structured sentence segment to obtain the first entity recognition result.

[0109] The input is a transcribed text of the interview's synchronous audio-visual recording, timestamped and aligned using ASR (Automatic Speech Recognition). Next, the text is segmented into structured segments using the BERT+CRF language model. These structured segments are then fed into the Qwen2.5-7B-Chat named entity model to identify and classify entities, including names, organizations / companies, locations, buildings, objects, events / activities, etc., and output a list of entities identified for that segment.

[0110]

[0111] Where e i Represents entity strings, c i Indicates entity category, s i This indicates the semantic weight reset reliability.

[0112] Step S602: Perform entity recognition on each valid empty shot segment to obtain the second entity recognition result.

[0113] To enhance the descriptive power of candidate footage for establishing shots at the semantic, compositional, and functional levels, this application designs a "three-layer detection + one-layer fusion" visual entity detection model, which performs entity recognition at the object level, scene level, and action / behavior level, respectively. The structure of the visual entity detection model is as follows: Figure 5 As shown, it includes:

[0114] Object detection layer: Responsible for identifying specific objects and people in the image. It uses the YOLOv8 or Grounding-DINO detection network to output category labels (e.g., "person", "car"), bounding boxes, and confidence scores for the detected content.

[0115] Scene recognition layer: Identifies the background environment category of the entire frame. It employs the Places365 scene recognition network to output scene category labels such as "construction site," "campus," and "conference room," which are used to align with the "place / organization / building" category in the text entities, improving the consistency of empty scene shots.

[0116] Action / pose recognition layer: When the scene contains people, it identifies their body movements or behavioral states. It utilizes the OpenPose network to identify action types, such as "speaking" or "walking," avoiding the incorrect selection of dynamic shots as empty frames for interpolation. Simultaneously, it is used to construct higher-order semantic matching features (e.g., a "teaching scene" requires the "teaching" behavior).

[0117] This application will use the keyframe sequences of each valid empty shot segment. The input to the visual entity detection model is an RGB image for each frame; the model outputs a set of visual entities corresponding to each valid empty shot. ,in Represent entity names (such as "teaching building", "pedestrian", "trees", etc.); Indicate category labels (scene, object, text, action); This represents the visual entity o. j The degree of semantic association with a structured semantic segment in the synchronous sound text; This indicates the visual entity The spatial location in the image is usually a rectangular box [x_min, y_min, x_max, y_max].

[0118] Step S603: Based on the first entity recognition result and the second entity recognition result, calculate the entity similarity between the structured sentence segment and each of the effective empty shot segments, and perform a second round of screening based on the entity similarity.

[0119] After the first two rounds of screening, the final step is time-length matching to filter segments that match the target sentence segment S. t The most suitable valid empty shot segment is selected and its insertion time is determined. The duration of the candidate footage must be within ±0.5s of the target footage. If the duration does not meet the requirement, non-linear trimming or slow motion is attempted to ensure that the duration matches the target statement. (If the duration Ti of the candidate valid empty shot segment does not meet the target synchronous audio statement Ts ±0.5s range, then if Ti > Ts + 0.5s, action boundary trimming is used; if Ti < Ts - 0.5s, video frame interpolation or slowing down the playback speed is used.)

[0120] This application's embodiments ensure high-quality insertion of effective establishing footage by constructing a triple constraint of "semantics + entity + duration". By integrating semantic similarity, entity consistency, and temporal continuity, it achieves multi-dimensional alignment between interview semantics and shot semantics, improving the accuracy of automatic shot selection and the logical consistency of the footage.

[0121] In one embodiment, inserting the most matching valid empty shot segment into the corresponding synchronous audio-visual file based on the editing strategy includes: determining the cut point of the most matching valid empty shot segment; the cut point includes one of a sentence end frame, a tone pause frame, and a mouth closing frame; and inserting the most matching valid empty shot segment into the corresponding synchronous audio-visual file based on the cut point and the editing strategy.

[0122] To prevent abrupt visuals, semantic breaks, and lip-syncing issues, this application introduces the following entry mechanism during video editing and splicing: Entry points are prioritized for semantically complete frames with closed lips, avoiding emphasis words or transition words, and removing filler words such as "um" and "ah" to ensure clear semantic structure and natural transitions. All entry points are preferentially aligned with sentence ends, pauses, and closed lips. If effective empty shot segments need to be inserted midway, emphasis words and logical transition words (such as "so," "but," and "we believe") are avoided. Lip-syncing logic: The degree of lip opening and vocal energy are used to determine the lip-syncing point, ensuring no lip jumps when effective empty shot segments are inserted.

[0123] Editing method control: J-Cut and L-Cut are introduced to control the switching between synchronous sound and video and effective establishing shot segments, improving the smoothness of audiovisual integration and enhancing the rhythm and professionalism of the final film.

[0124] For various scenes requiring effective supplementary empty shot segments, a suitable entry point is located based on the above-mentioned entry mechanism. After location, the insertion method is determined according to the position of the spoken lines and the editing strategy:

[0125] If an effective establishing shot is used as a "prelude" to a sentence, a J-Cut approach is adopted: the audio track appears first, followed by the visual, enhancing anticipation and continuity; the J-Cut method involves the audio starting early, with the visual switching later. Visual effect: the audience hears the sound of the new segment first, and then sees the corresponding visual, which can guide emotions or semantics in advance, creating a sense of "what is about to happen" transition.

[0126] If an effective establishing shot is used as a "sentence-end replacement," an L-Cut is employed: the visuals cut first, followed by the audio, to help delay the transition. In an L-Cut, the visuals leave before the audio, the video switches to the next shot, while the audio continues from the previous segment. The visual effect is that the audience hears the current dialogue while the visuals have already transitioned to the next segment, smoothly connecting scenes and strengthening semantic continuity.

[0127] If an effective empty shot is used as a "mid-section cover", the entry point is forcibly determined to be a lip-closed frame (based on double judgment of lip opening degree <20% + speech RMS <-30dB).

[0128] This application automatically selects the insertion method based on the effective insertion position of the empty shot segment, realizing a "sound and image separation" type of connection strategy to improve the smoothness of video rhythm and the integration of audiovisual elements. All cut points must ensure that the starting frame of the empty shot segment does not have motion blur to avoid a jumpy feeling that looks like it was cut off right after the dialogue was finished.

[0129] In one embodiment, before preprocessing each of the empty shot materials to obtain multiple valid empty shot segments, the method further includes: performing compliance and stability checks on each of the empty shot materials, and deleting blurry frames, sensitive information frames, and illegal character frames.

[0130] To ensure the compliance and stability of automatically generated video content, each establishing shot must undergo the following checks before use. First, image quality is checked for blurriness, jitter, exposure, and audio anomalies. Second, OCR is used to detect sensitive information such as logos, license plates, and vehicle emblems. Finally, a facial recognition algorithm is used to exclude unauthorized individuals by comparing against a blacklist. If suitable establishing shot footage is unavailable, "general establishing shots" from the default safe footage pool, such as clouds, landscapes, and office building exteriors, are used as a fallback. If an establishing shot clip has issues such as blurriness, ghosting, abnormal exposure, or mismatched aspect ratios, the relevant frames are automatically removed or the entire clip is deleted.

[0131] Figure 6 This is a preferred flowchart of the video editing method in this embodiment, such as... Figure 6 As shown, the video editing method includes the following steps: acquiring the original synchronous sound interview video; preprocessing the original synchronous sound interview video; performing speech recognition on the synchronous sound video to form structured sentence segments; calculating the fusion score of the sentence segments; determining whether the face is shown throughout the video; ensuring the face is shown for 1.5 seconds or more with effective empty shot segments; covering all effective empty shot segments throughout the video; performing face stability detection, slot layout, and security verification; matching effective empty shot segments; J_cut / L_cut alignment; semantic pauses / low-motion cut points; generating a timeline from the editing track; exporting multi-scale finished videos; quality inspection of the finished videos; and finally, producing the finished video.

[0132] This application also provides a video editing system, such as... Figure 7 As shown, the system includes:

[0133] The acquisition module 10 is used to acquire a set of synchronous audio and video footage and a set of establishing shot footage captured by the terminal device; wherein: the set of synchronous audio and video footage includes multiple synchronous audio and video footage, and the set of establishing shot footage includes multiple establishing shot footage.

[0134] The preprocessing module 20 is used to preprocess the audio signals in each of the synchronous audio and video recordings to obtain multiple structured segments; and to preprocess the empty shot materials to obtain multiple effective empty shot segments.

[0135] The scoring module 30 is used to determine the fusion score of each structured segment based on the scoring rules; and to determine whether the synchronous audio-visual segments corresponding to each structured segment need to be inserted with valid empty shot segments and editing strategies based on the fusion score.

[0136] The matching module 40 is used to determine, if so, the effective empty shot fragment that best matches each of the structured sentence segments based on the matching rules.

[0137] The video editing module 50 is used to insert the most matching valid empty shot segment into the corresponding synchronous sound and video segment based on the editing strategy; and to splice and render each synchronous sound and video segment to obtain the final video.

[0138] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above-described video production method embodiments.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A video tiling method, characterized by, The method comprises: acquiring a synchronous sound video set and an empty mirror material set photographed by a terminal device; wherein the synchronous sound video set comprises multiple synchronous sound videos, and the empty mirror material set comprises multiple empty mirror materials; preprocessing audio signals in each of the synchronous sound videos to obtain multiple structured sentence segments, and preprocessing each of the empty mirror materials to obtain multiple effective empty mirror segments; determining a fusion score of each of the structured sentence segments based on a scoring rule, and determining whether an effective empty mirror segment needs to be inserted into a synchronous sound video segment corresponding to the structured sentence segment and a corresponding editing strategy based on the fusion score; if an effective empty mirror segment needs to be inserted, determining an effective empty mirror segment that is most matched with the structured sentence segment based on a matching rule; inserting the most matched effective empty mirror segment into the corresponding synchronous sound video segment based on the editing strategy, and splicing and rendering each of the synchronous sound videos to obtain a completed video.

2. The method of claim 1, wherein, The preprocessing of the audio signals in each of the synchronous sound videos to obtain multiple structured sentence segments comprises: performing speech recognition on the audio signals in the synchronous sound videos to obtain corresponding text information; performing sentence segmentation on the text information based on speech features and language structures to obtain multiple sentence segments, and adding corresponding video start time and speech quality parameters to each of the sentence segments to obtain multiple structured sentence segments.

3. The method of claim 1, wherein, The preprocessing of each of the empty mirror materials to obtain multiple effective empty mirror segments comprises: calculating the interframe visual difference of each of the empty mirror materials, determining whether there is a shot split point in the corresponding empty mirror material based on the interframe visual difference, and if so, splitting the corresponding empty mirror material based on the split point; statistically analyzing the average optical flow and the time length of each of the empty mirror materials; filtering multiple empty mirror materials based on a preset optical flow threshold and a preset time length threshold to obtain multiple effective empty mirror segments.

4. The method of claim 1, wherein, The determination of the fusion score of each of the structured sentence segments based on the scoring rule comprises: determining the semantic importance and the keyword density of each of the structured sentence segments based on the structured sentence segments; determining the character emotion intensity, the face stability score, and the speech signal-to-noise ratio of the synchronous sound video segment corresponding to each of the structured sentence segments; obtaining the fusion score of the structured sentence segment based on the weighted sum of the semantic importance, the keyword density, the character emotion intensity, the face stability score, and the speech signal-to-noise ratio.

5. The method of claim 4, wherein, The determination of whether an effective empty mirror segment needs to be inserted into the synchronous sound video segment corresponding to each of the structured sentence segments and the corresponding editing strategy based on the fusion score comprises: determining whether the fusion score of the structured sentence segment is greater than or equal to a first preset value; if so, no effective empty mirror segment needs to be inserted; if not, an effective empty mirror segment needs to be inserted; in the case where an effective empty mirror material needs to be inserted, if the fusion score of the structured sentence segment is between a second preset value and the first preset value, the front part of the synchronous sound video segment is retained, and the remaining part of the synchronous sound video segment is replaced with the most matched effective empty mirror segment. If the fusion score of the structured sentence segment is less than the second preset value, the synchronous video segment is replaced by the most matched effective empty mirror segment.

6. The method of claim 1, wherein, The determining the most matched effective empty mirror segment based on the matching rule comprises: calculating semantic similarity and entity similarity between the structured sentence segment and each effective empty mirror segment; selecting the most matched effective empty mirror segment from the plurality of effective empty mirror segments based on the semantic similarity and the entity similarity.

7. The method of claim 5, wherein, The inserting the most matched effective empty mirror segment into the corresponding synchronous video based on the editing strategy comprises: determining a cut-in point of the most matched effective empty mirror segment; the cut-in point comprises one of a sentence end frame, a tone pause frame, and a mouth closing frame; inserting the most matched effective empty mirror segment into the corresponding synchronous video based on the cut-in point and the editing strategy.

8. The method of claim 3, wherein, Before the preprocessing each empty mirror material to obtain a plurality of effective empty mirror segments, the method further comprises: detecting compliance and stability of each empty mirror material, and deleting blurred frames, sensitive information frames, and illegal person frames.

9. A video taping system characterized by, The system comprises: an acquisition module configured to acquire a synchronous video set and an empty mirror material set captured by a terminal device; the synchronous video set comprises a plurality of synchronous videos, and the empty mirror material set comprises a plurality of empty mirror materials; a preprocessing module configured to preprocess audio signals in each synchronous video to obtain a plurality of structured sentence segments, and preprocess each empty mirror material to obtain a plurality of effective empty mirror segments; a scoring module configured to determine fusion scores of each structured sentence segment based on a scoring rule, and determine whether an effective empty mirror segment needs to be inserted into a synchronous video segment corresponding to each structured sentence segment and an editing strategy based on the fusion scores; a matching module configured to, if yes, determine the most matched effective empty mirror segment for each structured sentence segment based on a matching rule; an editing module configured to insert the most matched effective empty mirror segment into the corresponding synchronous video segment based on the editing strategy, and splice and render each synchronous video to obtain a finished video.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the method of any one of claims 1 to 8.

Images