Automatic video editing method and device based on text and shot similarity, and terminal

Abstract

The invention discloses an automatic video editing method and device based on text and shot similarity and a terminal, and belongs to the technical field of artificial intelligence. The method comprises the following steps: determining the end time of a mixed video based on a music interval identification result of specified music; selecting a lens of which the main body label is scenery as a lens head lens; performing similarity analysis with other lenses based on a subject recognition result, a behavior recognition result and a motion calculation result, calculating visual similarity between the lenses based on a deep learning model, recognizing repeated or redundant pictures, and selecting a high-energy lens as an in-chip high-combustion lens; retrieving and intercepting a corresponding lens as a tail lens based on a corresponding segment of the selected end word; and carrying out audio and video mixed cutting and assembling on the specified music, the head lens, the in-chip high-combustion lens and the tail lens. According to the invention, by fusing multi-modal feature analysis and intelligent editing logic generation, efficient and high-quality video automatic production is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent editing technology, and in particular to an automatic video editing method, device, intelligent terminal and storage medium based on text and shot similarity. Background Art

[0002] Amid the explosive growth of short video content, users are increasingly demanding efficient and high-quality video editing. Current mainstream video editing technologies fall into two categories: traditional manual editing and automated editing based on single features. Traditional editing relies on manual labor, requiring significant time to screen footage, adjust shot order, and add special effects, resulting in low efficiency and high costs.

[0003] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides an automated video editing method, device, intelligent terminal and storage medium based on text and shot similarity. The present invention realizes efficient and high-quality automated video production by integrating multimodal feature analysis and intelligent editing logic generation. The present invention can be applied to scenarios such as short video creation and film and television drama trailer generation and production, while increasing the attractiveness of content and reducing production costs. It has the advantages of improving editing efficiency and quality, realizing deep fusion of multimodal information, optimizing content coherence and picture expression, automatically adapting to music rhythm, accurately identifying high-energy shots and effectively eliminating redundant pictures.

[0005] This application provides an automated video editing method based on text and shot similarity, including: A. Obtain the specified music, perform music interval recognition and music rhythm recognition on the specified music, and find the rhythm points in the specified music; B. Obtain the video file to be edited, segment the video file into shots, and extract all shots that are in the shot flickering stage to avoid switching shots; C. Extract frames from all extracted shots, perform subject recognition analysis and behavior recognition analysis, and label the analyzed shots with corresponding text labels; and extract frames from all extracted shots for motion calculation; D. Extract frames from all extracted shots, use automatic speech recognition technology to extract video subtitles, emotional tendencies and semantic structures, and select ending words based on the large language model; E. Determine the end time of the mixed video based on the music interval recognition results of the specified music; select the shot with the subject label of scenery as the opening shot; and based on the subject recognition results, behavior recognition results and motion calculation results, perform similarity analysis with other shots, and calculate the visual similarity between shots based on the deep learning model, identify repeated or redundant images, and select high-energy shots as the high-energy shots in the film; and based on the corresponding fragments of the selected ending words, retrieve and intercept the corresponding shots as the ending shots; F. Based on the rhythm points in the specified music found, combined with the text theme and shot similarity results, dynamically plan the shot switching rhythm, insert transition effects to control the coherence of the content, and assemble the specified music, the opening shot, the high-energy shots in the film, and the ending shot for audio and video mixing.

[0006] Furthermore, the present application also proposes that after the step of mixing and assembling the specified music, the opening, the high-energy shots in the film and the ending shots for audio and video, it also includes: controlling the volume balancing, adding fade-in and fade-out effects of the audio and picture to the edited video after the audio and video mixing and assembly; and adding a logo mark to the edited video with the added effects.

[0007] Furthermore, the present application also proposes that the steps of obtaining specified music, performing music interval recognition and music rhythm recognition on the specified music, and finding rhythm points in the specified music include: obtaining specified music, performing music interval recognition on the specified music, and identifying the music intervals of the prelude, chorus, interlude, and outing included in the specified music; performing music rhythm recognition on the music interval, and finding weak beats and strong beats in the specified music, which are used to allow shot switching or high-energy shots to coincide with the music when performing subsequent mixing.

[0008] Furthermore, the present application also proposes that the steps of obtaining a video file to be edited, segmenting the video file into shots, and extracting all shots that avoid the position of switching shots in the lens flickering stage include: obtaining a video file to be edited, segmenting the video file into shots, treating the segment video between each two scene switches as a shot, and extracting all shots of the video file; detecting the position of the flickering stage of each shot, and when the flickering stage position of each shot is at the position of the video switching shot, using the shot segmentation to appropriately extend the position of the segment to avoid the flickering position.

[0009] Furthermore, the present application also proposes that all the extracted shots are framed, subject recognition analysis and behavior recognition analysis are performed, and corresponding text labels are added to the analyzed shots; and the steps of performing motion calculation on all the extracted shots include: pre-defining a high-energy text label list; frame-drawing each segment after the shot segmentation is completed, and performing data analysis on the frame-drawing, the data analysis includes subject recognition analysis, behavior recognition analysis, and motion calculation; wherein, after the subject recognition analysis and behavior recognition analysis of the frame-drawing are completed, the shot analysis results are marked with corresponding labels according to the pre-defined high-energy text label list; when performing motion calculation analysis, the motion energy is calculated according to the shot content, and the more frequent the shot switching, the more violent the shaking, and the more activity, the higher the motion energy; the steps of frame-drawing all the extracted shots, using automatic speech recognition technology to extract video subtitles, emotional tendencies and semantic structures, and selecting the ending words based on the large language model include: frame-drawing all the extracted shots, using automatic speech recognition technology to extract video subtitles, emotional tendencies and semantic structures, and placing the identified subtitles in a list; and using the large model to select a suitable sentence in the subtitle list as the ending word of the entire mixed-cut video as the ending.

[0010] Furthermore, the present application also proposes that, based on the result of music interval recognition of specified music, the end time of the mashup video is determined; the shot with the subject label of scenery is selected as the opening shot; and based on the subject recognition result, behavior recognition result and motion calculation result, similarity analysis is performed with other shots, and the visual similarity between shots is calculated based on the deep learning model, repeated or redundant pictures are identified, and high-energy shots are selected as the high-energy shots in the film; and based on the corresponding fragment of the selected ending word, the corresponding shot is retrieved and intercepted as the ending shot. The steps include: based on the result of music interval recognition of specified music, the end time of the mashup video is determined, and according to the music rhythm, the end time of the mashup video is determined. Based on the results of the beat analysis, the high-energy shots are controlled to accurately hit the music beat; based on the results of the shot analysis, the shots with the subject label of scenery are selected as the opening, and the music beats are controlled as the shot switching points, and the shots with subtitles are eliminated; based on the shot recognition data, the subject recognition results, behavior recognition results and motion calculation results of the shot are obtained, and similarity analysis is performed with other shots, the visual similarity between shots is calculated based on the deep learning model, repeated or redundant pictures are identified, and similar shots with high motion energy are treated as high-energy shots for subsequent editing; based on the end words selected by the large language model, the corresponding fragments of the end words are found, and the corresponding shots are retrieved and intercepted as the ending shots.

[0011] Furthermore, the present application also proposes that, based on the rhythm points found in the specified music, combined with the text theme and the shot similarity results, the shot switching rhythm is dynamically planned, and transition special effects are inserted to control the continuity of the content. The steps of mixing and assembling the specified music, the opening, the high-energy shots in the film and the ending shots for audio and video include: mixing and assembling the specified music, the opening, the high-energy shots in the film and the ending shots for audio and video, and allocating shot duration and switching frequency to different music segments based on the rhythm points found in the specified music, combined with the text theme and the shot similarity results, and based on the music rhythm points; adding transition special effects at the shot switching point, and selecting the special effect type according to the rhythm and emotion: and adjusting the audio parameters at the same time to ensure that the music and the picture action are synchronized.

[0012] Furthermore, the present application also proposes an automatic video editing device based on text and shot similarity, and the technical solution is as follows: the device includes: a music rhythm recognition module, which is used to obtain specified music, and perform music interval recognition and music rhythm recognition on the specified music, and find the rhythm points in the specified music; a shot segmentation module, which is used to obtain the video file to be edited, perform shot segmentation on the video file, and extract all shots in the shot flickering stage to avoid the position of switching shots; a frame analysis module, which is used to extract frames from all extracted shots, perform subject recognition analysis and behavior recognition analysis, and mark the analyzed shots with corresponding text labels; and perform motion calculation on all extracted frames; a text analysis module, which is used to extract frames from all extracted shots, use automatic speech recognition technology to extract video subtitles, emotional tendencies and semantic structures, and The large language model is used to select the ending word; the editing logic generation module is used to determine the end time of the mixed video based on the music interval recognition results of the specified music; the shot with the subject label of scenery is selected as the opening shot; and based on the subject recognition results, behavior recognition results and motion calculation results, similarity analysis is performed with other shots, and the visual similarity between shots is calculated based on the deep learning model, repeated or redundant pictures are identified, and high-energy shots are selected as the high-energy shots in the film; and based on the corresponding fragments of the selected ending words, the corresponding shots are retrieved and intercepted as the ending shots; the assembly module is used to dynamically plan the shot switching rhythm according to the rhythm points in the specified music found, combined with the text theme and the shot similarity results, and insert transition effects to control the coherence of the content, and assemble the specified music, the opening, the high-energy shots in the film and the ending shots for audio and video mixing.

[0013] Furthermore, the present application also proposes an intelligent terminal comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, including one or more programs for running the above method.

[0014] Furthermore, the present application also proposes a computer-readable storage medium, which enables the electronic device to run the above method when the instructions in the storage medium are executed by the processor of the electronic device.

[0015] From the above, it can be seen that the present application provides an automated video editing method, device, intelligent terminal and computer-readable storage medium based on text and shot similarity, which realizes deep coordination of text and visual features by integrating music rhythm recognition, multimodal shot analysis and deep learning similarity calculation, and at the same time optimizes the shot switching rhythm by combining dynamic planning and transition special effects, effectively solving the technical problems of multimodal information fragmentation, low editing efficiency and poor content coherence, and has the advantages of improving editing efficiency and quality, optimizing content coherence and picture expressiveness.

[0016] This invention solves the problems of logical discontinuities and image duplication in automated editing through the collaborative analysis of text and visual features. It also introduces a multi-objective optimization mechanism to improve editing efficiency while meeting the needs of personalized content production. Compared with existing technologies, editing time is reduced by over 60%, and the click-through rate and completion rate of the generated videos are significantly improved. Furthermore, this technology can be expanded to scenarios such as ad placement and paid content previews, balancing content appeal with commercial value through intelligent editing strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 3 is a flowchart of an automated video editing method based on text and shot similarity provided in Example 1 of the present invention.

[0019] Figure 2 This is a flowchart of an automated video editing method based on text and shot similarity provided in Example 2 of the present invention.

[0020] Figure 3 This is a schematic diagram of the data cleaning process of the automatic video editing method based on text and shot similarity provided in Example 2 of the present invention.

[0021] Figure 4 A principle block diagram of an embodiment of an automatic video editing device based on text and shot similarity provided by the present invention.

[0022] Figure 5 This is a block diagram of the internal structure of the smart terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0025] With existing technologies, the short video creation field has long faced the pain point of balancing efficiency and quality. Traditional editing tools rely on manual screening of footage and pacing adjustments, which is time-consuming and labor-intensive. Automated editing technologies are often based on matching single visual features, often resulting in logical discontinuities or repetitive redundancies. For example, when a sports brand needed to produce a product promotional video, existing tools struggled to synchronize the rhythmic changes in the background music with the energy fluctuations of the action shots. This resulted in the misalignment of close-ups of skateboarding moves with the background drum beats in the finished video, as well as the clutter of multiple shots from similar angles, severely impacting the viewing experience.

[0026] In addition, the prior art also has the following defects: Multimodal information fragmentation: The text and visual features are not deeply integrated, resulting in the editing results being difficult to balance in terms of content coherence and visual expressiveness.

[0027] Insufficient personalized adaptation: Failure to dynamically optimize based on user preferences (such as video rhythm, duration, and style) results in a monotonous output content.

[0028] Efficiency bottleneck: Automated editing algorithms are highly complex and rely on high-performance hardware when processing long videos, making them difficult to adapt to mobile devices or lightweight scenarios.

[0029] To address these issues, the applicant's R&D team discovered that multimodal feature fusion and intelligent decision-making mechanisms are key to overcoming existing technological bottlenecks. Analyzing user demand logs revealed that 85% of editing requirements involve synchronizing music rhythm with on-screen action. This led to key considerations: how to establish a mapping relationship between text semantics and visual features, and how to achieve dynamic matching of music structure and shot switching. After multiple algorithmic iterations, a three-dimensional decision-making model based on music beat recognition, shot energy calculation, and semantic tag matching was ultimately developed, successfully resolving the issues of music-on-screen synchronization and content duplication.

[0030] Therefore, the present application proposes an automatic video editing method based on text and shot similarity, which is described in detail in the following embodiments.

[0031] Example 1 like Figure 1 As shown, an automatic video editing method based on text and shot similarity in embodiment 1 of the present invention includes the following steps: Step S100: obtaining designated music, and performing music section recognition and music rhythm recognition on the designated music to find out the rhythm points in the designated music; In this step embodiment, Figure 2 As shown, the designated music is the background music for video editing, and the designated music will be subjected to music interval recognition and music rhythm recognition, that is, in this embodiment, the designated music is subjected to music interval recognition, where the contents of each part of the designated music can be recognized, such as prelude, chorus, interlude, outing, etc. After the music interval recognition is completed, the beat point (rhythm point) recognition and analysis will be continued for these music intervals to find out the up-beat (strong beat) points and down-beat (weak beat) points in the music. The purpose of analyzing the music is to facilitate the subsequent mixing and cutting to allow the camera to switch or the high-energy camera to hit the music, thereby improving the quality of the edited video.

[0032] Step S200: Obtain a video file to be edited, segment the video file into shots, and extract all shots that are in a shot flickering phase so as to avoid shot switching. In this embodiment, the video file to be edited can be a specified movie. Figure 2 As shown, the present invention can perform shot segmentation on the video file to be edited (specified movie), extract all the shots of the movie, and regard the segment between each two scene switches as one shot. The shot analysis is to avoid the position of the switching shot of the mixed video being in the shot flickering stage. If it is in the shot flickering stage, the position of the segment will be appropriately extended by using shot segmentation to avoid the flickering position.

[0033] Step S300: Extract frames from all the extracted shots, perform subject recognition analysis and behavior recognition analysis, and add corresponding text labels to the analyzed shots; and perform motion calculation on all the extracted shots; That is, in the embodiment of this step, a list of high-energy text tags (such as scenery, fighting, mecha, racing, etc.) will be defined first, and then each segment after the previous shot segmentation is completed will be framed, with a frame extraction frequency of one frame every two seconds. These frames will be subjected to data analysis, and the main analyses include subject recognition analysis, behavior recognition analysis, and motion calculation. Among them, the shots after the subject recognition analysis and behavior recognition analysis are completed will be labeled according to the analysis results of the shots and enter the previously defined high-energy text tag list. Motion calculation calculates the motion energy based on the content of the shot. The more frequent the shot switching, the more violent the shaking, and the more activity, the higher the motion energy.

[0034] Step S400: extract frames from all the extracted shots, use automatic speech recognition technology to extract video subtitles, emotional tendencies and semantic structures, and select ending words based on a large language model; In addition to analyzing the motion of the shots, the embodiment of the present invention also analyzes the subtitles that appear in the shots. After identifying the subtitles, they are placed in a list. The large model is then used to select appropriate sentences from the subtitle list as the ending words of the entire mashup video.

[0035] Step S500: Based on the music interval recognition result of the specified music, the end time of the mashup video is determined; a shot with a subject label of scenery is selected as the opening shot; similarity analysis is performed with other shots based on the subject recognition result, behavior recognition result, and motion calculation result, and visual similarity between shots is calculated based on a deep learning model to identify repeated or redundant images, and high-energy shots are selected as the high-energy shots in the film; and based on the corresponding segment of the selected ending word, the corresponding shot is retrieved and intercepted as the ending shot; Step S600: Based on the rhythm points found in the specified music, combined with the text theme and the shot similarity results, the shot switching rhythm is dynamically planned, and transition effects are inserted to control the continuity of the content, and the specified music, the opening shot, the high-energy shot in the film, and the ending shot are mixed and assembled into audio and video.

[0036] In the embodiment of the present invention, steps S500-S600 are steps for assembling a mixed video. Specifically, Figure 2 and Figure 3 As shown, regarding the total length of the edited video, the end time of the mixed video will be calculated based on the music interval recognition results. The entire mixed video will not exceed the time of the specified background music. The high-energy shots will also be stuck on the beat (rhythm) point according to the beat (rhythm) point analysis results.

[0037] In this embodiment, regarding the selection of opening shots of editing: the opening shot is based on the shot analysis results, and the shot with the subject label of scenery after subject recognition analysis is selected as the opening shot. The total length of the opening shot can be set to 10 seconds, and the time limit for each shot is no more than 2 seconds. The music card point is basically achieved (that is, the music card point is the shot switching point), and the shots with subtitles are eliminated.

[0038] Regarding the selection of high-energy shots in a film in an embodiment of the present invention: based on shot recognition data, the subject recognition result, behavior recognition result and motion calculation result of the shot are obtained, and based on these results, similarity analysis is performed with other shots, and similar shots with high motion energy are treated as high-energy shots for subsequent editing.

[0039] In the embodiment of the present invention, the selection of the ending shot is as follows: the ending word is selected based on the large language model, the corresponding segment of the ending word is found, and the corresponding shot is retrieved and intercepted as the ending shot.

[0040] Then the video is assembled, audio and video effects are added, volume is balanced, and fade-in and fade-out effects are added for audio and picture.

[0041] As can be seen from the above, in an embodiment of the present invention, by obtaining the specified music and identifying the rhythm points, the video file is segmented and the flickering shots are excluded, the subject, behavior recognition and motion calculation are realized through frame extraction analysis, the end word is selected by combining speech recognition and a large model, the video structure is determined based on the music interval and the adapted shots are screened, and finally the audio and video synchronization editing is realized through dynamic planning.

[0042] Specifically, music rhythm recognition involves detecting strong and weak beats in music through spectrum analysis and beat tracking algorithms. For example, this involves extracting audio features using Mel-frequency cepstral coefficients, combined with a hidden Markov model for beat prediction, identifying up-beats (strong beats) and down-beats (weak beats) within the music, providing temporal anchors for shot cuts. Shot segmentation involves using optical flow to detect scene transitions. For example, this involves determining shot boundaries when the pixel change rate between adjacent frames exceeds a threshold. Flicker detection algorithms are also used to extend the segmentation interval to avoid interfering segments. Motion calculation involves using optical flow vector fields to calculate motion amplitudes. For example, this involves calculating the average modulus length of pixel displacement vectors between consecutive frames to quantify the dynamic energy of a shot. Semantic label matching involves mapping visual features to text keywords. For example, convolutional neural networks are used to extract image features and align them with the text embedding space through an attention mechanism.

[0043] Specifically, when automatically editing a skiing highlight video, the system first analyzes a selected piece of music, interpreting the drum rhythm of the background music and identifying that each eight beats constitutes a musical segment. When segmenting the original video footage (i.e., the video to be edited), the system automatically skips the three-second flicker caused by fogging in the skiing footage. Frame analysis identifies shots containing "flip" motion characteristics and calculates the peak range of their motion energy values. Combined with speech recognition for keywords such as "challenging the limit," matching ending segments are selected. During the editing process, high-energy shots are precisely aligned with the strong beats of the chorus, and transition shots are inserted into the interlude, ultimately creating a tightly rhythmic, smoothly rendered video mix.

[0044] Compared to existing solutions, which simply splice shots based on their duration, this solution establishes editing logic based on the dual constraints of musical tempo and image energy, addressing the core issue of audio-visual asynchrony. While traditional methods rely on manual semantic labeling, this solution innovatively combines visual recognition and speech analysis to automatically generate multi-dimensional labels, significantly improving editing efficiency. Compared to single-line visual similarity matching, this solution incorporates motion energy calculation, effectively avoiding image duplication caused by similar static compositions.

[0045] Through the above technical solution, this application achieves precise synchronization between music rhythm and camera switching, avoiding the mismatch between high-dynamic shots and soothing music segments. Through multi-dimensional feature fusion analysis, the stacking of repeated composition shots is effectively eliminated, improving the information density of video content. Combined with automated tag generation and a camera optimization mechanism, this significantly shortens the professional-level video production cycle, enabling ordinary users to quickly produce high-quality video content with a sense of rhythm and storyline.

[0046] The present application further proposes performing volume equalization processing in the edited video after the audio and video are mixed and assembled, adding fade-in and fade-out effects of audio and picture, and embedding identification information.

[0047] Among them, volume balancing refers to dynamically adjusting the overall loudness consistency of the audio track, which can be achieved specifically by using a dynamic range compression algorithm. By performing peak detection and gain adjustment on the audio waveform, the auditory discomfort caused by sudden changes in volume between different shots can be eliminated. Among them, the fade-in and fade-out effect refers to the smooth transition of audio and image through a gradual transition, which can be achieved specifically by using a linear interpolation algorithm. For example, a gradual transition interval of 0.5 seconds to 1 second can be set in the audio editing software to make the transition between the beginning and the end of the audio and video materials natural. Among them, logo information embedding refers to superimposing preset graphic or text elements on a specified position on the video screen, which can be achieved specifically by using video synthesis technology, such as pixel-level superposition of the logo layer and the video frame through alpha channel blending.

[0048] Specifically, after completing the initial assembly of audio and video assets, the system first calls the audio processing interface to perform multi-track volume analysis on the remixed video, identifying intervals with sudden volume changes and automatically matching them to the target loudness threshold. It then sets timestamps at shot transition nodes, configuring fade-in and fade-out parameters for each transition based on the distribution of musical rhythm points. Finally, the graphics rendering engine is called to embed a preset logo file into the lower right corner of the video according to coordinate parameters, ensuring that the logo remains visible during playback and does not obscure the main content.

[0049] Compared to existing technologies, traditional editing tools often overlook volume differences between multiple sources after synchronizing audio and video, resulting in abrupt volume jumps in the finished product. Existing automated solutions also rarely consider the standardized embedding of brand logos, failing to meet the copyright management requirements of commercial video production. This solution integrates professional-grade audio processing algorithms and intelligent layer compositing technology to achieve industrial-level quality control of the finished product.

[0050] Through the above technical solution, this application effectively solves the common problems of abrupt sound switching and insufficient brand recognition in remixed videos, so that the automatically generated videos can retain the expressiveness of the original material content while achieving professional production-level audio-visual effects and copyright identification specifications.

[0051] The present application further proposes obtaining designated music, performing music interval recognition on the designated music, and identifying the music intervals of the prelude, chorus, interlude, and outing included in the designated music; performing music rhythm recognition on the music interval, and finding the weak and strong beats in the designated music, which are used to allow shot switching or high-energy shots to coincide with the music when performing subsequent remixing.

[0052] Among them, music interval recognition refers to the process of dividing the selected specified audio file into different structural units according to the function of the paragraph. Specifically, it can be achieved by using spectrum analysis and pattern matching algorithms, and distinguishing the prelude, chorus and other paragraphs by detecting the periodic characteristics of frequency distribution and energy changes. Music rhythm recognition refers to detecting the beat points and the law of changes in strength of the audio signal. Specifically, it can be achieved by using BPM detection algorithm combined with dynamic time warping technology, and identifying weak beat points and strong beat points by analyzing the waveform amplitude change and the spectrum center of gravity offset. Weak beat points refer to the rhythm points used for transition connection in music. Specifically, they can be determined by detecting the periodic position where the waveform amplitude is lower than the average value, and are used to control the timing of regular shot switching. Strong beat points refer to the rhythm points used to emphasize the key points in music. Specifically, they can be determined by detecting the characteristic position where the spectrum energy suddenly increases and the duration exceeds the threshold, and are used to match the timing of high-energy shot cutting.

[0053] Specifically, in step S100, the specified music is first converted into a digital signal through an audio decoder, and the spectral features are extracted using Mel-frequency cepstral coefficients. A hidden Markov model is used to perform pattern matching on a continuous 30-second audio clip. When it is detected that the main melody is repeated and the chord trend meets the chorus characteristics, it is marked as a chorus interval. For rhythm recognition, a dynamic beat tracking algorithm is used to calculate the zero-crossing rate and energy envelope in every 8-millisecond time window, and the position coordinates of the weak beat point and the strong beat point are identified in combination with the preset rhythm template library. For example, in the chorus interval, the position where the energy peak lasts for 120 milliseconds and matches the drum timbre is marked as a strong beat point. These data are stored in the timeline metadata for subsequent mixing and cutting.

[0054] Compared to existing technologies, traditional music editing relies on manual annotation of rhythm points, which suffers from low annotation efficiency and inability to adapt to different musical styles. However, this solution uses automated segmentation and multi-dimensional rhythm analysis to accurately identify the function of musical segments and extract complex rhythmic features. For example, in electronic music, it can effectively distinguish the rhythmic intensity differences between build-up and drop segments, thereby matching different shot switching strategies for different musical segments.

[0055] Through the above technical solution, this application achieves a precise mapping of music structure and video editing logic, achieving millisecond-level synchronization between shot switching points and music beats. For example, the shot with the highest motion energy is cut in at the strong beat of the chorus, while a gradual transition is used at the weak beat of the interlude. This improves the coordination between the video rhythm and the music mood, avoiding the visual discontinuity caused by rhythm misalignment.

[0056] The present application further proposes obtaining a video file that needs to be edited, segmenting the video file into shots, treating the segment video between each two scene switches as a shot, and extracting all the shots of the video file; detecting the position of the flickering stage of each shot, and when the flickering stage position of each shot is at the position of the video switching shot, using shot segmentation to appropriately extend the position of the segment to avoid the flickering position.

[0057] Among them, shot segmentation refers to cutting a continuous video stream into independent shot units by identifying the time points of scene switching in the video. Specifically, it can be implemented by using color histogram change detection or motion vector analysis algorithms to establish a structured video material library for subsequent processing. Among them, the position of the flicker stage refers to the abnormal fluctuation range of the picture brightness caused by the exposure adjustment of the shooting equipment or the sudden change of the ambient light. Specifically, it can be achieved by calculating the frame-by-frame brightness variance combined with the judgment of the difference threshold between adjacent frames, which is used to avoid performing editing operations in areas with unstable picture quality. Among them, the extension of the fragment position refers to extending the fragment interception range forward and backward to the time axis based on the boundary of the shot segmentation. Specifically, it can be implemented by using a boundary fine-tuning algorithm based on the integrity of the shot content to ensure that the switching point avoids abnormal areas of the picture.

[0058] Specifically, during the video editing process, the present application first divides the original video into multiple independent shots using a scene switch detection algorithm. Each shot uses the timestamps of two scene switches as the start and end boundaries to form a clip unit containing complete semantics. Frame-level brightness stability analysis is then performed on each shot unit to locate flickering intervals with sudden exposure changes or light interference. When it is detected that the shot switching point overlaps with the flickering interval, the switching point is offset to a frame position where the picture is stable by dynamically adjusting the segmentation boundary, thereby avoiding the introduction of visual defects during editing assembly.

[0059] Compared to existing technologies, traditional video segmentation methods rely solely on scene-change detection to segment shots, without considering image quality issues caused by shooting anomalies. This solution, however, dynamically calibrates segment boundaries by overlaying brightness stability analysis, effectively eliminating flicker interference caused by equipment or environmental factors while maintaining the semantic integrity of the shot.

[0060] Through the above technical solution, this application solves the problem of image quality degradation caused by improper selection of shot switching points in automated video editing, achieves the goal of improving the visual stability of the edited film while retaining the semantic coherence of the original video, and reduces the need for manual secondary correction.

[0061] The present application further proposes a technical solution for performing subject recognition analysis and behavior recognition analysis on all extracted shot frames and marking corresponding text labels while performing motion calculation on the frames, including: pre-defining a list of high-energy text labels; extracting frames for each completed shot segmentation and performing data analysis, the data analysis including subject recognition analysis, behavior recognition analysis and motion calculation; after completing the subject recognition analysis and behavior recognition analysis, marking the shot analysis results with corresponding labels according to the pre-defined list of high-energy text labels; calculating the motion energy according to the shot content when performing motion calculation analysis, the more frequent the shot switching, the more violent the shaking, and the more activities, the higher the motion energy; using automatic speech recognition technology to extract video subtitles, emotional tendencies and semantic structures and putting the recognized subtitles into a list; using a large language model to select suitable sentences from the subtitle list as the ending words of the mixed-cut video as the ending credits.

[0062] Among them, the high-energy text tag list refers to a pre-set set of keywords used to identify shots with high visual impact. Specifically, it can be generated by analyzing and classifying user behavior data using natural language processing technology, and is used to standardize shot screening criteria. Subject recognition analysis refers to the technology of identifying the main objects in video frames through computer vision algorithms. Specifically, it can be implemented using the YOLO target detection model to establish a structured description of the shot content. Motion calculation refers to the calculation method of quantifying the dynamic changes of the shot. Specifically, it can be implemented by counting the pixel displacement of adjacent frames using the optical flow method, which is used to objectively evaluate the dynamic intensity of the shot. Automatic speech recognition technology refers to the technology of converting video and audio into text. Specifically, it can be implemented using an end-to-end speech recognition model to extract video semantic information. The large language model refers to an artificial intelligence model with the ability to understand and generate text. Specifically, it can be implemented using the GPT series of models to intelligently select sentences that adapt to the ending from massive subtitles.

[0063] Specifically, after the shot is segmented, frames are extracted from each segment at equal intervals. The extracted video frames are fed into an object detection network to identify the subject in the scene and record its frequency of occurrence. Simultaneously, a behavior recognition model is used to analyze the character's action categories. When a combination of key elements that meet the definition of a high-energy label list is detected, such as the simultaneous presence of "running" behavior and "crowd" subjects, the corresponding label is added to the shot. Motion energy is calculated by calculating the standard deviation of the displacement vectors of feature points in adjacent frames. A greater displacement indicates a higher motion energy score. During the subtitle processing stage, the speech recognition results are annotated with positive, neutral, or negative sentiment tags using a sentiment analysis model. This is then combined with a large language model to assess sentence completeness and summary, prioritizing sentences with positive sentiment and summary characteristics as closing credits.

[0064] Compared to existing technologies, traditional methods lack a standardized mechanism for screening high-energy shots, resulting in inconsistent editing quality. This solution utilizes a predefined tag system to achieve structured screening. Existing motion assessments often rely on subjective judgment, but this solution introduces a quantitative computational model to enhance objectivity. Conventional subtitle processing methods only perform text conversion, while this solution combines semantic analysis with large-scale model reasoning to achieve intelligent word selection.

[0065] Through the above technical solution, this application effectively solves the problem of fluctuating editing effects caused by inconsistent high-energy shot recognition standards, realizes accurate screening of dynamic shots through quantitative motion energy calculation, and uses multimodal analysis to improve the matching degree between the ending words and the video theme, thereby reducing the workload of manual secondary adjustments overall and enhancing the expressiveness of the finished content.

[0066] The present application further proposes a technical solution for determining the end time of a mashup video based on the results of music interval recognition of specified music, selecting a shot with a subject label of scenery as the opening shot, performing similarity analysis with other shots based on the subject recognition results, behavior recognition results and motion calculation results, calculating visual similarity through a deep learning model to identify repeated or redundant images and select high-energy shots as the high-energy shots in the film, and retrieving and intercepting the corresponding shots as the ending shots based on the selected ending words.

[0067] Music segment recognition refers to the structural division of music segments identified by analyzing music waveform features and spectral energy distribution. This can be achieved using Mel-frequency cepstral coefficients combined with a dynamic time warping algorithm, and is used to determine the synchronization relationship between the mashup video and the music structure. Shots labeled "landscape" refer to video clips annotated with natural landscape elements through semantic segmentation and object detection of shot keyframes using a convolutional neural network. This can be achieved using a ResNet-50 model combined with a scene classifier, and is used to establish the visual tone of the video opening. High-motion-energy similar shots refer to activity indicators derived by calculating inter-frame pixel displacement using optical flow and analyzing the complexity of object motion within the shot. This can be achieved using a Horn-Schunck optical flow algorithm combined with motion vector histogram analysis, and is used to screen for visually impactful clips. Ending word segments refer to video segments containing sentences with summary semantic structures, obtained by matching speech recognition text with semantic role annotations. This can be achieved using a BERT model combined with dependency parsing, and is used to establish the logical conclusion of the video content.

[0068] Specifically, the music rhythm point analysis module extracts the music's beat intensity curve and drum detection results to generate a time-stamp sequence, dynamically aligning it with the video timeline to precisely match high-energy shot switching actions with strong beats. Shot similarity analysis compares the cosine distances of different shots in feature space, combines motion trajectory similarity metrics to construct a shot relationship graph, and uses graph neural networks to identify repeated content. The end credits retrieval process cross-validates the timestamps of the ending words with the video subtitle timeline, extracting segments that contain corresponding semantics and have stable images to ensure that the ending content resonates with the text theme.

[0069] Compared with existing technologies, traditional video editing tools rely solely on single visual features to match shots, resulting in music timing errors exceeding 200 milliseconds and a probability of over 35% for repeated shots. This solution, by collaboratively analyzing music beats, text semantics, and camera motion, reduces timing errors to under 50 milliseconds and increases the accuracy of repeated shot recognition to over 92%.

[0070] Through the above technical solution, this application achieves millisecond-level synchronization matching of music rhythm and camera switching, effectively avoiding viewing fatigue caused by repeated images. At the same time, it enhances the emotional resonance of video content through a semantically driven end-of-credits retrieval mechanism, solving the problems of rhythm imbalance and logical breaks in traditional automated editing.

[0071] This application further proposes to assemble the specified music, the opening, the high-energy shots in the film and the ending shots into an audio and video mix and cut, and allocate shot duration and switching frequency to different music segments based on the rhythm points in the specified music found, combined with the text theme and shot similarity results, and based on the music rhythm points; add transition effects at the shot switching point, select the special effect type according to the rhythm and emotion, and adjust the audio parameters at the same time to ensure that the music and the picture action are synchronized.

[0072] Among them, the music rhythm point refers to the position of the music beat extracted by audio signal processing technology. Specifically, it can be achieved by using a time-frequency analysis method combined with energy peak detection to determine the reference time axis for shot switching and screen card points. Among them, the text theme and shot similarity result refers to the matching degree after the fusion of video semantic features and visual features extracted by natural language processing technology. Specifically, it can be achieved by using a multimodal embedding vector to calculate the cosine similarity, which is used to screen shot sequences with logically coherent content. Among them, the dynamic planning of shot switching rhythm refers to a sequence optimization method based on time constraints and similarity constraints. Specifically, it can be achieved by solving the optimal editing path using the state transfer equation to balance the shot switching frequency and content fluency. Among them, the transition special effect type refers to the visual transition effect when switching between pictures. Specifically, it can be achieved by using the gradient, zoom or motion blur effects in the predefined special effects library to enhance the naturalness of the connection between different shots.

[0073] Specifically, in the process of audio and video mixing and assembly, the music is first divided into prelude, chorus, and interlude sections, and the playback time of the corresponding shot group is allocated according to the length of each section and the rhythm density. For the chorus section, a high-frequency shot switching strategy is adopted, such as switching the shot every 0.5 seconds, and superimposing a dynamic zoom effect; for the prelude section, a low-frequency switching strategy is adopted, such as switching the shot every 2 seconds, and superimposing a gradual fade effect. In the shot selection stage, by calculating the joint similarity of text labels and visual features, the shots with semantic relevance higher than the threshold and motion energy matching the rhythm intensity are screened to form a candidate shot pool. In the audio parameter adjustment stage, the shot switching time point is fine-tuned based on the beat position to align high-energy scenes such as explosions and jumps with strong beats, and at the same time, dynamic compression technology is used to eliminate audio and video delays.

[0074] Compared to existing editing tools, which typically use fixed switching frequencies or single feature matching strategies, this approach can lead to a disconnect between the music rhythm and the action, or abrupt shot transitions. This solution uses multimodal feature fusion and a dynamic programming algorithm to precisely match the structure of musical segments with the duration of shots. It also utilizes text semantic constraints to avoid logical discontinuities in the images, addressing the existing issues of poor audio-visual synchronization and lack of content coherence.

[0075] Through the above technical solutions, this application realizes the dynamic adaptation of music rhythm and lens switching, reducing the manual adjustment time; reduces the probability of repeated images through multimodal similarity analysis, and improves the information density of the editing content; enhances the viewing experience of the video through the association selection of special effects types and music emotions; and eliminates the phenomenon of audio and video asynchrony caused by differences in equipment performance through real-time adjustment of audio parameters.

[0076] The high-energy video interval of the embodiment of the present application is composed as follows Figure 3 As shown, the opening credits of the edit use landscape shots, the middle credits use a mix of high-energy shots, and the ending credits select shots corresponding to the ending words based on subtitle recognition. The audio component selects a specific piece of music as the background music for the edited video. The opening and middle credits lower the volume of the original film (original edited video) to reflect the selected music rhythm for the music punctuation. At the end of the first chorus of the background music (BGM), the ending credits use the original film soundtrack (i.e., the volume of the selected background music (BGM) is lowered). Finally, the film logo (icon) is added to the end of the edited video.

[0077] As can be seen from the above, this invention solves the problems of logical discontinuity and image duplication in automated editing through the collaborative analysis of text and visual features. At the same time, it introduces a multi-objective optimization mechanism to meet the needs of personalized content production while improving editing efficiency. Compared with existing technologies, editing time is reduced by more than 60%, and the click-through rate and completion rate of the generated videos are significantly improved. In addition, this technology can be extended to scenarios such as advertising placement and paid content previews, balancing content appeal and commercial value conversion through intelligent editing strategies.

[0078] Exemplary devices like Figure 4 As shown, an embodiment of the present invention provides an automatic video editing device based on text and shot similarity, the device comprising: The music rhythm recognition module 310 is used to obtain specified music, perform music interval recognition and music rhythm recognition on the specified music, and find the rhythm points in the specified music; The shot segmentation module 320 is used to obtain a video file to be edited, segment the video file into shots, and extract all shots in the shot flickering stage to avoid switching shots; The frame analysis module 330 is used to extract frames from all the extracted shots, perform subject recognition analysis and behavior recognition analysis, and label the analyzed shots with corresponding text labels; and perform motion calculation on all the extracted frames; The text analysis module 340 is used to extract frames from all the extracted shots, extract video subtitles, emotional tendencies and semantic structures using automatic speech recognition technology, and select end words based on a large language model; The editing logic generation module 350 is configured to determine the end time of the mashup video based on the music interval recognition results of the specified music; select a shot with a subject label of scenery as the opening shot; perform similarity analysis with other shots based on the subject recognition results, behavior recognition results, and motion calculation results; calculate visual similarity between shots based on a deep learning model, identify repeated or redundant images, and select high-energy shots as the high-energy shots in the film; and retrieve and intercept the corresponding shots based on the corresponding segments of the selected ending words as the ending shots; The assembly module 360 is used to dynamically plan the shot switching rhythm based on the rhythm points found in the specified music, combined with the text theme and the shot similarity results, and insert transition effects to control the continuity of the content, and assemble the specified music, the opening, the high-energy shots in the film, and the ending shots into an audio and video mix.

[0079] Among them, the music rhythm recognition module refers to the identification of music paragraphs and beat features through audio signal processing technology. Specifically, it can be achieved by using spectrum analysis combined with beat tracking algorithm. Its function is to provide a time reference for subsequent shot switching. The shot segmentation module refers to the segmentation of video clips based on the scene switching detection algorithm. Specifically, it can be achieved by using optical flow change detection combined with key frame comparison. Its function is to avoid screen flickering interference when switching shots. The subject recognition in the frame extraction analysis module refers to the identification of the main objects in the picture through convolutional neural networks, behavior recognition refers to the analysis of action sequences using timing models, and motion calculation refers to the evaluation of dynamic intensity based on pixel displacement. The three are used together to screen high-energy shots. The ending word selection in the text analysis module refers to the extraction of semantically complete sentences from subtitles using a natural language processing model. Specifically, it can be achieved by using a text coherence scoring algorithm. Its function is to enhance the emotional expression at the end of the video.

[0080] Specifically, the device analyzes the music structure through the music rhythm recognition module and establishes a time coordinate system for shot switching. The shot segmentation module synchronously detects the flickering area of the screen when segmenting the video, and avoids unstable pictures by extending the segmentation interval. The frame extraction analysis module performs multi-dimensional analysis on each shot, in which the motion energy calculation is achieved by counting the amplitude of the motion vector between adjacent frames, and high motion energy shots are marked as candidate materials. The text analysis module inputs the speech recognition results into the language model and filters the ending statement according to the semantic completeness and emotional intensity. The editing logic generation module calculates the visual similarity by comparing the similarity of the shot labels using the twin neural network, and generates an editing sequence after eliminating duplicate pictures. The assembly module dynamically adjusts the shot duration according to the music beat, inserts radial blur transition effects at the strong beat point, and synchronously adjusts the audio gain to align the picture action with the music peak.

[0081] Compared with existing technologies, traditional editing devices usually rely only on a single visual feature to screen shots, resulting in a logical break in the content. This solution achieves the collaborative optimization of multimodal information by establishing an association model between music rhythm, text semantics and visual features. The shot segmentation method in the existing technology does not take into account the problem of screen flickering, while this device significantly improves the quality of the material by dynamically adjusting the segmentation interval. In addition, traditional devices lack an intelligent selection mechanism for the ending sentence. This solution achieves emotional enhancement at the end of the video through language model analysis.

[0082] Through the above technical solutions, this application solves the problem of insufficient multimodal feature fusion in automated video editing, avoids visual discomfort caused by screen flickering, and improves the accuracy of screening high-energy shots. By dynamically matching the music rhythm with the shot switching points, the rhythmic appeal of the video is enhanced. The intelligent end word selection mechanism makes the video ending more complete, and multi-dimensional similarity analysis effectively reduces the use of redundant shots, ultimately achieving high-quality automated video editing.

[0083] Based on the above embodiment, the present invention also provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 5 The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a database connected via a system bus.

[0084] The present application further proposes an intelligent terminal comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, including a method for executing automated video editing based on text and shot similarity.

[0085] Among them, the term "smart terminal" refers to an electronic device with computing and storage capabilities, which can be implemented as a mobile terminal, tablet computer, or portable computer, and is used to run video editing programs and process multimedia data. Memory refers to the physical medium used to store program code and video data, which can be implemented as a solid-state drive or flash memory chip, using non-volatile storage technology to ensure the stability of program operation. The processor refers to the arithmetic unit that executes program instructions, which can be implemented as a multi-core central processing unit or graphics processing unit, accelerating video analysis and processing speed through parallel computing. The program refers to the software code containing video editing logic, which can be developed using a machine learning framework and computer vision library, and realizes automated editing functions through algorithm encapsulation.

[0086] Specifically, during the operation of the smart terminal, the memory loads the program code containing the music rhythm recognition, shot segmentation, and frame extraction analysis modules, and the processor executes music interval recognition, shot segmentation avoidance, subject behavior recognition and other operations through multi-threaded scheduling. When the user imports a music file, the processor calls the music rhythm recognition module to analyze the beat points and music paragraphs, and at the same time uses the shot segmentation module to perform scene segmentation on the video material and avoid flickering frames. In the frame extraction analysis stage, the processor performs image recognition and motion calculation in parallel to generate a shot feature library with text labels. The editing logic generation module dynamically plans the shot switching timing according to the music rhythm points, combines the deep learning model to screen high-energy clips, and finally completes the audio and video synchronous mixing through the assembly module.

[0087] In some specific implementations, the memory can be pre-loaded with a compressed and optimized neural network model, such as a ResNet network that uses quantization technology to reduce model size. The processor can then call upon this model to implement real-time inference when performing subject recognition. In mobile terminal scenarios, the program can adopt a phased processing strategy, for example, first uploading video footage to the cloud for coarse-grained analysis, and then completing fine-grained editing operations locally to reduce computing load.

[0088] Compared with existing technologies, traditional video editing solutions rely on workstation-level hardware, which has performance bottlenecks when processing long videos. However, this solution, through algorithm optimization and modular design, enables smart terminals to complete multimodal feature analysis under limited computing power. Existing technologies do not consider mobile terminal adaptability, resulting in the inability of editing software to run on lightweight devices. This solution adopts a layered processing architecture and model compression technology to effectively reduce memory usage and power consumption. In addition, existing automated editing tools lack a multi-feature fusion mechanism. This solution enhances the logical association of the picture while maintaining the editing rhythm through the collaborative analysis of text tags and visual similarity.

[0089] Through the above technical solutions, this application realizes the completion of high-quality automatic video editing on lightweight equipment, solving the problems of strong hardware dependence and insufficient multimodal feature integration of traditional solutions. A parallel processing architecture is used to improve the efficiency of lens analysis and feature extraction, and a dynamic programming algorithm is used to ensure that the music rhythm and screen switching are accurately matched. Combined with the end-cloud collaborative computing model, it not only guarantees the processing power in complex scenarios, but also maintains the smooth operation of mobile terminals, providing short video creators with a ready-to-use intelligent editing tool.

[0090] The present application further proposes a computer-readable storage medium. When the instructions in the storage medium are executed by the processor of an electronic device, the electronic device is enabled to perform an automated video editing method based on text and shot similarity, the method including: obtaining specified music and identifying its rhythm points and music intervals; performing shot segmentation processing on the video file and avoiding the flickering stage; performing subject recognition, behavior analysis and motion calculation on the shot frames; extracting subtitles and selecting ending words through speech recognition and language models; determining the editing logic according to the music structure and screening high-energy shots; dynamically planning the editing rhythm and performing audio and video assembly in combination with rhythm points and similarity analysis.

[0091] Computer-readable storage media refers to the physical medium used to store executable program code, and can be implemented as solid-state drives, optical disks, or flash memory chips. Its non-volatile storage ensures the persistent availability of editing algorithms. Processor execution refers to the central processing unit performing computations on the editing logic pre-installed in the storage medium. This can be achieved through the use of multi-core processor architectures for parallel computing, thereby supporting the efficient execution of video processing tasks.

[0092] Specifically, once the instructions from the storage medium are loaded by the processor, the electronic device automatically performs multimodal feature fusion processing: first, a timeline baseline is established through music rhythm recognition, followed by intelligent segmentation of the original video to avoid screen flicker interference. The frame extraction analysis phase simultaneously implements subject behavior recognition and motion energy calculation to label the shots with dynamic feature tags. The subtitle extraction and end word selection module constructs a content logic chain through semantic analysis and emotion recognition, ultimately achieving precise matching and transition connection of high-energy shots within the constraints of the music beat. This solution eliminates manual intervention through the solidified storage and standardized execution of algorithmic instructions, achieving standardized editing processes across hardware platforms.

[0093] Compared to existing technologies, which often rely on standalone software to implement functional modules and require manual operation for feature alignment and logic adaptation, this solution, by integrating a multimodal collaborative algorithm through storage media, automatically matches musical, visual, and textual features. Traditional technologies are susceptible to interference from screen flicker when processing shot fragments. This solution intelligently extends the fragment interval to avoid interference areas and ensure visual continuity between shot cuts.

[0094] Through the above technical solutions, this application achieves the rapid deployment of cross-platform automated video editing capabilities, resolving the problem of traditional solutions relying on high-performance hardware and enabling mobile devices to efficiently perform multimodal analysis tasks. The architectural design based on instruction-hardened storage effectively reduces the algorithm's dependency on the operating environment, adapting to lightweight application scenarios while maintaining stable editing quality, and eliminating screen jumps and logical discontinuities caused by multi-feature segmentation.

[0095] As can be seen from the above, this invention provides an automated video editing method based on text and shot similarity. By integrating multimodal feature analysis with intelligent editing logic generation, it achieves efficient, high-quality automated video production. This technology can be applied to scenarios such as short video creation and film and television trailer production, enhancing the appeal of content while reducing production costs.

[0096] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An automated video editing method based on text and shot similarity, characterized in that: include: Obtaining specified music, and performing music section recognition and music rhythm recognition on the specified music, and finding rhythm points in the specified music; Obtain a video file to be edited, segment the video file into shots, and extract all shots that are in a shot flickering stage so as to avoid shot switching; Extract all the extracted shots, perform subject recognition analysis and behavior recognition analysis, and add corresponding text labels to the analyzed shots; and perform motion calculation on all the extracted shots; Extract frames from all the extracted shots, use automatic speech recognition technology to extract video subtitles, emotional tendencies, and semantic structures, and select ending words based on a large language model; Determining an end time of the mashup video based on a result of identifying a music interval of the specified music; Select a shot with the subject label "landscape" as the opening shot; Based on the subject recognition results, behavior recognition results, and motion calculation results, similarity analysis is performed with other shots. The visual similarity between shots is calculated based on a deep learning model to identify repeated or redundant images and select high-energy shots as the most exciting shots in the film. Based on the corresponding fragments of the selected ending words, the corresponding shots are retrieved and intercepted as the ending shots. Based on the rhythm points found in the specified music, combined with the text theme and shot similarity results, the shot switching rhythm is dynamically planned, and transition effects are inserted to control the continuity of the content. The specified music, the opening, the high-energy shots in the film, and the ending shots are mixed and assembled into audio and video.

2. The automatic video editing method based on text and shot similarity according to claim 1, characterized in that: After the step of assembling the specified music, the opening shot, the high-energy shot in the film and the ending shot by audio and video mixing, the following steps are further included: Control the volume balance of the video clips after audio and video mixing, and add fade-in and fade-out effects to the audio and video; And add a logo to the video clips with added effects.

3. The automatic video editing method based on text and shot similarity according to claim 1, characterized in that: The steps of obtaining the specified music, performing music interval recognition and music rhythm recognition on the specified music, and finding the rhythm points in the specified music include: Acquire designated music, perform music section recognition on the designated music, and identify music sections including prelude, chorus, interlude, and outro; The music rhythm of the music interval is identified to find the weak beats and strong beats in the specified music, which are used to make the shots switch or high-energy shots coincide with the music when mixing and editing.

4. The automatic video editing method based on text and shot similarity according to claim 1, characterized in that: The steps of obtaining a video file to be edited, segmenting the video file into shots, and extracting all shots in a shot flickering stage to avoid shot switching include: Obtain a video file to be edited, segment the video file into shots, treat the segment between each two scene changes as a shot, and extract all shots of the video file; The position of the flickering stage of each lens is detected. When the flickering stage position of each lens is at the position of the video switching lens, the position of the segment is appropriately extended by using lens segmentation to avoid the flickering position.

5. The automatic video editing method based on text and shot similarity according to claim 1, characterized in that: The steps of extracting frames from all the extracted shots, performing subject recognition analysis and behavior recognition analysis, and labeling the analyzed shots with corresponding text labels; and performing motion calculation on the extracted frames from all the extracted shots include: Predefine a list of high-energy text labels; Extract frames from each completed shot segmentation, and perform data analysis on the extracted frames, wherein the data analysis includes subject recognition analysis, behavior recognition analysis, and motion calculation; After the subject recognition analysis and behavior recognition analysis of the extracted frames are completed, the shot analysis results are labeled with corresponding labels according to the pre-defined high-energy text label list; When performing motion calculation analysis, motion energy is calculated based on the shot content; The steps of extracting frames from all extracted shots, extracting video subtitles, emotional tendencies, and semantic structures using automatic speech recognition technology, and selecting ending words based on a large language model include: The method extracts frames from all the extracted shots, uses automatic speech recognition technology to extract video subtitles, emotional tendencies and semantic structures, and places the recognized subtitles in a list; And use the big model to select the appropriate sentence from the subtitle list as the ending words of the entire mashup video.

6. The automatic video editing method based on text and shot similarity according to claim 1, characterized in that: determining an end time of the mashup video based on a result of identifying a music interval of the specified music; Select a shot with the subject label "landscape" as the opening shot; Based on the subject recognition results, behavior recognition results, and motion calculation results, similarity analysis is performed with other shots. The visual similarity between shots is calculated based on the deep learning model, repeated or redundant images are identified, and high-energy shots are selected as the highlight shots in the film. Based on the corresponding segment of the selected ending word, the steps of retrieving and intercepting the corresponding shot as the ending shot include: Based on the music interval recognition result of the specified music, the end time of the mashup video is determined, and according to the music rhythm point analysis result, the high-energy shot is controlled to accurately hit the music beat; Based on the shot analysis results, we select shots with the subject label "landscape" as the opening sequence, use the music card points as shot switching points, and exclude shots with subtitles. Based on the shot recognition data, the subject recognition results, behavior recognition results, and motion calculation results of the shot are obtained, and similarity analysis is performed with other shots. The visual similarity between shots is calculated based on the deep learning model, and repeated or redundant images are identified. Similar shots with high motion energy are treated as high-energy shots for subsequent editing. Based on the ending words selected by the large language model, find the corresponding segment of the ending words, retrieve and capture the corresponding shots as the ending shots.

7. The automatic video editing method based on text and shot similarity according to claim 1, characterized in that: The step of dynamically planning the shot switching rhythm based on the rhythm points found in the specified music, combining the text theme and the shot similarity results, inserting transition effects to control the coherence of the content, and performing audio and video mixing and assembly of the specified music, the opening shot, the high-energy shot in the film, and the ending shot includes: The specified music, the opening credits, the high-energy shots in the film and the ending credits are assembled into an audio and video mix. According to the rhythm points in the specified music found, combined with the text theme and the shot similarity results, and according to the music rhythm points, the shot duration and switching frequency are allocated to different music sections; transition effects are added at the shot switching points, and the special effect type is selected according to the rhythm and emotion: at the same time, the audio parameters are adjusted to ensure that the music and the picture action are synchronized.

8. An automatic video editing device based on text and shot similarity, characterized in that: The device comprises: A music rhythm recognition module is used to obtain specified music, perform music interval recognition and music rhythm recognition on the specified music, and find the rhythm points in the specified music; A shot segmentation module is used to obtain a video file to be edited, segment the video file into shots, and extract all shots in the shot flickering stage to avoid switching shots; The frame analysis module is used to extract frames from all the extracted shots, perform subject recognition analysis and behavior recognition analysis, and add corresponding text labels to the analyzed shots; and perform motion calculation on all the extracted frames; The text analysis module is used to extract frames from all extracted shots, use automatic speech recognition technology to extract video subtitles, emotional tendencies, and semantic structures, and select ending words based on a large language model; The editing logic generation module is configured to determine the end time of the mashup video based on the music interval recognition result of the specified music; select a shot with a subject label of scenery as the opening shot; perform similarity analysis with other shots based on the subject recognition result, behavior recognition result, and motion calculation result; calculate the visual similarity between shots based on a deep learning model, identify repeated or redundant images, and select high-energy shots as the high-energy shots in the film; and retrieve and intercept the corresponding shots based on the corresponding segments of the selected ending words as the ending shots; The assembly module is used to dynamically plan the shot switching rhythm based on the rhythm points found in the specified music, combined with the text theme and the shot similarity results, and insert transition effects to control the coherence of the content, and assemble the specified music, the opening, the high-energy shots in the film, and the ending shots into audio and video mixes.

9. An intelligent terminal, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 7.

Images