Video editing method based on multi-model collaboration and related equipment

Abstract

The invention discloses a video editing method and related equipment based on multi-model collaboration, and the method comprises the steps: building an analysis task after receiving a subtitle file to be edited and a video file to be edited, firstly, a video semantic matching model in a GPU cloud server is called to analyze a to-be-edited subtitle file and a to-be-edited video file to obtain a target video clip, then a video analysis model is called to carry out subject object recognition on the target video clip to obtain preset parameters, and finally, according to the preset parameters and the target video clip, the subtitle file to be edited and the video file to be edited are edited. Calling a video fine-tuning model to perform video fine-tuning and then outputting a synthesized video; the to-be-edited subtitle file and the to-be-edited video file are automatically processed through multi-model cooperation, and finally the synthesized video file is output, so that the technical problems of long editing time, high professional requirement and poor applicability existing in editing of the video file in the related technology are solved.

Description

Technical Field

[0001] This invention relates to the field of video data technology, and in particular to a video editing method and related equipment based on multi-model collaboration. Background Technology

[0002] With the rise of the internet and the self-media industry, online content promotion has largely shifted from text-based to video-based dissemination. Currently, most people browse and view content through short videos. In related technologies, video creation typically requires manual editing, such as manually identifying key segments, adjusting shot composition, or manipulating the position and proportion of main objects in a video using video editing software. This method is time-consuming and requires skilled personnel to master video editing software, leading to low efficiency and poor applicability in video editing.

[0003] Therefore, how to solve the technical problems existing in traditional video editing methods in related technologies has become a technical challenge that needs to be overcome by those skilled in the art. Summary of the Invention

[0004] This invention proposes a video editing method and related equipment based on multi-model collaboration to solve the technical problems of long video editing time, high professional requirements, and poor applicability in related technologies.

[0005] In a first aspect, one embodiment of the present invention provides a video editing method based on multi-model collaboration, comprising: Receive the subtitle file and video file to be edited, and create a parsing task; The parsing task calls the video semantic matching model in the GPU cloud server to parse and process the subtitle file to be edited and the video file to be edited in order to obtain the target video segment; The video parsing model in the GPU cloud server is invoked to identify the main object in the target video segment and obtain the preset parameters of the main object; Based on the preset parameters and the target video segment, the video fine-tuning model in the GPU cloud server is invoked to perform video fine-tuning and then output a synthesized video file.

[0006] The video editing method based on a multi-model system according to the embodiments of the present invention has at least the following beneficial effects: This invention discloses a video editing method based on a multi-model system. After receiving the subtitle file and the video file to be edited, a parsing task is created. First, a video semantic matching model in a GPU cloud server is called to parse the subtitle file and the video file to be edited to obtain a target video segment. Then, a video parsing model is called to identify the main object of the target video segment to obtain preset parameters. Finally, based on the preset parameters and the target video segment, a video fine-tuning model is called to fine-tune the video and output a composite video. By automatically processing the subtitle file and the video file to be edited through multi-model collaboration and finally outputting a composite video file, this method solves the technical problems of long editing time, high professional requirements, and poor applicability in related technologies for editing video files. It provides an efficient, automated, and highly applicable video editing method based on multi-model collaboration.

[0007] According to other embodiments of the video editing method based on multi-model collaboration of the present invention, the video semantic matching model includes the CLIP model; the step of calling the video semantic matching model in the GPU cloud server to parse the subtitle file to be edited and the video file to be edited according to the parsing task includes: The CLIP model is invoked to parse the subtitle file to be edited line by line, using each line of subtitle as a keyword; Obtain the corresponding video frame segments based on the keywords; The target video segment is obtained by performing mathematical calculations based on the keywords and the video frame segments.

[0008] According to other embodiments of the video editing method based on multi-model collaboration of the present invention, the step of obtaining the target video segment by performing mathematical calculations based on the keywords and the video frame segments includes: Feature vectors are extracted and cosine similarity is calculated from the keywords and video frame segments to obtain multiple subtitle video similarity values; The target video segment is obtained by filtering based on multiple subtitle video similarity values.

[0009] According to other embodiments of the video editing method based on multi-model collaboration of the present invention, the step of filtering based on multiple subtitle similarity values ​​to obtain the target video segment includes: Based on the multiple subtitle video similarity values, the top three video segments with the highest similarity are obtained; The target video segment is synthesized based on the timestamps corresponding to the top three video segments with the highest similarity.

[0010] According to other embodiments of the video editing method based on multi-model collaboration of the present invention, the video parsing model includes the ADPN-MM model; the preset parameters include one or more of the following: person subject, person trajectory, object subject, or object trajectory; The step of calling the video parsing model in the GPU cloud server to perform subject object recognition on the target video segment includes: The ADPN-MM model is invoked to perform subject object recognition on the target video segment, obtaining one or more of the following: the person subject, the person trajectory, the object subject, or the object trajectory.

[0011] According to other embodiments of the present invention, the video fine-tuning model includes a PaddlePaddle model; the step of calling the video fine-tuning model in the GPU cloud server to perform video fine-tuning based on the preset parameters and the target video segment, and then outputting a synthesized video file includes: The PaddlePaddle model is invoked to perform video fine-tuning on one or more of the following: the main character, the trajectory of the character, the main object, or the trajectory of the object, according to preset trigger conditions.

[0012] According to other embodiments of the video editing method based on multi-model collaboration of the present invention, the step of calling the PaddlePaddle model to perform video fine-tuning on one or more of the main character, the trajectory of the main character, the main object, or the trajectory of the object according to preset triggering conditions includes: If a single subject moves within the frame, it triggers a minor centering adjustment. If the subject is located at the edge of the frame, a mirror adjustment is triggered; If the main subject occupies less than the preset ratio in the frame, the subject will be enlarged and fine-tuned. If multiple entities exist, multi-entity processing fine-tuning is triggered; If the subject is stationary and you need to add dynamism to the image, then trigger static movement fine-tuning.

[0013] Secondly, one embodiment of the present invention provides a video editing apparatus based on multi-model collaboration, comprising: The task creation module is used to receive subtitle files and video files to be edited, and to create parsing tasks; The video semantic matching module is used to call the video semantic matching model in the GPU cloud server to parse the subtitle file to be edited and the video file to be edited according to the parsing task, so as to obtain the target video segment; The video parsing module is used to call the video parsing model in the GPU cloud server to identify the main object of the target video segment and obtain the preset parameters of the main object; The video fine-tuning module is used to fine-tune the video based on the preset parameters and the target video segment by calling the video fine-tuning model in the GPU cloud server and then outputting a synthesized video file.

[0014] Thirdly, one embodiment of the present invention provides a video editing system based on multi-model collaboration, including: an ESC server and a GPU cloud server; The ESC server is communicatively connected to the GPU cloud server, enabling the ESC server to execute the multi-model collaborative video editing method described above.

[0015] Fourthly, one embodiment of the present invention provides a computer-readable storage medium storing an executable program, which is executed by a processor to implement the multi-model collaborative video editing method described above. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the steps of a specific embodiment of a video editing method based on multi-model collaboration according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a specific embodiment of a video editing method based on multi-model collaboration according to an embodiment of the present invention, in which step S200 includes sub-steps; Figure 3 This is a schematic diagram of a specific embodiment of a video editing method based on multi-model collaboration according to an embodiment of the present invention, in which step S230 includes sub-steps; Figure 4 This is a schematic diagram of a specific embodiment of a video editing method based on multi-model collaboration according to an embodiment of the present invention, in which step S232 includes sub-steps; Figure 5 This is a schematic diagram of a specific embodiment of a video editing method based on multi-model collaboration according to an embodiment of the present invention, in which step S300 includes sub-steps; Figure 6 This is a schematic diagram of a specific embodiment of a video editing method based on multi-model collaboration according to an embodiment of the present invention, in which step S400 includes sub-steps; Figure 7 This is a schematic diagram of the module composition of a specific embodiment of a video editing device based on multi-model collaboration according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a specific embodiment of a video editing system based on multi-model collaboration according to an embodiment of the present invention. Detailed Implementation

[0017] The following will describe the inventive concept and its resulting technical effects clearly and completely with reference to embodiments, so as to fully understand the purpose, features and effects of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention.

[0018] In the description of the embodiments of the present invention, the term "several" means one or more, and the term "multiple" means two or more. The terms "greater than," "less than," and "exceeding" should be understood as excluding the stated number, while the terms "above," "below," and "within" should be understood as including the stated number. The terms "first" and "second" should be understood as distinguishing technical features, and not as indicating or implying relative importance, the number of indicated technical features, or the order of the indicated technical features.

[0019] Reference Figure 1 This invention provides a video editing method based on multi-model collaboration, applied to a video editing system based on multi-model collaboration. The system includes an ECS server and a GPU cloud server connected by communication. The method is applied to the ECS server to call multiple video editing models in the GPU cloud server to implement the video editing process; it includes the following steps: S100: Receive the subtitle file and video file to be edited, and create a parsing task; In this embodiment, the subtitle file to be edited is a standard .srt format subtitle file, and the video file to be edited includes common formats such as .mp4, .mov, and .avi. After the user uploads the subtitle file and video file to be edited through the user browser provided by the ECS server, the ECS server creates a parsing task to parse and process the received subtitle file and video file to output the edited composite video file.

[0020] S200: Based on the parsing task, call the video semantic matching model in the GPU cloud server to process the subtitle file to be edited and the video file to be edited in order to obtain the target video segment; In this embodiment, the GPU cloud server carries a video semantic matching model to provide video editing services to users. After receiving the subtitle file and the video file to be edited, the ESC server transmits them to the GPU cloud server through the backend API task scheduling module. The ESC server then calls the video semantic matching model in the GPU cloud server to perform semantic and video parsing processing on the subtitle file and the video file to be edited to obtain the target video segment.

[0021] S300 calls the video parsing model in the GPU cloud server to identify the main object of the target video segment and obtains the preset parameters of the main object; In this embodiment, after obtaining the target video segment in step S200, it is necessary to identify the main object of the target video segment. The main object is the main object of video editing. In order to quickly edit the target video segment, it is necessary to parse it to obtain the corresponding preset parameters, and then perform subsequent video editing processing based on the obtained preset parameters.

[0022] S400: Based on preset parameters and the target video segment, the video fine-tuning model in the GPU cloud server is called to fine-tune the video and then output the synthesized video file.

[0023] In this embodiment, after obtaining the preset parameters of the main object of the target video segment through step S300, the preset parameters of the main object in the picture are fine-tuned to obtain the edited composite video file.

[0024] This invention provides a video editing method based on multi-model collaboration, which automatically processes the subtitle file and video file to be edited through multi-model collaboration, and finally outputs a composite video file. This solves the technical problems of long editing time, high professional requirements and poor applicability in related technologies for editing video files.

[0025] Reference Figure 2 In some embodiments, in order to accurately obtain video segments in the video file to be edited that match the subtitle file to be edited, the video semantic matching model includes the CLIP model. The CLIP model is used to find the uniquely matching correct image for each text segment. In this embodiment, the above step S200 specifically includes the following sub-steps: S210. Call the CLIP model to parse the subtitle file to be edited line by line, using each line of subtitle as a keyword; The CLIP model analyzes the received subtitle file line by line during parsing and uses each line of subtitles as a keyword for video matching.

[0026] S220. Obtain the corresponding video frame segments based on the keywords; The CLIP model uses each line of subtitles as a keyword, performs keyframe analysis on the video file to be edited based on the keyword, and selects the video frame segment that best corresponds to each keyword.

[0027] S230. Obtain the target video segment by performing mathematical calculations based on keywords and video frame segments.

[0028] In step S220, after obtaining the video frame segment corresponding to each keyword, mathematical calculations are performed on each keyword and its corresponding video frame segment, and the target video segment is selected according to the calculation results.

[0029] Reference Figure 3 In a specific embodiment, to accurately obtain the target video segment and ensure that the final output synthesized video meets the expected requirements, step S230 of the above embodiment specifically includes the following sub-steps: S231. Extract feature vectors and calculate cosine similarity for keywords and video frame segments to obtain multiple subtitle video similarity values; In this process, feature vectors are extracted from each keyword and its corresponding video frame segment. Then, cosine similarity is calculated between the obtained keyword feature vectors and the video frame segments to obtain multiple subtitle video similarity values. At the same time, the start and end times of the video frame segments corresponding to each keyword are marked.

[0030] S232. Filter based on the similarity values ​​of multiple subtitle videos to obtain the target video segment.

[0031] After obtaining multiple subtitle video similarity values ​​in step S231 above, video frame segments that meet preset conditions are selected based on the multiple subtitle video similarity values. Then, the video frame segments that meet the preset conditions are merged according to the start and end times of the marked video frame segments to obtain the target video segment.

[0032] Reference Figure 4 In one specific embodiment, step S232 of the above embodiment specifically includes: S2321. Obtain the three video segments with the highest similarity based on multiple subtitle video similarity values; In step S231, after obtaining multiple subtitle video similarity values, the three video segments with the highest similarity are selected in descending order (i.e., the top three subtitle video similarity values; if there are ties, multiple can be selected or one can be chosen).

[0033] S2322. Synthesize the target video segment based on the timestamps corresponding to the top three video segments with the highest similarity.

[0034] After determining the three corresponding video segments, the target video segment is synthesized based on their corresponding timestamps (i.e., the start and end times of the video frame segments marked in step S232).

[0035] Reference Figure 5 In some embodiments, the video parsing model includes the ADPN-MM model, and the preset parameters include one or more of the following: person subject, person trajectory, object subject, or object trajectory; in the above embodiments, step S300 includes sub-steps: S310: Call the ADPN-MM model in the GPU cloud server to perform subject object recognition on the target video segment, and then obtain one or more of the following: person subject, person trajectory, object subject, or object trajectory.

[0036] In this embodiment, after obtaining the target video segment, although the target video segment has been preliminarily edited, there may be situations such as disharmony between the main object and the screen, unsmooth transition of motion trajectory, or the main object being located at the edge of the screen. Therefore, it is necessary to identify the main object of the target video segment to obtain preset parameters, and then adjust the obtained preset parameters.

[0037] Reference Figure 6 In some embodiments, to accurately achieve video fine-tuning, the video fine-tuning model includes the PaddlePaddle model, and step S400 in the above embodiments includes sub-steps: S410. Call the PaddlePaddle model to perform video fine-tuning on one or more of the main character, the trajectory of the character, the main object, or the trajectory of the object according to preset trigger conditions.

[0038] Among them, the preset trigger conditions for video fine-tuning of one or more of the following: the main character, the character's trajectory, the main object, and the object's trajectory; If a single subject moves within the frame, it triggers a minor centering adjustment. For example, if a single person moves within the frame of a target video clip, the person is fine-tuned to be centered in the frame. After calculating the offset (relative to the frame center) based on the center point of the subject object detection box, the moving window is cropped to center the subject object within the frame.

[0039] If the main subject requires a symmetrical visual effect, then a mirror fine-tuning is triggered; After the subject is centered and fine-tuned, if a symmetrical visual effect is required, the image is horizontally flipped.

[0040] If the main subject occupies a smaller proportion of the image than the preset proportion, then the subject will be enlarged and fine-tuned. The preset scaling ratio can be set according to the user's preferences. Specifically, it is dynamically scaled with the main object detection box as the center. The typical scaling ratio is 1.2-1.5 times that of the main object.

[0041] If multiple subjects exist, multi-subject fine-tuning processing is triggered; Specifically, if there are two main objects, Subject 1 and Subject 2, the video fine-tuning process is to smoothly transition the camera from Subject 1 to Subject 2 along the center of the subject detection frame.

[0042] If the subject is stationary and you need to add dynamism to the image, then trigger static movement fine-tuning; Specifically, the subject moves smoothly by slowly translating along a single axis through the lens window (e.g., from left to right).

[0043] In this embodiment, by setting preset trigger conditions, the target video segment is fine-tuned and then output as a composite video after collaborative editing by multiple models. In this embodiment, after the user uploads the subtitle file and video file to be edited to the ECS server through a browser page provided by the ECS server, the ECS server creates a parsing task, calls multiple models in the GPU cloud server for collaborative editing, and finally sends the composite video file back to the ECS server. The user can then download the composite video file through a browser. This embodiment of the invention provides a video editing method based on multi-model collaboration, which only requires the user to upload the subtitle file and video file to be edited, and automatically and efficiently outputs the corresponding composite video file, solving the technical problems of long editing time, high professional requirements, and poor applicability in related technologies for editing video files.

[0044] Reference Figure 7 This invention also provides a video editing device based on multi-model collaboration, comprising a task creation module, a video semantic matching module, a video parsing module, and a video fine-tuning module. The task creation module receives a subtitle file and a video file to be edited and creates a parsing task. The video semantic matching module uses a video semantic matching model on a GPU cloud server to parse the subtitle file and the video file to be edited, obtaining a target video segment. The video parsing module uses a video parsing model on the GPU cloud server to identify the main object in the target video segment and obtain preset parameters for the main object. The video fine-tuning module uses a video fine-tuning model on the GPU cloud server to perform video fine-tuning based on the preset parameters and the target video segment, and then outputs a synthesized video file.

[0045] In practical applications, the video editing device based on multi-model collaboration in this embodiment of the invention is implemented using an ECS server. It is applied in a video editing system based on multi-model collaboration, which also includes a GPU cloud server. The ECS server communicates with the GPU cloud server. The ECS server is used to implement the video editing method based on multi-model collaboration as described in any of the above embodiments, processing the subtitle file and the video file to be edited, and finally outputting a composite video file. In this embodiment, the process principle of video editing implemented by the ECS server in the video editing system based on multi-model collaboration corresponds to the process principle of the video editing method based on multi-model collaboration in the above embodiments, and will not be elaborated further here.

[0046] ReferenceFigure 8 This invention also provides a video editing system based on multi-model collaboration, comprising an ECS server and a GPU cloud server. The ECS server and GPU cloud server are communicatively connected. The ECS server provides a user browser page, while the GPU cloud server provides CLIP, ADPN-MM, and PaddlePaddle model services. In this embodiment, the user uploads the subtitle file and video file to be edited through the browser provided by the ECS server. The ECS service then executes the multi-model collaborative video editing method described in any of the above embodiments to perform video editing, ultimately outputting a composite video file for the user to download and use. The process principle of the ECS server receiving the subtitle file and video file and performing video editing in this multi-model collaborative video editing system corresponds to the process principle of the multi-model collaborative video editing method in the above embodiments, and will not be repeated here.

[0047] This invention also provides a computer storage medium, wherein the computer-readable storage medium stores a computer-executable program, which, when executed by a processor, implements the multi-model collaborative video editing method described in any of the above embodiments.

[0048] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention. Furthermore, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

Claims

1. A video editing method based on multi-model collaboration, characterized in that, include: Receive the subtitle file and video file to be edited, and create a parsing task; The parsing task calls the video semantic matching model in the GPU cloud server to parse and process the subtitle file to be edited and the video file to be edited in order to obtain the target video segment; The video parsing model in the GPU cloud server is invoked to identify the main object in the target video segment and obtain the preset parameters of the main object; Based on the preset parameters and the target video segment, the video fine-tuning model in the GPU cloud server is invoked to perform video fine-tuning and then output a synthesized video file.

2. The video editing method based on multi-model collaboration according to claim 1, characterized in that, The video semantic matching model includes the CLIP model; the step of calling the video semantic matching model in the GPU cloud server to parse the subtitle file and the video file to be edited according to the parsing task includes: The CLIP model is invoked to parse the subtitle file to be edited line by line, using each line of subtitle as a keyword; Obtain the corresponding video frame segments based on the keywords; The target video segment is obtained by performing mathematical calculations based on the keywords and the video frame segments.

3. The video editing method based on multi-model collaboration according to claim 2, characterized in that, The step of obtaining the target video segment by performing mathematical calculations based on the keywords and the video frame segments includes: Feature vectors are extracted and cosine similarity is calculated from the keywords and video frame segments to obtain multiple subtitle video similarity values; The target video segment is obtained by filtering based on multiple subtitle video similarity values.

4. The video editing method based on multi-model collaboration according to claim 3, characterized in that, The step of filtering based on multiple subtitle similarity values ​​to obtain the target video segment includes: Based on the multiple subtitle video similarity values, the top three video segments with the highest similarity are obtained; The target video segment is synthesized based on the timestamps corresponding to the top three video segments with the highest similarity.

5. The video editing method based on multi-model collaboration according to any one of claims 1 to 4, characterized in that, The video parsing model includes the ADPN-MM model; the preset parameters include one or more of the following: person subject, person trajectory, object subject, or object trajectory. The step of calling the video parsing model in the GPU cloud server to perform subject object recognition on the target video segment includes: The ADPN-MM model is invoked to perform subject object recognition on the target video segment, obtaining one or more of the following: the person subject, the person trajectory, the object subject, or the object trajectory.

6. The video editing method based on multi-model collaboration according to claim 5, characterized in that, The video fine-tuning model includes the PaddlePaddle model; the step of calling the video fine-tuning model in the GPU cloud server to perform video fine-tuning based on the preset parameters and the target video segment, and then outputting the synthesized video file includes: The PaddlePaddle model is invoked to perform video fine-tuning on one or more of the following: the main character, the trajectory of the character, the main object, or the trajectory of the object, according to preset trigger conditions.

7. The video editing method based on multi-model collaboration according to claim 5, characterized in that, The step of calling the PaddlePaddle model to perform video fine-tuning on one or more of the following elements according to preset trigger conditions: the main character, the trajectory of the main character, the main object, or the trajectory of the object. If a single subject moves within the frame, it triggers a minor centering adjustment. If the subject is located at the edge of the frame, a mirror adjustment is triggered; If the main subject occupies less than the preset ratio in the frame, the subject will be enlarged and fine-tuned. If multiple entities exist, multi-entity processing fine-tuning is triggered; If the subject is stationary and you need to add dynamism to the image, then trigger static movement fine-tuning.

8. A video editing device based on multi-model collaboration, characterized in that, include: The task creation module is used to receive subtitle files and video files to be edited, and to create parsing tasks; The video semantic matching module is used to call the video semantic matching model in the GPU cloud server to parse the subtitle file to be edited and the video file to be edited according to the parsing task, so as to obtain the target video segment; The video parsing module is used to call the video parsing model in the GPU cloud server to identify the main object of the target video segment and obtain the preset parameters of the main object; The video fine-tuning module is used to fine-tune the video based on the preset parameters and the target video segment by calling the video fine-tuning model in the GPU cloud server and then outputting a synthesized video file.

9. A video editing system based on multi-model collaboration, characterized in that, Including ESC servers and GPU cloud servers; The ESC server is communicatively connected to the GPU cloud server, so that the ESC server can execute the video editing method based on multi-model collaboration as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an executable program, which is executed by a processor to implement the multi-model collaborative video editing method as described in any one of claims 1 to 7.

Images