Video automatic generation method, system and device and storage medium

Abstract

The invention discloses an automatic video generation method, system and device and a storage medium, and belongs to the technical field of video processing, the method comprises the following steps: importing video data to be processed, the video data at least comprising an initial video frame; identifying key elements of the initial video frame, wherein the key elements are elements needing to be monitored in the initial video frame; obtaining popular elements corresponding to the key elements according to the key elements; inserting the popular element into the initial video frame where the key element is located to obtain a target video frame; and generating and exporting final video data according to the target video frame. The popular element can be called according to the key element in the initial video frame, then the popular element is inserted into the initial video frame to obtain the target video frame, and finally the final video data is generated by the target video frame, so that automatic arrangement and automatic generation of the video data containing the popular element are realized, and the overall quality of the video data is improved.

Description

Technical Field

[0001] The present application belongs to the field of video processing technology, and specifically, relates to a method, system, device and storage medium for automatic video generation. Background Art

[0002] With the rise of social media and video sharing platforms, the creation and dissemination of video content has become an increasingly important way of disseminating information. However, the traditional video production process is cumbersome and requires a lot of time for material sorting, feature extraction, creative conception and post-production. Especially in the current pursuit of content novelty and attractiveness, how to quickly capture and use popular elements to integrate into video creation has become the key to improving video influence and dissemination efficiency.

[0003] Therefore, it is particularly important to develop a video generation system that can automatically analyze material features and intelligently integrate popular elements. Summary of the invention

[0004] The technical problem to be solved by the present application is to overcome the deficiencies of the prior art and provide a method, system, device and storage medium for automatic video generation.

[0005] In order to solve the above technical problems, the basic concept of the technical solution adopted in this application is: In a first aspect of the present application, a method for automatically generating a video is provided, the method comprising: Importing video data to be processed, wherein the video data includes at least one initial video frame; Identifying key elements of the initial video frame, wherein the key elements are elements in the initial video frame that need to be monitored; Obtaining popular elements corresponding to the key elements according to the key elements, where the popular elements at least include text, music, special effects, and filters; Inserting the popular element into the initial video frame where the key element is located to obtain a target video frame; Final video data is generated and exported according to the target video frame.

[0006] By adopting the above technical solution, first, the key elements in the initial video frame are identified, the popular elements are called according to the key elements, and then the popular elements are inserted into the initial video frame to obtain the target video frame. Finally, the final video data is generated with the target video frame, thereby realizing automatic organization and automatic generation of video data containing popular elements, and improving the efficiency and video quality of the automatically generated video data.

[0007] In a possible implementation manner, the identifying key elements of the initial video frame includes: Extracting elements contained in the initial video frame to obtain an element set; The same elements in the element set and the standard set are used as key elements. The standard set stores a plurality of elements and popular elements corresponding to each element.

[0008] In a possible implementation manner, extracting elements contained in the initial video frame to obtain an element set includes: The element set is obtained by extracting text, objects, characters, character postures, and background from the initial video frame.

[0009] By adopting the above technical solution, the present application extracts the text, objects, characters, character postures, background and other elements in the initial video frame to obtain an element set, thereby improving the breadth and accuracy of the obtained element set. Furthermore, by comparing an element set containing multiple elements with elements in a standard set, the same elements in the element set and the standard set are used as key elements. When the accuracy and breadth of the element set are guaranteed, the accuracy of the key elements obtained by corresponding matching is also improved.

[0010] In a possible implementation, inserting the popular element into the initial video frame where the key element is located to obtain a target video frame includes: Calculate the similarity between popular elements and key elements; Determining whether the similarity is greater than a preset value; If yes, then using the popular element to replace the key element to obtain the target video frame; If not, the popular element is inserted into the initial video frame and guide marks of key elements and popular elements are established to obtain the target video frame.

[0011] By adopting the above technical solution, after obtaining the popular elements, the similarity between the popular elements and the key elements is calculated and compared with the preset value. If the similarity is greater than the preset value, it means that the popular elements and the key elements are similar or the same. At this time, the key elements are directly replaced by the popular elements to achieve the purpose of removing duplicate elements, reduce the element redundancy in the target video frame, and improve the quality of the target video frame. If the similarity is less than or equal to the preset value, it means that the popular elements and the key elements are not the same. At this time, the key elements and the popular elements are retained and the guide marks of the key elements and the popular elements are established, so that when playing the target video frame, after locking the key elements, the popular elements can be quickly located according to the guide mark, and the popular elements attract more users' attention, thereby achieving the purpose of diversion.

[0012] In a possible implementation, establishing guide marks of the key elements and the popular elements includes: Connecting the key element and the popular element with a line segment; or The key elements and the popular elements are identified by the same color.

[0013] In a possible implementation, the method further includes: Obtaining a popular video frame corresponding to the key element according to the key element; The popular video frame is inserted after the initial video frame where the key element is located to obtain a target video frame.

[0014] By adopting the above technical solution, in addition to being able to insert popular elements into the initial video frames, the present application can also insert popular video frames between the initial video frames, that is, the insertion position of the popular video frames can be accurately located on the video timeline based on the time point when the key elements appear, thereby integrating the popular video frames, improving the quality of the video data, shortening the video production cycle, and improving the efficiency of automatically generated video data.

[0015] In a possible implementation, the method further includes: When the initial video frame corresponds to a plurality of popular video frames, determining whether there is a video frame set by the user among the plurality of popular video frames; If yes, retain the video frame set by the user as the popular video frame and insert it after the initial video frame; If not, the currently most popular video frame is retained and inserted after the initial video frame.

[0016] By adopting the above technical solution, when the initial video frame corresponds to multiple popular video frames, the video frame set by the user is preferentially inserted after the initial video frame, thereby realizing personalized generation of video data; otherwise, the video frame with the highest current popularity is retained and inserted after the initial video frame, avoiding the problem of incoherent content between the two initial video frames caused by multiple popular video frames being inserted after the initial video frame, thereby ensuring natural transition and coherent content, and improving the overall quality of the video.

[0017] In a second aspect of the present application, a video automatic generation system is provided, the system comprising: A data import module, used for importing video data to be processed, wherein the video data includes at least one initial video frame; A data identification module, used to identify key elements of the initial video frame, wherein the key elements are elements in the initial video frame that need to be monitored; A data matching module, used for obtaining a popular element corresponding to the key element according to the key element; A data processing module, used for inserting the popular element into the initial video frame where the key element is located to obtain a target video frame; The data generation module is used to generate and export final video data according to the target video frame.

[0018] In a third aspect of the present application, a video automatic generation device is provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any of the above-mentioned video automatic generation methods is implemented.

[0019] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above-mentioned video automatic generation methods is implemented.

[0020] In summary, the present application includes at least one of the following beneficial technical effects: First, the application extracts the text, objects, characters, character postures, background and other elements in the initial video frame to obtain an element set, compares the element set containing multiple elements with the elements in the standard set, and uses the same elements in the element set and the standard set as key elements to ensure the accuracy of the key elements obtained. Then, popular elements are called based on the key elements, and then the popular elements are inserted into the initial video frame to obtain the target video frame, and finally the target video frame is used to generate the final video data, so as to realize automatic sorting and automatic generation of video data containing popular elements, and improve the efficiency and video quality of the automatically generated video data; The present application can also call popular video frames based on key elements, and accurately locate the insertion position of popular video frames on the video timeline based on the time point when the key elements appear, so as to integrate popular video frames, improve the quality of video data, shorten the video production cycle, and improve the efficiency of automatically generated video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are part of this application and are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application, but do not constitute an improper limitation on this application. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 is a flow chart of a method for automatically generating a video in an embodiment of the present application; Figure 2 It is a block diagram of a video automatic generation system in an embodiment of the present application.

[0022] Explanation of the accompanying drawings: 1. Data import module; 2. Data identification module; 3. Data matching module; 4. Data processing module; 5. Data generation module.

[0023] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present application in any way, but rather to illustrate the concepts of the present application for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the appended examples of the embodiments of the present application. Figure 1-2 , clearly and completely describe the technical solutions in the embodiments.

[0025] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatuses.

[0026] like Figure 1 As shown, the present application provides a method for automatically generating videos, which is mainly used to automatically analyze the multi-dimensional elements such as language, text, images, and character actions of unorganized video materials, and intelligently match and insert them according to the current popular elements, and finally improve and supplement the video materials to generate the required video data, thereby improving the quality of the obtained video data and shortening the cycle of producing video data. Specifically, the steps of the method for automatically generating videos are as follows.

[0027] Step S1: importing video data to be processed, where the video data includes at least one initial video frame.

[0028] The video data to be processed is the unorganized video material uploaded by the user, and the video data to be processed includes at least one initial video frame. This application supports users to import the video data to be processed in different video formats, including but not limited to AVI, MP4, MOV and WMV.

[0029] Step S2: identifying key elements of the initial video frame, where the key elements are elements in the initial video frame that need to be monitored.

[0030] First, extract the elements contained in the initial video frame to obtain an element set. A multi-feature analysis model based on deep learning is used to extract the elements contained in the initial video frame. The elements in the initial video frame include but are not limited to speech, text, objects, character postures, backgrounds, etc. The multi-feature analysis model is obtained by training in advance, and the multi-feature analysis model at least includes a text extraction sub-model, an object extraction sub-model, and a character action extraction sub-model. Among them, the text extraction sub-model is mainly used to convert the speech in the initial video frame into text, and then extract the converted text and the text originally existing in the initial image frame to obtain text-type elements. The object extraction sub-model is mainly used to extract objects and characters in the initial video frame, specifically to identify the edges of objects or characters, and then set an identification frame around the object or character. The identification frame selects the local area of ​​the initial video frame where the object or character is located, and then the object or character in the identification frame is used as an element of the object type. The identification frame here is a virtual frame for marking objects or characters. The character action extraction sub-model is coupled with the object extraction sub-model. When the object extraction sub-model extracts a character, the identification frame and the character in the identification frame are sent to the character action extraction sub-model. The character action extraction sub-model uses the posture estimation algorithm to detect and classify the character's actions, expressions or other interactive behaviors in the identification frame. Furthermore, in the initial image frame, in addition to text, objects, and characters, the remaining area is used as the background, and the background will also be used as one of the elements. The background includes background size, color, brightness and other information.

[0031] Based on the multiple elements obtained by the multi-feature analysis model, the multiple elements belonging to the same initial video frame are taken as an element set. Therefore, when the video data to be processed contains multiple initial video frames, an element set corresponding to each initial video frame will be obtained, that is, multiple element sets will be obtained.

[0032] Then, according to the sequence of the initial video frames in the video data to be processed, the element set corresponding to the initial video frame is matched in the standard set to see whether there are elements identical to the elements in the element set. If there are identical elements, the identical elements are used as the key elements in the initial video frame.

[0033] It should be noted that the standard set stores multiple elements and popular elements corresponding to each element. Popular elements can be based on big data analysis, real-time monitoring and analysis of hot topics, popular trends, popular music, and elements involved in filter effects on current social media and video platforms, such as widely discussed titles, topics, and other texts, popular trends, and background music used in popular music, as well as special effects, filters, etc. in filter effects, that is, popular elements include but are not limited to text, music, special effects, and filters. At the same time, the elements that trigger these popular elements or are related to these popular elements are set as elements corresponding to popular elements, and whether an element is the key to triggering popular elements can be determined by the frequency of the simultaneous appearance of the element and the popular element. For example, if the number of times that element A and popular element B appear on the hot topic at the same time is greater than 1,000 times, it means that element A has a corresponding relationship with popular element B, and both element A and popular element B will be stored in the standard set. In practical applications, other judgment methods can also be used to judge whether an element is the key to triggering popular elements, and this application does not limit it.

[0034] It should also be noted that the elements in the standard set can also correspond to video frames. The judgment condition is also that when the number of times the element and the video frame appear on the hot topic is greater than a preset number, the element and the video frame are bound, and the binding relationship between the two is stored in the standard set. For easy distinction, the video frames stored in the standard set are called popular video frames.

[0035] In actual applications, the popular elements and popular video frames in the standard set can also be video frames that have been sorted and cropped in advance by the user. However, the user has not determined the specific insertion position, so they are stored in the standard set first. When elements that have a corresponding relationship with the popular elements and popular video frames appear, the popular elements and popular video frames are called and the insertion operation is performed.

[0036] Therefore, according to the elements in the initial video frame, popular elements and / or popular video frames are matched from the standard set.

[0037] In addition, the elements in the standard set and the popular elements and popular video frames corresponding to the elements are updated periodically, that is, the elements involved in the hot topics, popular trends, popular music, and filter effects on the current social media and video platforms are monitored and analyzed again at intervals, and the elements in the standard set and the popular elements and popular video frames corresponding to the elements are dynamically updated according to the monitoring results.

[0038] Step S3: Obtain popular elements and / or popular video frames corresponding to the key elements according to the key elements.

[0039] Since the standard set stores elements and popular elements and popular video frames corresponding to the elements, and the key elements refer to the elements in the element set that are the same as the elements in the standard set, the key elements also have a corresponding relationship with the popular elements and popular video frames. Therefore, after obtaining the key elements, the popular elements and / or popular video frames corresponding to the key elements are obtained according to the corresponding relationship between the elements in the standard set and the popular elements and popular video frames.

[0040] Step S4: insert the popular element into the initial video frame where the key element is located and / or insert the popular video frame after the initial video frame where the key element is located, to obtain a target video frame.

[0041] The process of inserting popular elements is: First, calculate the similarity between popular elements and key elements. This application adopts a similarity calculation model based on deep learning, which is obtained after pre-training and optimization. The similarity calculation model can analyze and calculate the similarity between key elements and popular elements. The closer the key elements and popular elements are to each other, the higher the similarity. For example, if the key element is text and the popular element is also text, the two are related. If the meaning of the text of the two is the same or the text is exactly the same, the similarity is 100%.

[0042] Then, it is determined whether the similarity between the popular element and the key element is greater than a preset value, which is set in advance to evaluate whether the key element and the popular element are similar or identical. The preset value set in this application is 80%, that is, when the similarity between the popular element and the key element is greater than 80%, it means that the two are similar or identical and can be replaced by each other.

[0043] If the comparison result shows that the similarity is greater than the preset value, it means that the popular element and the key element are similar or the same. In this case, the key element is directly replaced by the popular element, and the popular element is inserted into the position where the key element is located. The initial video frame into which the popular element is inserted is used as the target video frame, so that there are no repeated elements in the target video frame, the element redundancy in the obtained target video frame is reduced, and the quality of the target video frame is improved; If the comparison result shows that the similarity is less than or equal to the preset value, it means that the popular element and the key element are not the same. At this time, the key element and the popular element are retained and the guide marks of the key element and the popular element are established. The establishment of the guide mark includes but is not limited to: connecting the key element and the popular element with a line segment between the key element and the popular element, or using the same color to identify the key element and the popular element. Then, the initial video frame with the guide marks of the key element and the popular element is used as the target video frame, so that when the target video frame is played later, after locking the key element, the popular element can be quickly located according to the guide mark, so that the popular element can attract more users' attention, improve the influence and dissemination efficiency of video data, and achieve the purpose of diversion.

[0044] In actual applications, popular elements can also be set to float on top of key elements, or popular elements can be connected with key elements. For example, when the character's gesture is "OK", the popular elements are placed in the area adjacent to the gesture so that they can be quickly locked onto the popular elements based on the character's gesture.

[0045] The insertion process of popular video frames is: After obtaining the popular video frame, the popular video frame is inserted after the initial video frame where the key element is located, and the inserted popular video frame is used as the target video frame. Therefore, the present application can accurately locate the insertion position of the popular video frame on the video timeline based on the time point when the key element appears, thereby shortening the video production cycle.

[0046] It should be noted that when multiple key elements appear on the same initial video frame, and the multiple key elements correspond to different popular video frames, it is determined whether the popular video frame is a video frame set by the user. If so, the user-defined video frame is preferentially inserted after the initial video frame. Otherwise, the most popular popular video frame is retained and inserted after the initial video frame to avoid the problem of inconsistent content between the two initial video frames caused by inserting multiple popular video frames after the initial video frame. This ensures natural transition and coherent content to improve the overall quality of the video.

[0047] Step S5: Generate and export final video data according to the target video frame.

[0048] After obtaining the target video frame, the original initial video frame is replaced by the target video frame, or the target video frame is inserted after the initial video frame, and then output in sequence according to the sequence between the video frames, so as to obtain the final video data.

[0049] In summary, the implementation principle of a method for automatic video generation in the present application is as follows: first, extract the text, objects, characters, character postures, background and other elements in the initial video frame to obtain an element set, use the element set containing multiple elements to compare with the elements in the standard set, and use the elements in the element set that are the same as those in the standard set as key elements to ensure the accuracy of the key elements obtained. Then, based on the key elements, call popular elements and / or popular video frames, insert the popular elements into the initial video frame where the key elements are located and / or insert the popular video frame into the initial video frame where the key elements are located, and obtain the target video frame. Finally, use the target video frame as the basis for generating the final video data, realize automatic sorting and automatic generation of video data containing popular elements and popular video frames, and improve the efficiency of automatically generated video data and video quality.

[0050] like Figure 2 As shown, the present application provides a video automatic generation system, which includes a data import module 1, a data identification module 2, a data matching module 3, a data processing module 4 and a data generation module 5.

[0051] The data import module 1 is used to import the video data to be processed, and the video data includes at least one initial video frame.

[0052] The data identification module 2 is used to identify key elements of the initial video frame, where the key elements are elements in the initial video frame that need to be monitored.

[0053] The data matching module 3 is used to obtain the popular elements corresponding to the key elements according to the key elements.

[0054] The data processing module 4 is used to insert the popular element into the initial video frame where the key element is located to obtain the target video frame.

[0055] The data generation module 5 is used to generate and export final video data according to the target video frame.

[0056] The modules involved in the embodiments described in the present application may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a data import module 1, a data identification module 2, a data matching module 3, a data processing module 4 and a data generation module 5. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the data import module 1 may also be described as a "module for importing video data to be processed".

[0057] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0058] In order to better execute the program of the above method, the present application also provides a video automatic generation device, which includes a memory and a processor.

[0059] The memory may be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the above-mentioned video automatic generation method, etc.; the data storage area may store data involved in the above-mentioned video automatic generation method, etc.

[0060] The processor may include one or more processing cores. The processor calls the data stored in the memory by running or executing the instructions, programs, code sets or instruction sets stored in the memory, performs various functions of the present application and processes data. The processor may be at least one of a special purpose integrated circuit, a digital signal processor, a digital signal processing device, a programmable logic device, a field programmable gate array, a central processing unit, a controller, a microcontroller and a microprocessor. It is understandable that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and the embodiments of the present application are not specifically limited.

[0061] The present application also provides a computer-readable storage medium, for example, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the above-mentioned video automatic generation method.

[0062] The above is only a preferred embodiment of the present application and does not constitute any form of limitation to the present application. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any technician familiar with the present application can make some changes or modifications to equivalent embodiments of the technical contents suggested above without departing from the scope of the technical solution of the present application. The implementation schemes in the above embodiments can also be further combined or replaced. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the solution of the present application.

Claims

1. A method for automatically generating a video, characterized in that: include: Importing video data to be processed, wherein the video data includes at least one initial video frame; Identifying key elements of the initial video frame, wherein the key elements are elements in the initial video frame that need to be monitored; Obtaining popular elements corresponding to the key elements according to the key elements, where the popular elements at least include text, music, special effects, and filters; Inserting the popular element into the initial video frame where the key element is located to obtain a target video frame; Final video data is generated and exported according to the target video frame.

2. The video automatic generation method according to claim 1, characterized in that: The identifying key elements of the initial video frame includes: Extracting elements contained in the initial video frame to obtain an element set; The same elements in the element set and the standard set are used as key elements. The standard set stores a plurality of elements and popular elements corresponding to each element.

3. The video automatic generation method according to claim 2, characterized in that: The step of extracting the elements contained in the initial video frame to obtain an element set includes: The element set is obtained by extracting text, objects, characters, character postures, and background from the initial video frame.

4. The video automatic generation method according to any one of claims 1 to 3, characterized in that: The step of inserting the popular element into the initial video frame where the key element is located to obtain a target video frame includes: Calculate the similarity between popular elements and key elements; Determining whether the similarity is greater than a preset value; If yes, then using the popular element to replace the key element to obtain the target video frame; If not, the popular element is inserted into the initial video frame and guide marks of key elements and popular elements are established to obtain the target video frame.

5. The video automatic generation method according to claim 4, characterized in that: Establishing guide marks for the key elements and the popular elements includes: Connecting the key element and the popular element with a line segment; or The key elements and the popular elements are identified by the same color.

6. The video automatic generation method according to claim 1, characterized in that: The method further comprises: Obtaining a popular video frame corresponding to the key element according to the key element; The popular video frame is inserted after the initial video frame where the key element is located to obtain a target video frame.

7. The video automatic generation method according to claim 6, characterized in that: The method further comprises: When the initial video frame corresponds to a plurality of popular video frames, determining whether there is a video frame set by the user among the plurality of popular video frames; If yes, retain the video frame set by the user as the popular video frame and insert it after the initial video frame; If not, the currently most popular popular video frame is retained and inserted after the initial video frame.

8. A video automatic generation system, characterized in that: include: A data import module (1), used for importing video data to be processed, wherein the video data comprises at least one initial video frame; A data identification module (2), used for identifying key elements of the initial video frame, wherein the key elements are elements in the initial video frame that need to be monitored; A data matching module (3), used for obtaining a popular element corresponding to the key element according to the key element; A data processing module (4) is used to insert the popular element into the initial video frame where the key element is located to obtain a target video frame; A data generation module (5) is used to generate and export final video data according to the target video frame.

9. A video automatic generation device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images