A method and system for intelligent image-to-video conversion based on structured video data

Abstract

This invention discloses an intelligent text-to-video conversion method and system based on structured video data. The method includes transcoding, shot segmentation, and content recognition of video files in a large-scale video dataset; manually reviewing and modifying inaccurate recognition results before storing them in a structured form in a database to generate a material library; processing user-submitted text based on text type, performing paragraph splitting, keyword extraction, and named entity recognition on the processing results; generating narration audio and corresponding subtitle files based on the processing results; matching the keyword extraction and named entity recognition results with the materials in the material library to obtain the best matching material; and merging the best matching material with the narration audio and subtitle files. The advantages are: solving the tedious and time-consuming problem of material organization and production processes in video production, and improving video production efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer video synthesis and artificial intelligence content generation technology, and in particular to a method and system for intelligent text-to-video conversion based on structured video data. Background Technology

[0002] For copywriters, video production is a professional field with a high entry barrier. Traditional video production requires writing scripts, collecting and editing materials to build a material library, and then going through processes such as rough editing, fine editing, audio track synthesis, and proofreading to complete the video production. The whole process is time-consuming and cumbersome, which cannot meet the needs of today's era of information overload. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for intelligent text-to-video conversion based on video structured data, thereby solving the aforementioned problems existing in the prior art.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A method for intelligent text-to-video conversion based on structured video data includes the following steps:

[0006] S1. Establish a resource library:

[0007] The system transcodes, segments, and identifies video files in a large-scale video dataset. Inaccurate identification results are manually reviewed and corrected. The accurate identification results and the reviewed and corrected results are stored in a structured form in the database to generate a material library.

[0008] S2. Graphical Analysis:

[0009] The system processes user-submitted text based on its text type, performs paragraph splitting, keyword extraction, and named entity recognition on the processed results, and generates narration audio and corresponding subtitle files based on the results.

[0010] S3, Material Matching:

[0011] The results of keyword extraction and named entity recognition based on paragraphs are matched with materials in the material library to obtain the best matching material;

[0012] S4, Video Compositing:

[0013] The video compositing algorithm is used to merge the best-matching footage with the narration and corresponding subtitle files to generate a complete video file.

[0014] Preferably, step S1 specifically includes the following:

[0015] S11. Convert the video file bitstream to a preset format bitstream to achieve transcoding of all video files in a large-scale video dataset;

[0016] S12. Determine whether the transcoded video file needs to be segmented. If so, segment the transcoded video file into short video segments and proceed to step S13; otherwise, proceed directly to step S13.

[0017] If the cosine similarity between two adjacent frames of a video file is greater than or equal to the similarity threshold, then the video file needs to be segmented; otherwise, segmentation is not necessary.

[0018] S13. Recognize the shots, people, scenes, events, objects and subtitles appearing in short video clips;

[0019] S14. Determine the accuracy of the recognition results and manually review and modify inaccurate recognition results, adding subjective descriptive information including time, location, people, and events;

[0020] S15. Store the accurate recognition results and the recognition results after manual review in a structured JSON format in the database to generate a material library.

[0021] Preferably, step S13 specifically involves constructing multiple deep learning models, including shot recognition, face recognition, OCR recognition, and speech recognition, through a deep learning convolutional neural network; and using these deep learning models to extract shot information, character information, scene information, event information, object information, and subtitle information from short video clips.

[0022] Preferably, step S14 specifically includes the following:

[0023] S141. Extract keyframes from short video clips to obtain images of people, scenes, and objects. Calculate the similarity between the images of people, scenes, and objects and sample images in the face image library, scene image library, and object image library, respectively. Determine whether the similarity score is greater than or equal to the score threshold. If it is, the recognition is accurate; otherwise, the recognition is inaccurate.

[0024] S142. Manually review and modify inaccurate identification information, and add subjective descriptive information that cannot be recognized by the computer.

[0025] Preferably, step S2 specifically includes the following:

[0026] S21. Determine whether the text submitted by the user is plain text or a web link. If it is plain text, proceed directly to step S22. If it is a web link, extract the web page text and image content, format the extracted content to remove html tags and meaningless characters, and then proceed to step S22.

[0027] S22. Split the text into paragraphs, and extract keywords and named entities from the split text.

[0028] S23. Using speech synthesis technology, the text processed in step S21 is converted into narration and a corresponding subtitle file is generated.

[0029] Preferably, step S22 specifically includes the following:

[0030] S221. Use the TextRank algorithm to extract key sentences from the text, and segment the text into paragraphs based on the key sentences.

[0031] S222. Extract keywords and named entities from the segmented paragraphs. Keywords include time, scene, person, and event. Named entities include person name, place name, organization name, and verb. Paragraphs that do not meet the word count threshold or that have not had keywords and / or named entities extracted are merged.

[0032] Preferably, step S23 specifically involves synthesizing sound using speech synthesis technology, performing grammatical analysis on the text processed in step S21 to extract the subject, predicate, and object, converting them into speech waveforms, and then using time-domain waveform splicing technology based on the PSOLA method to synthesize them into complete audio, which is then converted into narration sound and a corresponding subtitle file is generated.

[0033] Preferably, step S3 specifically involves performing text semantic matching between the materials in the material library and the processing results of step S22, calculating the text semantic similarity of the four key elements including time, location, people, and events, and sorting them in descending order according to the similarity calculation results, with the material ranked first being the best matching material.

[0034] Preferably, the textual semantic similarity calculation of key elements specifically includes the following:

[0035] S31. Represent the two texts as vectors and convert the texts into vector matrices.

[0036] S32. Process the two texts separately and encode them using a deep neural network to obtain the synthetic representation of the two texts respectively;

[0037] The synthetic representation includes: a token embeddings stage, which processes words and converts each word into a fixed-dimensional vector; a segment embedding stage, which processes sentences and extracts the sentence representation; a position embedding stage, which processes the same word appearing in different positions; and the representations from the above three stages are added element-wise to obtain the synthetic representation.

[0038] S33. Calculate the cosine similarity of the composite representations of the two texts and obtain the similarity calculation results.

[0039] The present invention also aims to provide an intelligent image-to-video conversion system based on video structured data. The system is used to implement any of the methods described above, and the system includes...

[0040] Material Library Module: This module transcodes, segments, and identifies video files in a large-scale video dataset. It also manually reviews and modifies inaccurate identification results and stores accurate identification results and modified results in a structured format in the database to generate the material library.

[0041] The image and text analysis module processes user-submitted text based on text type, performs paragraph splitting, keyword extraction, and named entity recognition on the processing results, and generates narration audio and corresponding subtitle files based on the processing results.

[0042] Material matching module: Based on the results of keyword extraction and named entity recognition processing of paragraphs, it matches the materials in the material library to obtain the best matching material;

[0043] Video compositing module: Uses video compositing algorithms to merge the best matching footage with the narration and corresponding subtitle files to generate a complete video file.

[0044] The beneficial effects of this invention are: 1. It uses artificial intelligence to perform structured analysis of video content, and based on the analyzed data, it converts text and image content into video output through algorithms, solving the tedious and time-consuming problems in the material preparation and production process of video production, and improving video production efficiency. 2. It lowers the threshold for copywriters to create videos, eliminating the need for scriptwriting, material preparation, rough editing, fine editing, audio track synthesis, proofreading, dubbing, etc.; once the text is written, it can be converted into video with one click. 3. It reduces the difficulty for video creators to search for materials, enabling them to quickly and accurately locate ideal materials from a vast amount of video content, shortening the video production cycle. Attached Figure Description

[0045] Figure 1 This is a flowchart of the image-to-video conversion method in an embodiment of the present invention;

[0046] Figure 2 This is a flowchart illustrating the creation of the material library in an embodiment of the present invention;

[0047] Figure 3 This is a flowchart of the graphic analysis process in an embodiment of the present invention;

[0048] Figure 4 This is a flowchart of the material matching process in an embodiment of the present invention;

[0049] Figure 5 This is a structural diagram of the image-to-video conversion system in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] Example 1

[0052] like Figure 1 As shown in the figure, this embodiment provides an intelligent text-to-video conversion method based on video structured data, comprising four parts: establishing a material library, text-image analysis, material matching, and video synthesis. These four parts will be described in detail below.

[0053] I. Establish a resource library

[0054] The system transcodes, segments, and identifies video files in a large-scale video dataset. Inaccurate identification results are manually reviewed and corrected. The accurate identification results and the reviewed and corrected results are stored in a structured format in a database to generate a material library.

[0055] like Figure 2 As shown, this section specifically includes the following content:

[0056] 1. Video file transcoding:

[0057] This function converts video file streams to a preset format, enabling transcoding of all video files in a large-scale video dataset. For example, it can convert all video file streams to MP4 format.

[0058] 2. Shot segmentation:

[0059] Determine whether the transcoded video file needs to be segmented. If so, segment the transcoded video file into individual short video segments and proceed to step 3; otherwise, proceed directly to step 3. Segmentation involves comparing adjacent frames of the video file and calculating the cosine similarity between the two frames. If the similarity is greater than or equal to a similarity threshold, segmentation is performed, dividing the long video into individual short video segments; otherwise, the video file itself consists of short video segments and does not require segmentation.

[0060] 3. Perform video content comprehension on short video clips:

[0061] Identify shots, people, scenes, events, objects, and subtitles appearing in short video clips;

[0062] Specifically, this step involves constructing multiple deep learning models, including shot recognition, face recognition, OCR recognition, and speech recognition, using deep learning convolutional neural networks. These deep learning models are then used to extract shot information, character information, scene information, event information, object information, and subtitle information from short video clips. (Specifically, this involves recognizing text, speech, objects, people, and actions in the video).

[0063] 4. Accuracy assessment and manual review of identification results:

[0064] The accuracy of the recognition results is assessed, and inaccurate results are manually reviewed and modified, with subjective descriptive information including time, location, people, and events added.

[0065] This step specifically includes...

[0066] 4.1 Judging the accuracy of recognition results:

[0067] Keyframes are extracted from short video clips to obtain images of people, scenes, and objects. The similarity scores of these images are then calculated against sample images from face, scene, and object databases, respectively. If the similarity score is greater than or equal to a threshold, the recognition is considered accurate; otherwise, the recognition is considered inaccurate.

[0068] 4.2 Manual review:

[0069] Inaccurate identification information undergoes manual review to confirm the accuracy of the descriptive information extracted by the computer. Manual modifications are made, including the addition of subjective descriptive information that the computer cannot recognize. Manual review primarily focuses on verifying key elements such as time, location, people, and events; any incorrectly identified information is corrected manually.

[0070] 5. Establish a resource library:

[0071] The accurate recognition results and the recognition results after manual review are stored in a structured JSON format in the database to form a material library.

[0072] II. Graphical Analysis

[0073] The system processes user-submitted text based on its text type, performs paragraph splitting, keyword extraction, and named entity recognition on the processing results, and generates narration audio and corresponding subtitle files based on the processing results.

[0074] like Figure 3 As shown, this section specifically includes the following content:

[0075] 1. User-submitted text validation:

[0076] After the user submits content, the system first determines whether the submitted text is plain text or a web link. If it is plain text, the system proceeds directly to step 2. If it is a web link, the system extracts the web page content (text and images), formats the extracted content to remove HTML tags and some meaningless characters, and then proceeds to step 2.

[0077] 2. Text paragraph splitting, keyword extraction, and named entity extraction:

[0078] The text is split into paragraphs, and keywords and named entities are extracted from the split paragraphs.

[0079] This step specifically includes the following:

[0080] 2.1. Paragraph segmentation involves extracting key sentences from the text using the TextRank algorithm and then segmenting the text into paragraphs based on these key sentences. TextRank is a weighted algorithm designed for sentences in text. It uses the voting principle, where each word votes for its neighboring nodes. The weight of each vote depends on the number of votes. Key sentences are identified based on the number of votes, and then the article is segmented into paragraphs based on these key sentences.

[0081] 2.2 Extract keywords and named entities from the segmented paragraphs. Keywords include time, scene, person, and event, and named entities include person name, place name, organization name, and verb. Paragraphs that do not meet the word count threshold (e.g., paragraphs with fewer than 30 words) or paragraphs for which keywords and / or named entities have not been extracted are merged.

[0082] 3. Subtitle file generation:

[0083] Using speech synthesis technology, the text processed in step S21 is converted into narration and a corresponding subtitle file is generated.

[0084] Specifically, this step involves synthesizing sound using speech synthesis technology, performing grammatical analysis on the text processed in step 2.1 to extract the subject, predicate, and object, converting them into speech waveforms, and then using time-domain waveform splicing technology based on the PSOLA method to synthesize them into complete audio, which is then converted into narration and generates corresponding subtitle files.

[0085] III. Material Matching

[0086] The results of keyword extraction and named entity recognition based on paragraphs are matched with materials in the material library to obtain the best matching material.

[0087] like Figure 4 As shown, this part specifically involves performing text semantic matching between the materials in the material library and the processing results of step 2.2. Text semantic similarity is calculated for the four key elements, including time, location, people, and events. The materials are then sorted in descending order (i.e., in reverse order) based on the similarity calculation results, and the material ranked first is taken as the best matching material.

[0088] Calculating the textual semantic similarity of key elements specifically includes the following:

[0089] 3.1. Represent the two texts as vectors and convert them into vector matrices;

[0090] 3.2. Process the two texts separately, encoding them using deep neural networks. (encode) The composite representations (embeddings) of the two texts are obtained respectively;

[0091] The synthetic representation is formed by summing three representations: (1) the token embedding stage, which processes words and converts each word into a vector of fixed dimensions; (2) the segment embedding stage, which processes sentences and extracts the representation of the sentences; and (3) the position embedding stage, which processes the same words appearing in different positions. The representations of the above three stages are added element by element to obtain the synthetic representation.

[0092] 3.3 Calculate the cosine similarity of the composite representations of the two texts and obtain the similarity calculation results.

[0093] IV. Video Compositing

[0094] The best-matching footage is combined with the narration and corresponding subtitle files using a video compositing algorithm (such as ffmpeg) to generate a complete video file.

[0095] Example 2

[0096] like Figure 5As shown, this embodiment provides an intelligent image-to-video conversion system based on video structured data. The system is used to implement the method described above, and the system includes...

[0097] Material Library Module: This module transcodes, segments, and identifies video files in a large-scale video dataset. It also manually reviews and modifies inaccurate identification results and stores accurate identification results and modified results in a structured format in the database to generate the material library.

[0098] The image and text analysis module processes user-submitted text based on text type, performs paragraph splitting, keyword extraction, and named entity recognition on the processing results, and generates narration audio and corresponding subtitle files based on the processing results.

[0099] Material matching module: Based on the results of keyword extraction and named entity recognition processing of paragraphs, it matches the materials in the material library to obtain the best matching material;

[0100] Video compositing module: Uses video compositing algorithms to merge the best matching footage with the narration and corresponding subtitle files to generate a complete video file.

[0101] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0102] This invention provides a method and system for intelligent text-to-video conversion based on structured video data. It uses artificial intelligence to perform structured analysis on video content, and then uses algorithms to convert the text and image content into video output. This solves the tedious and time-consuming problems in the material preparation and production process of video production, improving video production efficiency. It lowers the barrier to entry for copywriters to create videos, eliminating the need for scriptwriting, material preparation, rough editing, fine editing, audio mixing, proofreading, and voice-over. Simply write the text, and it can be converted into video with one click. It also reduces the difficulty for video creators to search for materials, allowing them to quickly and accurately locate ideal materials from a vast amount of video content, shortening the video production cycle.

[0103] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent text-to-video conversion based on structured video data, characterized in that: Includes the following steps, S1. Establish a resource library: The system transcodes, segments, and identifies video files in a large-scale video dataset. Inaccurate identification results are manually reviewed and corrected. The accurate identification results and the reviewed and corrected results are stored in a structured form in the database to generate a material library. Step S1 specifically includes the following: S11. Convert the video file bitstream to a preset format bitstream to achieve transcoding of all video files in a large-scale video dataset; S12. Determine whether the transcoded video file needs to be segmented. If so, segment the transcoded video file into short video segments and proceed to step S13. Otherwise, proceed directly to step S13; If the cosine similarity between two adjacent frames of a video file is greater than or equal to the similarity threshold, then the video file needs to be segmented; otherwise, segmentation is not necessary. S13. Recognize the shots, people, scenes, events, objects and subtitles appearing in short video clips; S14. Determine the accuracy of the recognition results and manually review and modify inaccurate results, adding subjective descriptive information including time, location, people, and events; Step S14 specifically includes the following: S141. Extract keyframes from short video clips to obtain images of people, scenes, and objects. Calculate the similarity between the images of people, scenes, and objects and sample images in the face image library, scene image library, and object image library, respectively. Determine whether the similarity score is greater than or equal to the score threshold. If it is, the recognition is accurate; otherwise, the recognition is inaccurate. S142. Manually review and modify inaccurate identification information, and add subjective descriptive information that cannot be recognized by the computer. S15. Store the accurate recognition results and the recognition results after manual review in the database in JSON format to generate a material library; S2. Graphical Analysis: The system processes user-submitted text based on its text type, performs paragraph splitting, keyword extraction, and named entity recognition on the processed results, and generates narration audio and corresponding subtitle files based on the results. Step S2 specifically includes the following: S21. Determine whether the text submitted by the user is plain text or a web link. If it is plain text, proceed directly to step S22. If it is a web link, extract the web page text and image content, format the extracted content to remove html tags and meaningless characters, and then proceed to step S22. S22. Segment the text into paragraphs, and extract keywords and named entities from the segmented text; Step S22 specifically includes the following: S221. Use the TextRank algorithm to extract key sentences from the text, and segment the text into paragraphs based on the key sentences; S222. Extract keywords and named entities from the segmented paragraphs. Keywords include time, scene, person, and event. Named entities include person name, place name, organization name, and verb. Paragraphs that do not meet the word count threshold or that have not had keywords and / or named entities extracted are merged. S23. Using speech synthesis technology, the text processed in step S21 is converted into narration and a corresponding subtitle file is generated. S3, Material Matching: The results of keyword extraction and named entity recognition based on paragraphs are matched with materials in the material library to obtain the best matching material; S4, Video Compositing: The video compositing algorithm is used to merge the best-matching footage with the narration and corresponding subtitle files to generate a complete video file.

2. The intelligent image-to-video conversion method based on video structured data according to claim 1, characterized in that: Step S13 specifically involves constructing multiple deep learning models, including shot recognition, face recognition, OCR recognition, and speech recognition, using deep learning convolutional neural networks; and using these deep learning models to extract shot information, character information, scene information, event information, object information, and subtitle information from short video clips.

3. The intelligent image-to-video conversion method based on video structured data according to claim 1, characterized in that: Step S23 specifically involves synthesizing sound using speech synthesis technology, performing grammatical analysis on the text processed in step S21 to extract the subject, predicate, and object, converting them into speech waveforms, and then using time-domain waveform splicing technology based on the PSOLA method to synthesize them into complete audio, which is then converted into narration sound and generates corresponding subtitle files.

4. The intelligent image-to-video conversion method based on video structured data according to claim 1, characterized in that: Step S3 specifically involves performing text semantic matching between the materials in the material library and the processing results of step S22. Text semantic similarity is calculated for the four key elements, including time, location, people, and events. The materials are then sorted in descending order based on the similarity calculation results, and the material ranked first is taken as the best matching material.

5. The intelligent image-to-video conversion method based on video structured data according to claim 4, characterized in that: Calculating the textual semantic similarity of key elements specifically includes the following: S31. Represent the two texts as vectors and convert the texts into vector matrices. S32. Process the two texts separately and encode them using a deep neural network to obtain the synthetic representation of the two texts respectively; The synthetic representation includes: a token embeddings stage, which processes words and converts each word into a fixed-dimensional vector; a segment embedding stage, which processes sentences and extracts the sentence representation; a position embedding stage, which processes the same word appearing in different positions; and the representations from the above three stages are added element-wise to obtain the synthetic representation. S33. Calculate the cosine similarity of the composite representations of the two texts and obtain the similarity calculation results.

6. A system for intelligent text-to-video conversion based on structured video data, characterized in that: The system is used to implement the method according to any one of claims 1 to 5, the system comprising, Material Library Module: This module transcodes, segments, and identifies video files in a large-scale video dataset. It also manually reviews and modifies inaccurate identification results and stores accurate identification results and modified results in a structured format in the database to generate the material library. The image and text analysis module processes user-submitted text based on text type, performs paragraph splitting, keyword extraction, and named entity recognition on the processing results, and generates narration audio and corresponding subtitle files based on the processing results. Material matching module: Based on the results of keyword extraction and named entity recognition processing of paragraphs, it matches the materials in the material library to obtain the best matching material; Video compositing module: Uses video compositing algorithms to merge the best matching footage with the narration and corresponding subtitle files to generate a complete video file.

Images