Video rendering method and device, model training method and device, equipment and storage medium

Abstract

The invention provides a video rendering method and device, a model training method and device, equipment and a storage medium, and relates to the technical field of computers, in particular to the field of video rendering and deep learning. According to the specific implementation scheme, an initial model is trained based on a pre-configured data set, and a video analysis model is obtained; and performing analysis processing on the to-be-processed video, and performing rendering processing on the to-be-processed video based on the configuration information to obtain a target video. By training the model, a video analysis model capable of accurately performing video processing is obtained. Compared with parameter generation by adopting a fixed template, the analysis processing of the to-be-processed video has the advantages that the flexibility is improved, the to-be-processed video is rendered based on the configuration information, the flexibility of video processing is improved, the time is saved, and the efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to video rendering and deep learning in computer technology, and more particularly to a video rendering method, model training method, apparatus, device and storage medium. Background Technology

[0002] Video rendering and editing refers to the process of editing, processing, and optimizing video footage using computer software. This process includes cutting the original video footage, adding special effects, adjusting colors, applying transition effects, and other operations to ultimately generate the final video file.

[0003] Today, edited and rendered short videos have become a convenient and quick form of entertainment, and an important tool for people to learn, exchange information, and understand the world. Therefore, with social development and technological progress, people's demand for short videos continues to increase.

[0004] High-quality short videos are usually created by professionals manually adjusting a large number of parameters for rendering and editing. However, manual editing and rendering of short videos by professionals results in low processing efficiency and a lack of flexibility. Summary of the Invention

[0005] This disclosure provides a video rendering method, a model training method, an apparatus, a device, and a storage medium.

[0006] According to a first aspect of this disclosure, a video rendering method is provided, comprising:

[0007] The video to be processed is analyzed and processed to obtain the configuration information corresponding to the video to be processed. The configuration information includes at least one of title features, background image features, and background music features.

[0008] Based on the configuration information, the video to be processed is rendered to obtain the target video.

[0009] According to a second aspect of this disclosure, a model training method is provided, comprising:

[0010] Extract the configuration information from multiple pre-collected videos, and combine each video and its corresponding configuration information into data pairs to obtain a dataset;

[0011] Construct an initial model, which includes a convolutional neural network layer, a long short-term memory network layer, and an attention mechanism layer;

[0012] The initial model is trained based on the dataset to obtain the video analysis model.

[0013] According to a third aspect of this disclosure, a video rendering apparatus is provided, comprising:

[0014] An analysis and processing unit is used to analyze and process the video to be processed to obtain configuration information corresponding to the video to be processed. The configuration information includes at least one of title features, background image features, and background music features.

[0015] The first processing unit is used to render the video to be processed based on the configuration information to obtain the target video.

[0016] According to a fourth aspect of this disclosure, a model training apparatus is provided, comprising:

[0017] The extraction module is used to extract configuration information from multiple pre-collected videos, and to form data pairs with each video and its corresponding configuration information to obtain a dataset.

[0018] A building unit is used to build an initial model, which includes a convolutional neural network layer, a long short-term memory network layer, and an attention mechanism layer.

[0019] The training unit is used to train the initial model based on the dataset to obtain the video analysis model.

[0020] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0021] At least one processor; and

[0022] A memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any aspect of the method.

[0024] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in any aspect.

[0025] According to a seventh aspect of this disclosure, a computer program product is provided, the computer program product comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the method described in any aspect.

[0026] The technology disclosed herein solves the problem of low flexibility in video rendering. By analyzing the video to be processed, configuration information with high adaptability to the video is obtained, and then rendering processing is performed on the video to be processed based on the configuration information, thereby achieving highly flexible video rendering and reducing the error rate of manual rendering.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0028] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0029] Figure 1 This is a scene diagram that can implement the video rendering method and model training method of the embodiments of this disclosure;

[0030] Figure 2 A flowchart illustrating a video rendering method provided in the first embodiment of this disclosure;

[0031] Figure 3 A flowchart illustrating a video rendering method provided in the second embodiment of this disclosure;

[0032] Figure 4 A flowchart illustrating a video rendering method provided in the third embodiment of this disclosure;

[0033] Figure 5 A flowchart illustrating a video rendering method provided in the fourth embodiment of this disclosure;

[0034] Figure 6 A flowchart illustrating a video rendering method provided in the fifth embodiment of this disclosure;

[0035] Figure 7 A schematic flowchart of a model training method provided in the sixth embodiment of this disclosure;

[0036] Figure 8 A schematic flowchart of a model training method provided in the seventh embodiment of this disclosure;

[0037] Figure 9 A structural block diagram of a video rendering apparatus provided in an embodiment of this disclosure;

[0038] Figure 10 A structural block diagram of a model training device provided in an embodiment of this disclosure;

[0039] Figure 11 A schematic block diagram of an electronic device for implementing embodiments of the present disclosure. Detailed Implementation

[0040] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0041] In related technologies, video rendering and editing refers to the process of editing, processing, and optimizing video footage using computer software. This process includes cutting the original video footage, adding special effects, adjusting colors, applying transition effects, and ultimately generating the final video file. Today, rendered and edited short videos have become a convenient and quick form of entertainment, and an important tool for education, learning, and information exchange. Therefore, with social development and technological progress, the demand for short videos continues to increase. However, video rendering and editing, which typically involves individual videos and professionals manually configuring rendering parameters, is ineffective for large-scale video rendering and editing. Furthermore, for rendering large amounts of data, manually configuring parameters one by one by professionals is a time-consuming and error-prone task, easily leading to significant errors.

[0042] To address these existing problems, the technical conception process of this disclosure is as follows: During their research on video rendering and editing, the inventors discovered that manual video rendering and editing is inefficient. Using a fixed template and a video processor to render the video lacks flexibility, resulting in processed videos that do not meet the needs of most users. Therefore, the inventors considered whether it was possible to construct a model, train it based on a large number of videos and their configuration information, and then accurately analyze the videos using the trained model to obtain highly adaptable configuration information. This, combined with user needs, would achieve highly efficient and flexible video rendering processing.

[0043] Figure 1 This is a scene diagram that enables the video rendering method and model training method of the embodiments of this disclosure. For example... Figure 1As shown, the scenario includes at least a user 100, a terminal device 101, an electronic device 102 for video analysis, and a server 103. The terminal device 101, the electronic device 102, and the server 103 are communicatively connected. The terminal device 101 is held by the user 100, who can operate the terminal device 101. The terminal device 101 can be a smartphone, a laptop, a smart bracelet, or other such device. Similarly, the electronic device 102 can also be a smartphone, a laptop, a smart bracelet, or other such device, and can also be a server.

[0044] Optionally, the communication connection between the above devices can be wireless or wired. Wired connections can be Ethernet, using network cables (such as Cat5e or Cat6) to connect the devices. Serial communication can also be used, for example, standards including RS-232 and RS-485. Fiber optic communication can also be used, for example, in metropolitan area networks (MANs) and wide area networks (WANs). Wireless communication can be wireless LAN, Bluetooth, Zigbee, cellular networks, or low-power wide area networks. For remote areas, satellite communication can also be used.

[0045] Server 103 is used to collect video data for model training. The trained video analysis model is sent to electronic device 102 for configuration. User 100 sends the video to be processed to electronic device 102 through terminal device 101. Electronic device 102 processes the video to be processed based on the video analysis model and renders the video based on the processing results. The obtained target video can be fed back to terminal device 101 for user 100 to view. User 100 can also send the target video to the network through terminal 101 for more people to view.

[0046] In one possible implementation, during the processing of the video to be processed, the electronic device 102 can also send the configuration information obtained during the processing to the terminal device 101, and generate a modification interface on the terminal device 101 for the user 100 to modify the configuration information.

[0047] Optionally, server 103 can be a standalone device or part of electronic device 102.

[0048] Understandable, Figure 1 The illustrated scenario diagram is merely an example. In practical applications, this scenario diagram may also include other devices, such as storage devices, and can be adjusted according to actual needs. This disclosure does not limit these devices. Furthermore, the embodiments of this disclosure do not limit the actual form of the various devices included in the application scenario, nor do they limit the interaction methods between devices. In the specific application of the solution, these can be set according to actual needs.

[0049] This disclosure provides a video rendering method, a model training method, an apparatus, and a storage medium, which are applied to video rendering and deep learning in the field of computer technology to achieve high efficiency and high flexibility in video processing.

[0050] The following detailed description uses specific embodiments to illustrate the technical solutions of this disclosure and how these solutions solve the aforementioned technical problems. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0051] The following section introduces the specific implementation scheme of the video rendering method provided in this disclosure.

[0052] Figure 2 This is a flowchart illustrating a video rendering method provided in the first embodiment of this disclosure, with reference to... Figure 2 The video rendering method shown can be configured as a video rendering device, which can be located in an electronic device, such as a mobile phone, cloud server, server cluster, computer, laptop, etc. This disclosure does not impose excessive limitations on the specific type of electronic device. The video rendering method may include the following steps:

[0053] S201: Analyze and process the video to be processed to obtain the configuration information corresponding to the video to be processed.

[0054] In this step, in order to efficiently render the video to be processed, the video to be processed can be accurately analyzed and processed to obtain the configuration information corresponding to the video to be processed. The configuration information includes at least one of the following: title features, background image features, and background music features.

[0055] Optionally, to accurately analyze and process the video to be processed, a video analysis model can be used. For example, a deep learning model trained on a large amount of video data and the corresponding configuration information can be used, such as a convolutional neural network, recurrent neural network, long short-term memory network, gated recurrent unit, generative adversarial network, contrastive learning model, and transfer learning model. Alternatively, a fusion model of machine learning model and deep learning model can be trained on a large amount of video data and the corresponding configuration information. For example, a machine learning model can be used to analyze the low-level features of the video, while a deep learning model can be used to analyze the complex features of the video, such as content recognition and scene understanding. The machine learning model can be a support vector machine, decision tree, random forest, or other similar models.

[0056] In one possible approach, the analysis and processing of the video to be processed can involve analyzing the scene in the video, as well as analyzing objects, colors, language, and human body movements and expressions in the video, thereby obtaining configuration information that is highly compatible with the video to be processed.

[0057] S202: Based on the configuration information, render the video to be processed to obtain the target video.

[0058] In this step, after obtaining the configuration information through the above steps, the video to be processed can be rendered based on at least one of the title features, background image features, and background music features in the configuration information, thereby obtaining the target video.

[0059] Optionally, if the configuration information includes title features, a target title can be generated based on the title features, and the video to be processed can be rendered based on the target title; if the configuration information includes background image features, a target background image can be generated based on the background image features, and the video to be processed can be rendered based on the target background image; if the configuration information includes background music features, a target background music can be generated based on the background music features, and the video to be processed can be rendered based on the target background music; if the configuration information includes any and more of the title features, background image features, and background music features, corresponding target information can be generated based on each feature, and the video to be processed can be rendered.

[0060] For example, taking the configuration information as including both title features and background music features, after generating the target title based on the title features and the target background music based on the background music features, the video to be processed can be rendered first based on the target title to obtain the rendered video, and then the rendered video can be rendered based on the target background music.

[0061] Alternatively, the video to be processed can be rendered first based on the target background music to obtain the rendered video, and then the rendered video can be rendered based on the target title.

[0062] Optionally, the video to be processed can be rendered simultaneously based on the target title and the target background music, or the video to be processed can be rendered separately based on the target title and the target background music, and then the separately rendered videos can be superimposed.

[0063] The video rendering method provided in this embodiment accurately analyzes and processes the video to be processed to obtain the configuration information corresponding to the video. Based on the configuration information, the video to be processed is rendered to obtain the target video. Compared with manually configuring the video to be processed, this method improves the efficiency of video rendering and increases flexibility compared with rendering the video to be processed using fixed templates or rules. It also avoids errors caused by the influence of human subjectivity.

[0064] Figure 3 This is a flowchart illustrating a video rendering method provided in the second embodiment of this disclosure, with reference to... Figure 3 The video rendering method shown differs from the previous embodiment in that, in this embodiment, the configuration information also includes a target duration. Based on the configuration information, the video to be processed is rendered to obtain the target video. This can be further refined as follows: for the video to be processed, rendering is performed based on at least one of the title features, background image features, and background music features, and the video is edited according to the target duration to obtain the target video.

[0065] like Figure 3 As shown, the method includes the following steps:

[0066] S301: Analyze and process the video to be processed to obtain the configuration information corresponding to the video to be processed.

[0067] This step is implemented in the same way as step S201 in the previous embodiment, and will not be described again here.

[0068] S302: For the video to be processed, render it based on at least one of the title features, background image features, and background music features, and edit it according to the target duration to obtain the target video.

[0069] In this step, the configuration information obtained in the above steps may also include the target duration. Then, rendering can be performed based on at least one of the title features, background image features, and background music features, and the video can be edited according to the target duration to obtain the target video.

[0070] For example, based on the title features, background image features, background music features, and target duration in the configuration information, the specific rendering process can be performed in the following order: first, rendering is performed based on the features, and then video editing is performed based on the target duration; or video editing is performed first, and then rendering is performed based on the features on the edited video; or rendering and editing are performed simultaneously.

[0071] Optionally, if the configuration information includes title features, a target title is generated based on the pre-defined mapping relationship between title features and titles, and the title features themselves; if the configuration information includes background image features, a target background image is obtained by analyzing the background image features using a computer vision algorithm based on a pre-defined background image database; if the configuration information includes background music features, a target background music is obtained by analyzing the background music features using a sentiment analysis algorithm based on a pre-defined background music database; and the video to be processed is rendered based on at least one of the target title, target background image, and target background music.

[0072] For example, the title features may include the title's position, number of characters, color, type, etc., and the mapping relationship between the title features and the title can be pre-defined, and the target title can be generated according to pre-defined rules.

[0073] Rules are based on video type, for example: sports videos: title prefix is ​​"Exciting Sports:"; news videos: title prefix is ​​"Latest News:"; entertainment videos: title prefix is ​​"Entertainment Headlines:"; technology videos: title prefix is ​​"Technological Frontiers:"; educational videos: title prefix is ​​"Educational Knowledge:".

[0074] The rules are based on time. For example, if the basic title of a video is "Opening Ceremony 2024" and it is published on June 13, 2024, then the generated title will be "Opening Ceremony 2024 June 13".

[0075] Rules are formulated based on content keywords. For example, if the keyword contains "opening", the title could be "An unmissable opening ceremony: Opening in 2024"; if the keyword contains "technology", the title could be "Unveiling revolutionary technology: Latest AI technology release".

[0076] Rules are set based on the target audience. For example, videos aimed at young people are prefixed with "Must-See Trend:" while videos aimed at older people are prefixed with "Classic Review:".

[0077] It should be noted that the target title generated based on the pre-defined rules can be one or more of the rules mentioned above, or it can be rules that are not limited to the examples above, or it can include other pre-defined rules.

[0078] In one possible implementation, a machine learning model can be trained based on multiple videos and their titles. The trained model then generates the target title. A title generation model is trained using a natural language processing model, and the title features are input into this model to generate the target title. For example, using a trained Seq2Seq model to generate video titles, with the title feature "Highlights of the 2024 Opening Ceremony," the model, after processing the title features and understanding information such as "2024," "Opening Ceremony," and "highlights," generates the most suitable title based on the pre-trained knowledge. For example, it could be "Unmissable Highlights of the 2024 Opening Ceremony." The output title can be a single target title or multiple titles, which are then provided to the user, allowing the user to choose one as the target title.

[0079] In another possible approach, the specific position and layout of the title can be determined by combining deep learning and image processing techniques to automatically detect a suitable location within the video frame and render the generated title into the video. Taking a video about natural scenery as an example, the generated title "Beautiful Natural Scenery" needs to be automatically inserted into the video. Specific implementation methods include:

[0080] The process involves extracting keyframes from the video to be processed. For example, selecting a frame showcasing a vast blue sky and green land, and then using a pre-trained convolutional neural network model to analyze the keyframe image, identifying the main elements in the image, such as the sky, trees, and grass. The model might output semantic segmentation results: sky region, grass region, tree region, etc. Based on the semantic segmentation results, the sky region is determined to be a suitable location for the title, as it typically has a simple background and does not obscure the main visual elements. The flatter portion of the sky region is chosen as the title placement area, for example, the middle of the upper third of the image. The font size, color, and style of the title are then determined. Assuming a bold white font of a moderate size is chosen to ensure clear visibility against the blue sky background, the layout is designed to be center-aligned with the title, with appropriate margins on both sides. Through these steps, the generated title will be clearly displayed in the sky region of the video frame, without obscuring the main scene, and will be harmonious and aesthetically pleasing with the video content.

[0081] The video rendering method provided in this embodiment analyzes and processes the video to be processed, and then renders it based on at least one of the obtained title features, background image features, and background music features. It then edits the video according to the target duration to obtain the target video. This method achieves highly efficient rendering of the video to be processed, and compared to traditional methods that require manual configuration of rendering parameters, it increases flexibility, saves time, and avoids frequent errors.

[0082] Figure 4 This is a flowchart illustrating a video rendering method provided in the third embodiment of this disclosure, with reference to... Figure 4 The video rendering method shown differs from the previous embodiments in that, in this embodiment, the video to be processed is analyzed and processed to obtain the configuration information corresponding to the video to be processed. This can be further refined as follows: a video analysis model is used to analyze and process the video to be processed to obtain the configuration information.

[0083] like Figure 4 As shown, the method includes the following steps:

[0084] S401: The video analysis model is used to analyze and process the video to be processed, and configuration information is obtained.

[0085] In order to accurately analyze the video to be processed and obtain accurate configuration information, a video analysis model can be used to analyze the video. The video analysis model is trained based on multiple videos and the configuration information of each video, and is used to analyze the configuration information of the video.

[0086] For example, a video analysis model can be trained on a deep learning model based on multiple videos and configuration information for each video. The video analysis model can be used to analyze and process videos, such as a video recording a person's activities on a beach. The video contains many scenes, such as the beach, waves, and people playing, for a detailed explanation:

[0087] First, extract representative frames (keyframes) from the video. Assuming one frame is extracted every second, this yields a series of images that represent the main scenes and changes in the video. For example, the following frames might be obtained:

[0088] First frame: A person stands on the beach with waves in the distance; Second frame: The same person is running towards the sea; Third frame: The person jumps into the sea, splashing water.

[0089] Next, a video analytics model is used to analyze these keyframes. The model can identify different objects in the image and provide their labels and locations. For example:

[0090] First frame: Two objects, "person" and "waves", were detected; Second frame: "person", "beach", and "waves" were detected; Third frame: "person" and "splashes" were detected.

[0091] Videos typically also contain audio. The audio portion of a video file can be extracted using certain modules within this model, and then audio processing tools (such as Librasa) can be used to analyze audio features, such as rhythm and frequency. For example, from the audio, the rhythm of the background music can be detected as 120 beats per minute (bpm). Sounds of ocean waves and laughter can also be detected.

[0092] The above steps of analysis extract various features from the video, including visual features (such as objects and scenes in the images) and audio features (such as rhythm and sound type). For example, the video's theme involves outdoor activities, beaches, and entertainment; the video has a strong rhythm, suitable for pairing with fast-paced background music; the frequent appearance of "people" and "waves" in the scene indicates that the video mainly shows people's activities on the beach.

[0093] Optionally, after extracting multiple features from the video, the features can be further divided into title features, background image features, and background music features. For example, the feature of the subject type can be divided into title features. If "beach" and "people" appear multiple times in the video, it can be inferred that the video theme is related to beach activities, and title features such as "beach vacation" or "happy time at the beach" can be generated.

[0094] The overall style and atmosphere of the video are categorized into background image features, and keyframes with rich colors and aesthetically pleasing compositions are selected. Typically, scenes with bright colors and strong visual impact are chosen, selecting images that reflect the main content and atmosphere of the video. For example, if the video primarily features a beach, an image containing blue sky, white clouds, and waves can be selected as the target background image.

[0095] The video content and emotional characteristics are categorized into background music features. By analyzing the original audio of the video, the rhythm and pitch characteristics of the background music are extracted. If the video has its own background music, music of a similar style can be used directly or found. Appropriate music should be selected based on the type of activity and mood in the video. For example, fast-paced music is suitable for active scenes, while soft music is suitable for calm scenes. For instance, if the video contains many joyful activities and laughter, a lively and energetic piece of music can be chosen as the background music.

[0096] S402: Based on the configuration information, render the video to be processed to obtain the target video.

[0097] The specific implementation method in this step is the same as that in the aforementioned embodiments, and will not be repeated here.

[0098] The video rendering method provided in this embodiment uses a video analysis model to analyze and process the video to be processed, obtain configuration information, and then renders the video to be processed based on the configuration information to obtain the target video. By using a pre-trained video analysis model to process the video to be processed, the efficiency of video rendering is improved, and compared with rendering the video based on a fixed template, the flexibility is increased.

[0099] Figure 5 This is a flowchart illustrating a video rendering method provided in the fourth embodiment of this disclosure, with reference to... Figure 5 The video rendering method shown differs from the previous embodiments in that, in this embodiment, the video to be processed can also be rendered based on the user's needs.

[0100] like Figure 5 As shown, the method includes the following steps:

[0101] S501: Send configuration information to the user terminal device.

[0102] To improve the flexibility of video rendering, the generated configuration information can be sent to the user's terminal device. Users can adjust the title, background image, and background music according to their needs and preferences, thereby personalizing the video content they like, increasing human-computer interaction, and improving flexibility.

[0103] For example, the generated configuration information can be sent to the user's terminal device application or webpage, allowing the user to modify and customize it on their own device. Alternatively, the configuration information can be sent to the user's terminal device in the form of an editable template, allowing the user to directly modify the template, such as selecting different background image features, changing title features, or selecting other background music features. An interactive interface can also be created, allowing users to directly modify the information on the interface, such as dragging and dropping images or selecting music.

[0104] S502: Receive updated configuration information from user terminal equipment.

[0105] S503: Based on the updated configuration information, render the video to be processed to obtain a new target video.

[0106] After the configuration information is sent to the user's terminal device, in order to flexibly consider user needs, the user can provide feedback on the updated configuration information. Upon receiving the updated configuration information from the user, the system can then render the video to be processed based on the updated configuration information to obtain a new target video.

[0107] Optionally, the updated configuration information can include newly added clickable text information, which can then be used to modify the target video. This clickable text information refers to the text in the lower left corner of the video that viewers can click on; this text can describe the video content or include comments from the video publisher.

[0108] In one possible implementation, the updated configuration information may also include user modifications to the configuration information, which can then be used to render the video to be processed.

[0109] In another possible implementation, the updated configuration information may also include user instructions to delete the configuration information content, or it may include one or more of the target title, target background image, and target background music directly specified by the user.

[0110] The video rendering method provided in this embodiment sends configuration information to the user terminal device and receives updated configuration information from the user terminal device. Based on the updated configuration information, it renders the video to be processed to obtain a new target video. Through interaction with the user, the user can participate in the video content generation process, customize their preferred video content, and increase user participation and interactive experience. Such an interactive experience can improve user satisfaction and better meet the user's personalized needs.

[0111] Figure 6 This is a flowchart illustrating a video rendering method provided in the fifth embodiment of this disclosure, with reference to... Figure 6 The video rendering method shown differs from the previous embodiments in that, in this embodiment, the analysis and processing of the video to be processed to obtain a refined scheme of configuration information corresponding to the video to be processed may further include: extracting configuration information of multiple pre-collected videos, forming data pairs with each video and the configuration information corresponding to each video to obtain a dataset; constructing an initial model; and training the initial model based on the dataset to obtain a video analysis model.

[0112] like Figure 6 As shown, the method includes the following steps:

[0113] S601: Extract the configuration information of multiple pre-collected videos, and combine each video and its corresponding configuration information into a data pair to obtain a dataset.

[0114] During model training, the training dataset is crucial as it can affect the model's accuracy and generalization ability. Therefore, it is necessary to collect a large number of videos in advance, extract the configuration information from each video, pair each video with its corresponding configuration information, and combine all the data pairs into a dataset.

[0115] For example, the large amount of pre-collected video needs to be diverse, including videos with various subjects, scenes and styles.

[0116] S602: Construct the initial model.

[0117] To obtain a highly accurate model, having a diverse dataset is not enough; a model with good generalization ability and strong learning ability is also needed for training.

[0118] For example, during the construction of the initial model, since the video contains multiple elements, such as people, colors, and sounds, a multimodal fusion model can be selected as the initial model. The initial model can include convolutional neural network layers, long short-term memory network layers, and attention mechanism layers. In order to better process the video, other layers, such as fully connected layers, can also be set in the initial model according to actual needs.

[0119] S603: Train the initial model based on the dataset to obtain the video analysis model.

[0120] After obtaining the dataset, it can be divided into training and test sets. The specific ratio can be 50% each, or 70% for training and 30% for testing, etc. The exact ratio is not limited. Based on the training set, the gradient descent method is used to iteratively optimize the parameters of the initial model to obtain the initial video analysis model. Then, based on the test set, the initial video analysis model is evaluated and adjusted to obtain the final video analysis model.

[0121] Optionally, during the iterative optimization of the initial model based on the training set, evolutionary algorithms, such as genetic algorithms and differential evolution algorithms, can be used; Bayesian optimization can also be used, which approximates the objective function by constructing a surrogate model (such as a Gaussian process or tree-structured Parzen estimation), thereby iteratively optimizing the initial model based on uncertainty sampling; or trust region methods can be used, which update the parameters of the initial model by approximating the objective function in a local region and solving subproblems, thereby ensuring the stability and accuracy of each update step.

[0122] S604: The video analysis model is used to analyze and process the video to be processed, and configuration information is obtained.

[0123] This step is the same as the specific method that can be implemented in the aforementioned embodiments, and will not be described again here.

[0124] The video rendering method provided in this embodiment constructs an initial model and dataset, trains the initial model based on the dataset to obtain a video analysis model, and then uses the video analysis model to analyze and process the video to obtain configuration information. By using a pre-trained model to analyze the video, the accuracy of video analysis is ensured, the efficiency of video processing is improved, and processing time is saved.

[0125] The specific implementation scheme of the model training method provided in this disclosure is described below.

[0126] Figure 7 This is a flowchart illustrating a model training method provided in the sixth embodiment of this disclosure, with reference to... Figure 7 The model training method shown includes the following steps:

[0127] S701: Extract the configuration information of multiple pre-collected videos, and combine each video and its corresponding configuration information into a data pair to obtain a dataset.

[0128] S702: Construct the initial model.

[0129] S703: Train the initial model based on the dataset to obtain the video analysis model.

[0130] The specific implementation of this embodiment is the same as that of steps S601-S603 in the aforementioned embodiment, and will not be repeated here.

[0131] The model training method provided in this embodiment obtains a video analysis model by constructing an initial model and a dataset, and then training the initial model based on the dataset. Compared to rendering videos using fixed templates or rules, this method improves the accuracy and flexibility of video processing.

[0132] Figure 8 This is a flowchart illustrating a model training method provided in the seventh embodiment of this disclosure, with reference to... Figure 8 The model training method shown differs from the previous embodiments in that, in this embodiment, training the initial model based on the dataset to obtain the video analysis model can be refined as follows: dividing the dataset into a training set and a test set; inputting the training set into the initial model, iteratively optimizing the parameters of the initial model based on the gradient descent method to obtain the initial video analysis model; and adjusting the initial video analysis model based on the test set to obtain the final video analysis model.

[0133] like Figure 8 As shown, the method includes the following steps:

[0134] S801: Extract the configuration information of multiple pre-collected videos, and combine each video and its corresponding configuration information into a data pair to obtain a dataset.

[0135] The specific implementation of this step is the same as the implementation of dataset construction in the aforementioned embodiments, and will not be repeated here.

[0136] S802: Construct the initial model.

[0137] The specific implementation of this step is the same as the implementation method constructed in the aforementioned embodiments, and will not be repeated here.

[0138] S803: Divide the dataset into training and test sets.

[0139] The specific implementation of this step is the same as the specific implementation of dataset partitioning in the aforementioned embodiments, and will not be repeated here.

[0140] S804: Input the training set into the initial model, and iteratively optimize the parameters of the initial model based on the gradient descent method to obtain the initial video analysis model.

[0141] In this step, after inputting the training set into the initial model, the training set data needs to be preprocessed. Then, the loss function of the initial model is defined. For example, a loss function is determined to measure the difference between the model output and the true labels (configuration information corresponding to the videos in the training set), such as Mean Squared Error (MSE) or Cross Entropy Loss. During the initial model computation, the loss between the model output and the true labels is calculated. Using the backpropagation algorithm, the gradient of each parameter of the initial model with respect to the loss is calculated, and the parameters of the initial model are updated using gradient descent or its variants (such as stochastic gradient descent or batch gradient descent) to gradually reduce the loss function until a preset number of iterations or other preset conditions are met, thus obtaining the initial video analysis model.

[0142] S805: Adjust the initial video analysis model based on the test set to obtain the video analysis model.

[0143] After obtaining the initial video analysis model, it is validated on the test set, and the hyperparameters or structure of the model are adjusted according to the validation results to improve the generalization ability of the model.

[0144] For example, after each training epoch in the aforementioned steps, the model's performance is evaluated on the test set, and the loss and accuracy on the test set are recorded. If the performance on the test set does not improve significantly, it indicates that the model may have problems, such as underfitting or overfitting.

[0145] Optionally, if the model performs poorly on both the training and test sets, it indicates that the model complexity is insufficient and it cannot capture the features of the data. In this case, you can increase the number of layers or neurons in the initial model; you can also fuse other network layers; or you can update the learning parameters and increase the learning rate to accelerate convergence.

[0146] Optionally, if a model performs well on the training set but poorly on the test set, it indicates that the model is overly complex and has absorbed noise from the training set. In this case, regularization terms such as L2 regularization or Dropout can be added. Alternatively, the number of layers in the initial model can be reduced, or data transformations can be used to increase the diversity of the training data. This helps the model generalize better, reduces the learning rate, and makes model updates smoother and more stable.

[0147] After repeated iterative training and optimization, the video analysis model was finally obtained.

[0148] The model training method provided in this embodiment divides a pre-acquired dataset into a training set and a test set. An initial model is trained based on the training set, and then the trained initial video analysis model is evaluated and validated based on the test set, thereby obtaining a video analysis model. This method yields a video analysis model capable of accurately analyzing videos, improving the accuracy and flexibility of video processing compared to rendering videos using fixed templates or rules.

[0149] Figure 9 This is a structural block diagram of a video rendering apparatus provided in an embodiment of the present disclosure, with reference to... Figure 9 The video rendering apparatus 900 shown includes the following units:

[0150] The analysis and processing unit 901 is used to analyze and process the video to be processed to obtain the configuration information corresponding to the video to be processed. The configuration information includes at least one of title features, background image features, and background music features.

[0151] The first processing unit 902 is used to render the video to be processed based on the configuration information to obtain the target video.

[0152] In one possible design, the configuration information also includes the target duration;

[0153] Accordingly, the first processing unit 902 includes:

[0154] The rendering and editing module is used to render the video to be processed based on at least one of the following features: title features, background image features, and background music features, and to edit it according to the target duration to obtain the target video.

[0155] Optional, the rendering clip module includes:

[0156] The first generation submodule is used to generate a target title based on the pre-defined mapping relationship between title features and titles, and the title features themselves, if the configuration information includes title features.

[0157] The second generation submodule is used to analyze the background image features according to a pre-set background image database and use computer vision algorithms to obtain the target background image if the configuration information includes background image features.

[0158] The third generation submodule is used to analyze the background music features based on a pre-set background music database and a sentiment analysis algorithm to obtain the target background music if the configuration information includes background music features.

[0159] The rendering submodule is used to render the video to be processed based on at least one of the target title, target background image, and target background music.

[0160] In one possible implementation, the analysis and processing unit 901 includes:

[0161] The processing module is used to analyze and process the video to be processed using a video analysis model to obtain configuration information. The video analysis model is a model trained based on multiple videos and the configuration information of each video for analyzing the configuration information of the video.

[0162] Optionally, the analysis and processing unit 901 further includes:

[0163] The extraction module is used to extract the configuration information of multiple pre-collected videos, and to form a data pair between each video and its corresponding configuration information to obtain a dataset;

[0164] The building module is used to build the initial model, which includes a convolutional neural network layer, a long short-term memory network layer, and an attention mechanism layer.

[0165] The training module is used to train the initial model based on the dataset to obtain the video analysis model.

[0166] In another possible design, the video rendering device 900 also includes:

[0167] The sending unit 903 is used to send configuration information to the user terminal device;

[0168] The receiving unit 904 is used to receive updated configuration information fed back by the user terminal device;

[0169] The second processing unit 905 is used to render the video to be processed based on the updated configuration information to obtain a new target video.

[0170] Optionally, the updated configuration information includes newly added click text information;

[0171] Accordingly, the second processing unit 905 includes:

[0172] The editing module is used to modify the target video based on the clicked text information.

[0173] Figure 10 This is a structural block diagram of a model training device provided in an embodiment of the present disclosure, with reference to... Figure 10 The model training device 1000 shown includes the following units:

[0174] Extraction unit 1001 is used to extract configuration information from multiple pre-collected videos, and to form a data pair between each video and its corresponding configuration information to obtain a dataset;

[0175] Building unit 1002 is used to build the initial model, which includes a convolutional neural network layer, a long short-term memory network layer, and an attention mechanism layer.

[0176] Training unit 1003 is used to train the initial model based on the dataset to obtain the video analysis model.

[0177] Optionally, training unit 1003 includes:

[0178] The partitioning module is used to divide the dataset into training and testing sets;

[0179] The iteration module is used to input the training set into the initial model and iteratively optimize the parameters of the initial model based on the gradient descent method to obtain the initial video analysis model.

[0180] The adjustment module is used to adjust the initial video analysis model based on the test set to obtain the final video analysis model.

[0181] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0182] According to embodiments of this disclosure, this disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the scheme provided in any of the above embodiments.

[0183] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0184] like Figure 11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0185] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of displays, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0186] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the video rendering method and the model training method. For example, in some embodiments, the video rendering method and the model training method can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the video rendering method and the model training method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured in any other suitable manner (e.g., by means of firmware) to perform a video rendering method or a model training method.

[0187] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0188] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0189] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0190] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0191] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0192] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0193] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0194] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video rendering method, comprising: The video to be processed is analyzed and processed to obtain the configuration information corresponding to the video to be processed. The configuration information includes at least one of title features, background image features, and background music features. Based on the configuration information, the video to be processed is rendered to obtain the target video.

2. The method according to claim 1, wherein, The configuration information also includes the target duration; Accordingly, the rendering process of the video to be processed based on the configuration information to obtain the target video includes: For the video to be processed, rendering is performed based on at least one of the title feature, the background image feature, and the background music feature, and the video is edited according to the target duration to obtain the target video.

3. The method according to claim 1, wherein, The process involves analyzing and processing the video to be processed to obtain configuration information corresponding to the video, including: The video to be processed is analyzed using a video analysis model to obtain the configuration information. The video analysis model is a model trained based on multiple videos and the configuration information of each video for analyzing the configuration information of the videos.

4. The method according to any one of claims 1 to 3, further comprising: The configuration information is sent to the user terminal device; Receive updated configuration information from the user terminal device; Based on the updated configuration information, the video to be processed is rendered to obtain a new target video.

5. The method according to claim 4, wherein, The updated configuration information includes newly added click text information; Accordingly, the step of rendering the video to be processed based on the updated configuration information to obtain a new target video includes: The target video is modified based on the clicked text information.

6. The method according to claim 3, further comprising: Extract the configuration information from multiple pre-collected videos, and combine each video and its corresponding configuration information into data pairs to obtain a dataset; Construct an initial model, which includes a convolutional neural network layer, a long short-term memory network layer, and an attention mechanism layer; The initial model is trained based on the dataset to obtain the video analysis model.

7. The method according to claim 1 or 2, wherein, For the video to be processed, rendering is performed based on at least one of the following: title features, background image features, and background music features, including: If the configuration information includes title features, then a target title is generated based on the pre-defined mapping relationship between title features and titles and the title features. If the configuration information includes background image features, then the background image features are analyzed using a computer vision algorithm based on a pre-set background image database to obtain the target background image; If the configuration information includes background music features, then according to the pre-set background music database, the background music features are analyzed using a sentiment analysis algorithm to obtain the target background music; The video to be processed is rendered based on at least one of the target title, the target background image, and the target background music.

8. A model training method, comprising: Extract the configuration information from multiple pre-collected videos, and combine each video and its corresponding configuration information into data pairs to obtain a dataset; Construct an initial model, which includes a convolutional neural network layer, a long short-term memory network layer, and an attention mechanism layer; The initial model is trained based on the dataset to obtain the video analysis model.

9. The method according to claim 8, wherein, The process of training the initial model based on the dataset to obtain the video analysis model includes: The dataset is divided into a training set and a test set; The training set is input into the initial model, and the parameters of the initial model are iteratively optimized based on the gradient descent method to obtain the initial video analysis model; The initial video analysis model is adjusted based on the test set to obtain the video analysis model.

10. A video rendering apparatus, comprising: An analysis and processing unit is used to analyze and process the video to be processed to obtain configuration information corresponding to the video to be processed. The configuration information includes at least one of title features, background image features, and background music features. The first processing unit is used to render the video to be processed based on the configuration information to obtain the target video.

11. The apparatus according to claim 10, wherein, The configuration information also includes the target duration; Accordingly, the first processing unit includes: The rendering and editing module is used to render the video to be processed based on at least one of the title features, the background image features, and the background music features, and to edit it according to the target duration to obtain the target video.

12. The apparatus according to claim 10, wherein, The analysis and processing unit includes: The processing module is used to analyze and process the video to be processed using a video analysis model to obtain the configuration information. The video analysis model is a model trained based on multiple videos and the configuration information of each video for analyzing the configuration information of the videos.

13. The apparatus according to any one of claims 10 to 12, further comprising: A sending unit is used to send the configuration information to a user terminal device; The receiving unit is used to receive the updated configuration information fed back by the user terminal device; The second processing unit is used to render the video to be processed based on the updated configuration information to obtain a new target video.

14. The apparatus according to claim 13, wherein, The updated configuration information includes newly added click text information; Accordingly, the second processing unit includes: The modification module is used to modify the target video based on the clicked text information.

15. The apparatus of claim 12, further comprising: The extraction module is used to extract the configuration information of multiple pre-collected videos, and to form a data pair between each video and its corresponding configuration information to obtain a dataset; A building module is used to build an initial model, which includes a convolutional neural network layer, a long short-term memory network layer, and an attention mechanism layer. The training module is used to train the initial model based on the dataset to obtain the video analysis model.

16. The apparatus according to claim 12, wherein, The rendering and editing module includes: The first generation submodule is used to generate a target title based on a pre-defined mapping relationship between title features and titles, and the title features, if the configuration information includes title features. The second generation submodule is used to analyze the background image features according to a pre-set background image database and a computer vision algorithm to obtain the target background image if the configuration information includes background image features. The third generation submodule is used to analyze the background music features according to a pre-set background music database and a sentiment analysis algorithm to obtain the target background music if the configuration information includes background music features. The rendering submodule is used to render the video to be processed based on at least one of the target title, the target background image, and the target background music.

17. A model training device, comprising: The extraction module is used to extract configuration information from multiple pre-collected videos, and to form data pairs with each video and its corresponding configuration information to obtain a dataset. A building unit is used to build an initial model, which includes a convolutional neural network layer, a long short-term memory network layer, and an attention mechanism layer. The training unit is used to train the initial model based on the dataset to obtain the video analysis model.

18. The apparatus according to claim 17, wherein, The training unit includes: A partitioning module is used to divide the dataset into a training set and a test set; An iterative module is used to input the training set into the initial model and iteratively optimize the parameters of the initial model based on the gradient descent method to obtain an initial video analysis model. An adjustment module is used to adjust the initial video analysis model based on the test set to obtain the video analysis model.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-9.

Images