Video processing method, electronic device, storage medium and program product

Abstract

The application relates to the computer technical field and discloses a video processing method, an electronic device, a storage medium and a program product. The method comprises the following steps: acquiring a plurality of reference videos related to a preset resource category; performing shot division processing on the reference videos to obtain each shot segment of the reference videos; performing analysis processing on each shot segment according to a preset label library corresponding to the preset resource category to determine the structured information of each shot segment; forming a corresponding label sequence according to the shot labels of each shot segment in the reference videos; performing clustering processing according to the label sequences of each reference video to obtain a plurality of video clusters; the video clusters comprise a plurality of reference videos; for any video cluster, a video framework template corresponding to the preset resource category is generated according to the structured information corresponding to each reference video in the video cluster. The clustering and generation by using the shot labels can guarantee the reproducibility of the video template framework and improve the video generation quality.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to video processing methods, electronic devices, storage media, and program products. Background Technology

[0002] With the rapid development of the video content ecosystem, short videos are no longer limited to entertainment and are increasingly linked to product marketing, becoming an important channel for brand exposure, user engagement, and sales conversion. Product promotion models based on short videos offer advantages such as precise targeting, efficient dissemination, and strong interactivity, and are increasingly becoming one of the core tools for enterprise digital marketing.

[0003] Currently, there are highly popular viral short videos, which have been market-tested and have proven effective in product marketing. Most viral short videos are not accidental but rather follow reusable creative patterns. Related solutions primarily analyze the video as a whole or the narration script; however, analyzing the narration script cannot analyze the video's visual dimensions. While analyzing the overall creative patterns of the video can extract video features to some extent, this holistic analysis method struggles to accurately extract video features and cluster videos with similar characteristics. Consequently, it is difficult to obtain a framework template that represents the common features of the videos, and the automatic generation capability of structured video framework templates is poor. Summary of the Invention

[0004] To address the aforementioned problems, this application provides a video processing method, an electronic device, a storage medium, and a program product.

[0005] In a first aspect, this application provides a video processing method, the method comprising: Retrieve multiple reference videos related to a preset resource category; For any of the aforementioned reference videos, the reference video is divided into shots to obtain individual shot segments of the reference video; Based on the preset tag library corresponding to the preset resource category, each of the shot segments is parsed to determine the structured information of each shot segment; the structured information includes the description information and shot tags of the shot segment, and the shot tags match the tags in the preset tag library; Based on the shot tags of each shot segment in the reference video, a corresponding tag sequence is formed; Clustering is performed on the tag sequences of each of the reference videos to obtain multiple video clusters; each video cluster includes multiple of the reference videos. For any of the video clusters, a video frame template corresponding to the preset resource category is generated based on the structured information corresponding to each of the reference videos in the video cluster; the video frame template includes multiple shot storyboard templates, used to generate multi-shot videos according to each of the shot storyboard templates.

[0006] Secondly, this application provides a video processing method, the method comprising: In response to a selection instruction for a video frame template, a target video frame template corresponding to a target resource category is obtained; the video frame template is obtained based on the video processing method of the first aspect described above or any corresponding embodiment thereof; In response to a video generation instruction for a resource to be processed, resource description information of the resource to be processed is obtained; the resource to be processed is a resource belonging to the target resource category. Based on the resource description information, the storyboard templates for at least some shots in the target video frame template are adjusted to obtain the corresponding target storyboards; Based on the prompts containing the respective target storyboards, the target video corresponding to the resource to be processed is generated.

[0007] In some optional implementations, adjusting the storyboard templates for at least some shots in the target video frame template based on the resource description information to obtain corresponding target storyboards includes: If the storyboard template contains value feature tags, the resource value feature information corresponding to the value feature tags is determined based on the resource description information; the value feature tags are included in the shot tags obtained by parsing the shot segments. The value feature labels in the storyboard template are replaced according to the resource value feature information to obtain the target storyboard corresponding to the storyboard template.

[0008] Thirdly, this application provides a video processing apparatus, the apparatus comprising: The video acquisition module is used to acquire multiple reference videos related to a preset resource category; A shot segmentation module is used to perform shot segmentation processing on any of the aforementioned reference videos to obtain various shot segments of the reference video. The processing module parses each shot segment according to a preset tag library corresponding to the preset resource category to determine the structured information of each shot segment. The structured information includes the description information and shot tags of the shot segment, and the shot tags match the tags in the preset tag library. Based on the shot tags of each shot segment in the reference video, a corresponding tag sequence is formed. Based on the tag sequences of each reference video, clustering processing is performed to obtain multiple video clusters. Each video cluster includes multiple reference videos. The template generation module is used to generate a video frame template corresponding to the preset resource category for any video cluster, based on the structured information corresponding to each reference video in the video cluster; the video frame template includes storyboard templates for multiple shots, and is used to generate multi-shot videos according to each storyboard template.

[0009] Fourthly, this application provides a video processing apparatus, the apparatus comprising: The acquisition module is configured to acquire a target video frame template corresponding to a target resource category in response to a selection instruction for a video frame template; the video frame template is obtained based on the video processing method of the first aspect described above or any corresponding embodiment thereof; and to acquire resource description information of the resource to be processed in response to a video generation instruction for a resource to be processed; the resource to be processed is a resource belonging to the target resource category. The adjustment module is used to adjust the storyboard template of at least some shots in the target video frame template according to the resource description information, so as to obtain the corresponding target storyboard. The video generation module is used to generate the target video corresponding to the resource to be processed based on the prompt information containing each of the target storyboards.

[0010] Fifthly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the video processing method described in the first aspect or any corresponding embodiment.

[0011] In a sixth aspect, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the video processing method described in the first aspect or any of its corresponding embodiments.

[0012] In a seventh aspect, this application provides a computer program product, including computer instructions for causing a computer to perform the video processing method described in the first aspect or any corresponding embodiment thereof.

[0013] The video processing method provided in this application performs shot-level segmentation on reference videos and uses a preset tag library to determine the structured information of each shot segment, thereby constructing a tag sequence corresponding to the reference videos. Using this tag sequence as the basic feature of the framework, the various reference videos are clustered to identify reference videos with relatively consistent frameworks. Features of reference videos belonging to the same video cluster are then extracted, forming a video framework template containing various shot-level storyboard templates. This embodiment refines the granularity of the video framework template from the video level to the shot segment level, thus capturing the structured information of visual dimensions such as the composition and shooting techniques of each shot. The framework description is more refined, improving the reproducibility of the video template framework. Using a preset tag library to represent shot tags ensures the consistency and relevance of tag meanings across multiple stages, including parsing, clustering, and generation. This allows for the clustering of videos with similar characteristics, ensuring the accuracy of the clustering results. Furthermore, it enables the extraction of common structural features from videos of the same type, forming a video framework template. This video framework template can be directly used as a video generation instruction, and the shots of the generated video are consistent with the corresponding shot tags in the template, improving the quality of video generation. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a schematic flowchart of a first method for video processing according to an embodiment of this application; Figure 3 This is a schematic diagram of a second process of a video processing method according to an embodiment of this application; Figure 4 This is a schematic diagram of reference video structured information according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the determination of segmentation points for audio-visual dual channels according to an embodiment of this application; Figure 6 This is a schematic diagram of a third process of a video processing method according to an embodiment of this application; Figure 7 This is a structural block diagram of a video processing apparatus according to an embodiment of this application; Figure 8 This is another structural block diagram of a video processing apparatus according to an embodiment of this application; Figure 9 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0018] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0019] Before providing a detailed description of the embodiments of this application, some of the terms and concepts involved in the embodiments of this application will be explained. These explanations are intended to make the embodiments of this application easier to understand and should not be considered as limiting the scope of protection claimed in this application.

[0020] (1) Spec: A general abbreviation for Specification, which refers to the visual specifications of a product. It is used to describe the structured information of the product’s objective physical parameters (such as material, size, weight, etc.) and sales differentiation attributes (such as version, package configuration, etc.).

[0021] (2) GMV (Gross Merchandise Volume): refers to the total amount of all orders on the platform within a certain period of time, which may include the amount of paid, unpaid, cancelled and returned orders.

[0022] (3) Multimodal large model: A large language model that can understand multiple modal data such as text and images at the same time. For example, a multimodal large model can be used to identify video frames and related text.

[0023] (4) few-shot example: When calling a large model, manually labeled standard samples are input as context examples to make the model output consistent with the label system.

[0024] With the development of short videos, short video e-commerce has become a core channel for brand marketing. High-quality "viral" short videos exhibit replicable patterns in content structure, camera rhythm, and narration framework. How to automatically extract reusable content frameworks from a massive number of viral videos and efficiently apply them to the generation of one's own product videos is a crucial technical challenge in the current content marketing field.

[0025] Currently, the industry's analysis and framework extraction of viral short videos mainly rely on the following methods: Method 1: Manual Viewing and Analysis. Operations or content staff manually watch viral videos, extract key points based on experience, and output unstructured text descriptions. This method relies on individual skills and is difficult to standardize.

[0026] Method 2: Voiceover Script Extraction Solution. This method extracts the voiceover script from the video using Automatic Speech Recognition (ASR) and generates content based on relevance to trending topics. However, this solution does not perform frame-by-frame semantic analysis of the video footage itself, and therefore cannot capture visual framing information such as composition and camera rhythm.

[0027] Method 3: Video Material Matching Solution. This method generates content based on a script library and a material library. Typically, the user selects materials, and a complete video is generated directly, but the quality of the generated video is difficult to guarantee.

[0028] Current solutions rely on manual experience to break down videos one by one and extract patterns, which is inefficient and difficult to systematize. Generating videos based on spoken scripts or overall descriptions cannot provide structured analysis of visual dimensions such as camera rhythm, composition, and shooting techniques. The framework is difficult to replicate, and the extracted framework cannot be directly used as video generation instructions.

[0029] As one optional application scenario in the embodiments of this application, such as Figure 1 As shown, application 101 is installed in terminal device 110, and user 130 can interact with application 101 through terminal device 110 and / or access device of terminal device 110.

[0030] For example, application 101 can be any application that provides question-and-answer related services. For instance, application 101 could be a question-and-answer interactive application, such as a text-to-text application, an image-to-text application, etc. Figure 1In the application scenario shown, if application 101 is active, the terminal device 110 can display the interface 102 of application 101. The interface 102 may include various pages that application 101 can provide, such as interactive pages, settings pages, query pages, etc.

[0031] In some embodiments, terminal device 110 is communicatively connected to server 120 to provide services to application 101. Terminal device 110 may be a mobile terminal, fixed terminal, or portable terminal, including but not limited to mobile phones, desktop computers, laptop computers, multimedia tablets, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 may also support any type of interface, and server 120 may be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, and computing devices in cloud environments.

[0032] It should be noted that, Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of this application.

[0033] This application provides a video processing method that performs shot-level segmentation on a reference video and uses a preset tag library to determine the structured information of each shot segment, thereby constructing a tag sequence corresponding to the reference video. Using this tag sequence as the basic feature of the framework, the various reference videos are clustered to identify reference videos with relatively consistent frameworks. Features of reference videos belonging to the same video cluster are then extracted, forming a video framework template containing various shot-level storyboard templates. This embodiment refines the granularity of the video framework template from the video level to the shot segment level, thus capturing the structured information of visual dimensions such as the composition and shooting techniques of each shot. The framework description is more refined, improving the reproducibility of the video template framework. Using a preset tag library to represent shot tags ensures the consistency and relevance of tag meanings across multiple stages, including parsing, clustering, and generation. While ensuring high accuracy in clustering results, the framework extraction results can also be directly used as video generation instructions, and the shots in the generated video are consistent with the corresponding shot tags in the template, improving the quality of video generation.

[0034] According to an embodiment of this application, a video processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] This embodiment provides a video processing method that can be applied to the aforementioned terminal device or server. Figure 2 This is a flowchart of a video processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps.

[0036] Step S201: Obtain multiple reference videos related to the preset resource category.

[0037] The video processing method provided in this embodiment is mainly used to generate a replicable video template based on a video related to a certain resource, so that other users can directly generate the videos they need based on the video framework template. The resource can be an object for marketing and promotion based on the video, which can be a specific product, such as a mobile phone, cat food, or face mask, or a digital product, such as a newly released song or movie.

[0038] Various resources can be pre-classified into multiple categories, i.e., multiple preset resource categories. These preset resource categories may include, for example, physical goods and digital items; or they may be divided according to resource type, including: food, clothing, toys, digital products, skincare, etc. Alternatively, a more granular division can be made, for example, each resource may correspond to a preset resource category, such as mobile phones, cat food, face masks, etc. This embodiment does not limit the method of resource classification.

[0039] Since different types of resources may have different video creation patterns, this embodiment generates video frame templates for various preset resource categories. Accordingly, when a video frame template needs to be generated for a certain preset resource category, an existing video related to that preset resource category can be obtained, and this video will be referred to as a reference video.

[0040] For example, when you need to generate a video frame template for promoting a mobile phone, you can obtain promotional videos related to mobile phones and use some or all of them as reference videos for later use.

[0041] Step S202: For any reference video, perform shot segmentation processing on the reference video to obtain individual shot segments of the reference video.

[0042] In order to accurately extract the creative patterns in the reference video, this embodiment divides the reference video into shots to obtain the segment content corresponding to each shot, i.e., shot segments; subsequently, by processing the shot segments, shot-level feature extraction can be achieved.

[0043] The shot clip can be a portion of a video segment cut from a reference video, or it can be formed based on the content of a corresponding video segment in the reference video. For example, the shot clip may include keyframes from the relevant shot, audio data corresponding to the audio channel (which can represent the spoken content corresponding to the shot), etc.

[0044] In some examples, the segmentation rules for shot clips can be pre-configured, and these rules can be based on expert experience. For instance, the reference video can be divided into shot segments according to a preset time window (e.g., 3 to 5 seconds), which is suitable for narrated videos with a uniform pace. Alternatively, intelligent shot segmentation of the reference video can be performed based on models to adapt to different video formats.

[0045] Step S203: Based on the preset tag library corresponding to the preset resource category, each shot fragment is parsed to determine the structured information of each shot fragment. This structured information includes the description information and shot tags of the shot fragment, and the shot tags match the tags in the preset tag library.

[0046] In this embodiment, for each preset resource category, a set of multiple tags, i.e., a preset tag library, is pre-configured. A structured Spec configuration can be established for each resource to determine the corresponding Spec tags, which are then used as tags in the preset tag library.

[0047] This preset tag library can contain one type of tag or multiple types of tags. For example, it can include shooting tags to describe lens composition and shooting techniques, such as "close-up," "overhead shot," "movement tracking," and "product hovering." The library can also include value feature tags to represent the selling points of resources, describing their core benefits, such as "ingredient efficacy," "usage scenarios," "price advantage," and "comparative evaluation." Understandably, other types of tags can be set for this preset tag library based on actual needs, such as people tags and music tags.

[0048] After segmenting the footage, for any given footage segment, descriptive information related to that footage segment can be identified. This descriptive information is text-formatted information used to describe the characteristics or content of the footage segment; for example, the descriptive information may include a natural language description of the content within the footage segment, such as "close-up display of the bottle texture while holding a product." Alternatively, if the footage segment contains spoken words or subtitles, the spoken words or subtitles within the footage segment can be extracted and used as part of the descriptive information.

[0049] Furthermore, for any given shot, tag matching can be performed based on a preset tag library corresponding to the preset resource category to determine the shot-level tag corresponding to that shot, i.e., the shot tag. Specifically, the shot tag is selected from the preset tag library as the tag that best matches the shot segment.

[0050] For example, based on the shooting technique corresponding to the shot, the corresponding shooting tag can be selected from the preset tag library, such as "overhead shot"; if the shot currently involves selling points or other content, the value feature tags of the shot can be further identified, such as "ingredient efficacy" or "ingredient efficacy".

[0051] In this embodiment, for any shot segment, the descriptive information and shot tag of that shot segment can be identified separately, and finally the identified descriptive information and shot tag are combined into structured information. Alternatively, a large language model can be used for simultaneous recognition, directly outputting structured information containing descriptive information and shot tags.

[0052] For example, for each shot segment, a multimodal large model can be invoked for semantic parsing, thereby directly obtaining the structured information of the shot segment. Alternatively, all shot segments of a reference video can be input into the multimodal large model at once (e.g., the complete reference video and the division method of each shot segment can be input into the multimodal large model), instructing the multimodal large model to output the structured information of each shot segment.

[0053] During the parsing process, a few-shot prompting learning approach can be adopted. Specifically, manually annotated standard shot segments (including correct mapping examples of scene descriptions and shot labels) can be input as context examples into the multimodal large model to guide it to output correct structured information, with the shot labels consistent with the Spec label system. For example, multiple (e.g., 5, 10, etc.) standard samples can be input, each containing a shot segment and its corresponding scene description, the matched shooting label (specifically, a label ID, etc.), and a value feature label, enabling the large model to learn the label mapping rules.

[0054] By using few-shot-guided multimodal large model parsing, we ensure that the label output is strictly aligned with the preset label system, which guarantees the consistency of labels in multiple stages such as parsing, clustering, and mapping.

[0055] Step S204: Based on the shot tags of each shot segment in the reference video, form a corresponding tag sequence.

[0056] For any reference video, after determining the shot tags for each shot segment, the shot tags can be integrated according to the chronological order of the shot segments in the reference video to form a serialized feature containing the shot tags, i.e., a tag sequence. This tag sequence can represent the framing operations of a single video and can explain the shooting techniques, selling points, etc. of each shot.

[0057] The tag sequence can be stored using data structures such as arrays and lists. For example, if a reference video is split into four shot segments with corresponding shot tags of [close-up, ingredients and efficacy], [overhead shot, usage scenario], [motion follow-up shot, price advantage], and [close-up, comparison review], then the tag sequence for the reference video can be: [(close-up, ingredients and efficacy); (overhead shot, usage scenario); (motion follow-up shot, price advantage); (close-up, comparison review)].

[0058] Step S205: Clustering is performed based on the tag sequences of each reference video to obtain multiple video clusters. Each video cluster includes multiple reference videos.

[0059] In this embodiment, after determining the tag sequence of each reference video, the structural similarity between the tag sequences can be calculated, and reference videos with highly similar tag sequences can be grouped into the same cluster to achieve the clustering of reference videos. Each type of video corresponds to a video cluster, and the video cluster includes multiple reference videos belonging to the same type (i.e., with highly similar tag sequences).

[0060] The structural similarity between label sequences can be determined by calculating cosine similarity, among other methods. This embodiment does not limit the specific calculation method for structural similarity. Clustering algorithms such as DBSCAN and K-Means can be used, or clustering can be based on a pre-trained clustering model; this embodiment also does not limit this approach.

[0061] In some examples, the preset tag library may also include style tags to describe the overall type and style of the video, such as "unboxing," "review," "lifestyle," and "product recommendation." These style tags represent the overall style at the video level and are not shot-level tags.

[0062] For the same preset resource category, since videos of different styles (such as evaluation videos, life scene videos, etc.) may have different creative rules, different styles of video frame templates can be generated when generating video frame templates later.

[0063] For example, after obtaining multiple reference videos of a certain preset resource category, the style tags corresponding to each reference video can be identified. Then, reference videos with the same style tags are grouped together, and subsequent clustering processes are performed to cluster reference videos of each style separately. Different style frames do not perform cross-category clustering, so that video frame templates of each style can be maintained separately to ensure the consistency of the frame content.

[0064] Step S206: For any video cluster, generate a video frame template corresponding to a preset resource category based on the structured information corresponding to each reference video in the video cluster. This video frame template includes storyboard templates for multiple shots, used to generate multi-shot videos according to each storyboard template.

[0065] In this embodiment, after completing the tag sequence clustering, for multiple reference videos within the same video cluster, feature extraction can be performed on the structured information in these reference videos to summarize and identify the common structure of each reference video within the same cluster, thereby obtaining a general video frame template. This video frame template can be used to generate videos corresponding to resources belonging to a preset resource category, such as mobile phone promotion videos. It can be understood that a corresponding video frame template can be generated for any video cluster.

[0066] Since each reference video includes shot-level structural information, when summarizing the features of multiple reference videos, the common structure at the shot level is still extracted to determine the common features corresponding to each shot, and then the storyboard template corresponding to the shot is generated based on the common features.

[0067] This storyboard template is a structured template that includes shot format requirements. For example, the storyboard template includes shot number, shot duration, shooting tags, and specific scene descriptions. When users subsequently generate videos using the video framework template, they can determine the shooting method and scene content for each shot according to the storyboard template, thus generating a final video containing multiple shots.

[0068] The video processing method provided in this embodiment performs shot-level segmentation on reference videos and uses a preset tag library to determine the structured information of each shot segment, thereby constructing a tag sequence corresponding to the reference videos. Using this tag sequence as the basic feature of the framework, the various reference videos are clustered to identify reference videos with relatively consistent frameworks. Features of reference videos belonging to the same video cluster are then extracted, forming a video framework template containing various shot-level storyboard templates. This embodiment refines the granularity of the video framework template from the video level to the shot segment level, thus capturing the structured information of visual dimensions such as the composition and shooting techniques of each shot. The framework description is more refined, improving the reproducibility of the video template framework. Using a preset tag library to represent shot tags ensures the consistency and relevance of tag meanings across multiple stages, including parsing, clustering, and generation. This allows for the clustering of videos with similar characteristics, ensuring the accuracy of the clustering results. Furthermore, it enables the extraction of common structural features from videos of the same type, forming a video framework template. This video framework template can be directly used as a video generation instruction, and the shots in the generated video are consistent with the corresponding shot tags in the template, improving the quality of video generation.

[0069] This embodiment provides a video processing method that can be applied to the aforementioned terminal device or server. Figure 3 This is a flowchart of a video processing method according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps.

[0070] Step S301: Obtain multiple reference videos related to the preset resource category.

[0071] Please see details Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0072] In some alternative implementations, video framework templates are typically generated from videos with good marketing performance (such as viral videos). Related solutions often rely on metrics such as views, popularity, CTR (click-through rate), and completion rate to select viral videos from a massive pool. However, these metrics are lagging indicators; by the time a video reaches its threshold, its lifecycle has often passed its peak. Furthermore, these metrics are disconnected from the resource's sales objectives; for example, a high CTR doesn't necessarily translate to high sales, leading to a mismatch between the learned video content framework and the resource's sales goals.

[0073] To ensure the marketing effectiveness of the selected videos, this embodiment uses transaction data for video filtering. Specifically, step S301, "obtaining multiple reference videos related to preset resource categories," includes steps a1 to a3.

[0074] Step a1: Obtain multiple original videos related to the preset resource category; the original videos are used to display the original resources belonging to the preset resource category.

[0075] In this embodiment, the original video refers to videos showcasing products of this resource category collected from various channels. Specifically, videos publicly posted by other users within a given time period (e.g., one day, one week) can be collected and used as the original video. Alternatively, users can provide videos to be analyzed, and these videos can be used as the original video.

[0076] It is understandable that when generating video frame templates for preset resource categories, it is only necessary to focus on the original videos related to the preset resource categories.

[0077] Step a2: Obtain the transaction data corresponding to each original video; the transaction data includes the transaction volume and / or transaction amount of the original video.

[0078] For any given original video used to promote or market corresponding resources, transaction data corresponding to that original video can be obtained. This transaction data is a quantitative indicator reflecting the transaction status of resources, and it can quantify the effectiveness of video marketing. Specifically, this transaction data may include transaction volume (such as the number of items sold) and / or transaction value (such as GMV).

[0079] Specifically, transaction data can be obtained through APIs (Application Programming Interfaces) provided by video platforms or third-party platforms, and can be obtained synchronously with the original videos. This means that the files and transaction data corresponding to each original video can be obtained from the relevant platform all at once. This embodiment does not limit the method of obtaining transaction data. Furthermore, obtaining transaction data requires relevant authorization. In actual processing, only necessary data such as transaction volume and transaction amount, as well as the correspondence between transaction data and corresponding videos, can be obtained. This transaction data is only used for backend processing and cannot be displayed on the frontend to avoid privacy data leakage and other issues.

[0080] Step a3: Use the original video whose transaction data meets the preset conditions as the reference video.

[0081] Pre-set conditions for video filtering based on transaction data, i.e., preset conditions. After determining the transaction data of the original video, it can be judged whether its transaction data meets the preset conditions. Original videos that meet the preset conditions are considered viral videos and are used as reference videos for subsequent processing.

[0082] The preset conditions could be a transaction amount exceeding a certain threshold (e.g., GMV exceeding 100,000 yuan) or a high transaction volume ranking (e.g., being among the top 100 in transaction volume). Alternatively, based on the transaction data, dissemination metrics such as play counts could be further introduced to calculate the overall score of the original video. If the overall score exceeds a threshold, it is considered to meet the preset conditions.

[0083] This embodiment uses transaction volume and / or transaction amount as the screening criteria, ensuring that the reference video set directly corresponds to resource transaction results, thus avoiding the problem of a disconnect between dissemination metrics and transaction goals. Videos with high dissemination metrics may just be gimmicks to attract clicks but have poor actual sales performance, while videos with high transaction amounts are the truly effective content for promotion and marketing, thereby ensuring that the video content framework being studied is consistent with the resource transaction goals.

[0084] Optionally, step a3 above, "using the original video whose transaction data meets the preset conditions as a reference video", may specifically include steps a31 to a33.

[0085] Step a31: For any original video, extract temporal features based on the transaction data of the original video to determine the temporal features used to represent the changes in the transaction data.

[0086] Step a32: Determine the growth potential indicators of transaction data in the future based on time series characteristics.

[0087] Step a33: Use the original videos with growth potential indicators greater than the preset threshold as reference videos.

[0088] In this embodiment, when screening the original videos, predictions are made based on the changes in the original video transaction data to select reference videos with the potential to become hits.

[0089] Specifically, for any original video, the temporal characteristics of the transaction data can be determined based on the changes in its transaction data. This involves acquiring transaction data corresponding to multiple time points, thereby extracting temporal characteristics, which are feature indicators reflecting the patterns of transaction data changes over time.

[0090] Taking transaction data, including GMV, as an example, time-series characteristics can include GMV time-series slope (growth rate) and GMV growth acceleration (rate of change of growth rate). If there are relevant transaction rankings, the speed of entering the rankings can also be determined (the GMV level of newly listed videos, or the time difference from publication to entry into the rankings), ranking volatility (whether it can rise steadily), etc.

[0091] Furthermore, the prediction model can be pre-trained, for example, based on the lifecycle data of a large number of historical viral videos. Based on this prediction model, the growth potential index of the original video's temporal characteristics in the future (e.g., the next 24 to 48 hours) can be determined. This growth potential index is a quantitative indicator or probability of explosive growth of the original video's transaction data in the future, based on the prediction of temporal characteristics. The growth potential index indicates whether the original video will become popular in the future, that is, whether it has the potential to become popular.

[0092] If the growth potential index of an original video is greater than a preset threshold (for example, the preset threshold is a preset probability of explosion, such as 0.8), it means that the original video is currently showing an upward trend, rather than a historical hit video that has completed its life cycle. Therefore, the original video can be used as a reference video.

[0093] This embodiment extracts temporal features and predicts growth potential to include videos in a reference video set during the window before a video becomes a viral hit, thus creating a valuable time window for downstream content creation. By utilizing the growth potential indicator as a forward-looking parameter, it is possible to capture videos that are about to go viral, thereby improving the timeliness and business value of video framework templates.

[0094] Step S302: For any reference video, perform shot segmentation processing on the reference video to obtain individual shot segments of the reference video.

[0095] Specifically, step S302, "performing shot segmentation on the reference video to obtain individual shot segments of the reference video," may include steps S3021 to S3022.

[0096] Step S3021: Perform semantic segmentation on the reference video to obtain multiple semantic segmentation points.

[0097] Step S3022: Divide the reference video into shots based on multiple semantic segmentation points to obtain shot segments between two adjacent semantic segmentation points.

[0098] When dividing a reference video into shots, using a fixed time window (e.g., 3 seconds) can easily lead to inaccurate shot division. Therefore, this embodiment performs semantic recognition on the reference video and automatically determines the segmentation points corresponding to different semantics based on the semantic recognition results. These semantic segmentation points can then be used to divide the reference video into shots based on each semantic segmentation point, resulting in corresponding shot segments. It can be understood that the video segment between two adjacent semantic segmentation points can be considered as a single shot segment.

[0099] For example, semantic judgment can be performed on shot transition points (such as product close-up → usage scene transition) based on semantic recognition models or large language models to identify semantic boundaries. These semantic boundaries used for switching scenes or content can be used as semantic segmentation points to segment into individual shot segments, which can ensure the semantic integrity of the content of each shot segment.

[0100] In addition, a shot segmentation template can be formed based on expert experience. Subsequent large AI models can use this shot segmentation template as a benchmark to perform segmentation, ensuring the consistency and professionalism of the resolution granularity.

[0101] In some alternative implementations, audio-visual misalignment is frequently encountered in actual video analysis projects. For example, we have found a key technical challenge: the points where the video shot changes do not correspond to the points where the spoken semantics change. Typical scenarios include: when the anchor says, "The key ingredient of this product is XX," the video may still be showing the previous usage scene; or the video may have already cut to a close-up of the product, but the spoken statement may still be continuing the closing statement of the previous sentence; or the video switch may occur in the middle of a complete spoken sentence. The problem of audio-visual misalignment directly affects the accuracy of subsequent segmentation, label recognition, and frame extraction.

[0102] To avoid audio-visual misalignment affecting the accuracy of segmentation results, this embodiment employs a dual-channel audio-visual collaborative segmentation method. Specifically, step S3021, "performing semantic segmentation on the reference video to obtain multiple semantic segmentation points," may include steps b1 to b2.

[0103] Step b1: Perform semantic segmentation of the reference video channels to obtain multiple candidate video segmentation points.

[0104] Step b2 involves performing audio channel semantic segmentation on the reference video to obtain multiple candidate audio segmentation points.

[0105] Furthermore, the aforementioned step S3022, "dividing the reference video into shots based on multiple semantic segmentation points to obtain shot segments between two adjacent semantic segmentation points," includes steps b3 to b6.

[0106] Step b3: Determine the corresponding target video segmentation point and target audio segmentation point based on the candidate video segmentation point and candidate audio segmentation point.

[0107] Step b4: Segment the image data in the reference video according to the target image segmentation points to obtain multiple image segments.

[0108] Step b5: Segment the audio data in the reference video according to the target audio segmentation points to obtain multiple audio segments.

[0109] Step b6: Match each video clip with each audio clip to obtain the corresponding shot clip; a shot clip includes a video clip and its corresponding audio clip.

[0110] In this embodiment, the reference video is divided into a video portion and an audio portion, and correspondingly, appropriate segmentation points are determined for each. Specifically, video channel semantic segmentation is performed on the video portion of the reference video to obtain multiple segmentation points, i.e., candidate video segmentation points; at this time, only the image frames in the reference video are considered, and the audio content is not. Similarly, audio channel semantic segmentation is performed on the audio portion of the reference video to obtain multiple segmentation points, i.e., candidate audio segmentation points; at this time, only the audio track in the reference video is considered, and the video content is not.

[0111] Among them, the video channel can perform shot switching point detection based on the video semantic understanding model to obtain a sequence of candidate cutting points at the video level; the audio channel can perform speech semantic boundary recognition based on the ASR (Automatic Speech Recognition) + semantic segmentation model to obtain a sequence of candidate cutting points at the speech level, that is, the sequence corresponding to the candidate audio segmentation points.

[0112] Each candidate screen segmentation point and candidate audio segmentation point can be directly used as the target screen segmentation point and target audio segmentation point for subsequent segmentation; alternatively, a comprehensive analysis can be performed based on both to determine the appropriate target screen segmentation point and target audio segmentation point. The number of target screen segmentation points and target audio segmentation points is the same.

[0113] For the image data (i.e., the image portion) in the reference video, segmenting it according to the target image segmentation points yields segments that are only related to the image and not to the audio (or spoken content), i.e., image segments. Similarly, for the audio data (i.e., the audio portion) in the reference video, segmenting it according to the target audio segmentation points yields segments that are only related to the audio and not to the image, i.e., audio segments. It can be understood that since image segments and audio segments are obtained based on their respective segmentation points, they are unaffected by audio-visual misalignment.

[0114] Each video clip and each audio clip can be matched one-to-one to form a shot clip that contains both video and audio. That is, the shot clip includes a video clip and the corresponding audio clip.

[0115] By analyzing and processing each shot segment, structured information for each shot segment can be generated. Figure 4 This illustrates a schematic diagram of the structured information of a reference video. For example... Figure 4As shown, the structured information of each shot segment in the reference video is presented in tabular form. This includes: video segments (i.e., image segments), narration extracted from audio segments, and keyframes within the video segments, such as frame 1-1, frame 1-2, etc. Furthermore, by combining the shot segments, the overall script structure, camera movement style, and descriptive information representing the scene can be extracted for each shot segment. Based on this camera movement style, corresponding shot tags can be determined. Figure 4 The structured information shown is primarily presented to the user, therefore the lens labels required for subsequent clustering processing are not shown.

[0116] Optionally, the video segmentation points are analyzed and processed using audio segmentation points as the primary benchmark. Step b3 above, "determine the corresponding target video segmentation points and target audio segmentation points based on candidate video segmentation points and candidate audio segmentation points," includes steps b31 to b38.

[0117] Step b31: For any candidate image segmentation point, determine the minimum distance between the candidate image segmentation point and the nearest candidate audio segmentation point.

[0118] Step b32: If the minimum spacing is less than a preset threshold, the candidate image segmentation point is used as the target image segmentation point.

[0119] Step b33: If the minimum spacing is greater than a preset threshold, determine the first candidate audio segmentation point and the second candidate audio segmentation point that are adjacent to the candidate screen segmentation point.

[0120] Step b34: Obtain the text content between the first candidate audio segmentation point and the second candidate audio segmentation point in the reference video.

[0121] Step b35: Divide the text content into a first sub-text and a second sub-text based on the candidate screen segmentation points.

[0122] Step b36: If the semantics of the first subtext and / or the second subtext are incomplete, delete the candidate screen segmentation points.

[0123] Step b37: If both the first and second subtexts are semantically complete, the candidate screen segmentation points are used as the target screen segmentation points, and target audio segmentation points corresponding to the positions of the candidate screen segmentation points are generated.

[0124] Step b38: After traversing each candidate frame segmentation point, each candidate audio segmentation point is used as the corresponding target audio segmentation point.

[0125] Figure 5 This diagram illustrates one method for determining the split point in a dual-channel audio-visual setup. (Example:) Figure 5As shown, if a reference video is analyzed using both audio and video channels, candidate frame segmentation points and candidate audio segmentation points are determined respectively; Figure 5 For example, the candidate screen segmentation points include: A1, A2, A3, A4, A5, and the candidate audio segmentation points include: B1, B2, B3.

[0126] For any candidate frame segmentation point, first determine the minimum distance between that candidate frame segmentation point and the nearest candidate audio segmentation point; for example, the distance between that candidate frame segmentation point and each candidate audio segmentation point can be determined, and the minimum distance is taken as the minimum distance. This is understandable. Figure 5 The horizontal axis represents the timeline of the reference video, and the spacing is a time distance or time difference.

[0127] If the minimum spacing is less than a preset threshold, it means that the candidate frame segmentation point is close to the beginning or end of the spoken sentence, and the segmentation results of the audio and video channels are consistent. Therefore, the candidate frame segmentation point can be directly used as the target frame segmentation point. Figure 5 As shown, the candidate screen segmentation points A1, A3, and A5 correspond one-to-one with the candidate audio segmentation points B1, B2, and B3, and the spacing is very small. Therefore, the candidate screen segmentation points A1, A3, and A5 can be directly used as the target screen segmentation points.

[0128] For candidate image segmentation points A2 and A4, which are far from each candidate audio segmentation point, a second calibration is required.

[0129] Specifically, for the candidate video segmentation point A2, the first and second candidate audio segmentation points adjacent to it can be determined, namely candidate audio segmentation points B1 and B2. Furthermore, ASR recognition is performed on the portion of video between candidate audio segmentation points B1 and B2 to determine the text content between them; then, based on the time point corresponding to the candidate video segmentation point A2, the text content is segmented to obtain the corresponding first and second sub-texts.

[0130] For the first and second sub-texts, further determine whether they are semantically complete. If they are semantically complete, it means that the two segments of the speech (i.e., the first and second sub-texts) can still express complete and reasonable semantics. Therefore, the entire speech (i.e., the text content) can be further split, and step b37 can be executed. Otherwise, it means that the two segments of the speech after being split according to the candidate screen segmentation point cannot independently express complete semantics. This belongs to a scene with one sentence and multiple shots. Therefore, the entire speech needs to be retained without splitting. Instead, the screen descriptions of multiple shots need to be merged into a whole shot description, that is, the candidate screen segmentation point is deleted, and it is transformed into a standard one-sentence one-shot structure.

[0131] It is understandable that, for candidate screen segmentation point A2, the first sub-text mentioned above is the text between segmentation points B1 and B4, and the second sub-text is the text between segmentation points B4 and B2.

[0132] Similarly, for candidate screen segmentation point A4, the first and second candidate audio segmentation points adjacent to it can also be determined, namely candidate audio segmentation points B2 and B3. Then, steps b34 to b37 can be executed, which will not be described in detail here. Figure 4 Taking the candidate screen segmentation point A4 as an example, which is not suitable for segmenting voiceovers, the candidate screen segmentation point A4 was discarded.

[0133] Furthermore, since there may be situations where the audio channel deems segmentation necessary, but the video channel deems it unnecessary, after traversing each candidate video segmentation point in step b38, only the candidate audio segmentation point corresponding to the target video segmentation point can be used as the target audio segmentation point. Alternatively, for a candidate audio segmentation point that does not correspond to any candidate video segmentation point (or target video segmentation point) (for example, the sum of the distances between the candidate audio segmentation point and the nearest candidate video segmentation point is greater than a preset threshold), a comprehensive video and audio dual-channel analysis can be performed on the candidate audio segmentation point to determine whether it can be used as a segmentation point.

[0134] For example, similar to the processing methods in steps b35 to b37 above, if the image is segmented using the above candidate audio segmentation points and the segmented image is still semantically complete, it can be used as a segmentation point, and a new target image segmentation point needs to be generated; conversely, if the segmented image is not semantically complete, the candidate audio segmentation point needs to be discarded, that is, it cannot be used as a target audio segmentation point.

[0135] The audio-visual dual-channel collaborative segmentation method provided in this embodiment overcomes the technical deficiency of single-modal segmentation methods in ensuring the semantic integrity of paragraphs through parallel analysis of visual and audio channels, time axis comparison, and secondary calibration. It can provide a clean and complete input data foundation for subsequent multimodal large-scale model label recognition, and improve the accuracy of label recognition and the replicability of the framework.

[0136] Step S303: Based on the preset tag library corresponding to the preset resource category, each shot fragment is parsed to determine the structured information of each shot fragment. This structured information includes the description information and shot tags of the shot fragment, and the shot tags match the tags in the preset tag library.

[0137] Please see details Figure 2 Step S203 of the illustrated embodiment will not be described again here.

[0138] Step S304: Based on the shot tags of each shot segment in the reference video, form a corresponding tag sequence.

[0139] Please see details Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0140] Step S305: Clustering is performed based on the tag sequences of each reference video to obtain multiple video clusters. Each video cluster includes multiple reference videos.

[0141] Please see details Figure 2 Step S205 of the illustrated embodiment will not be described again here.

[0142] Step S306: For any video cluster, generate a video frame template corresponding to a preset resource category based on the structured information corresponding to each reference video in the video cluster. This video frame template includes storyboard templates for multiple shots, used to generate multi-shot videos according to each storyboard template.

[0143] Please see details Figure 2 Step S206 of the illustrated embodiment will not be described again here.

[0144] In some optional implementations, the method further includes: after generating the video frame template, modifying the video frame template in response to a modification operation on the video frame template; the modification operation includes at least one of adjusting the shot order, modifying the shot duration, modifying the shot label, and deleting the shot.

[0145] In this embodiment, after automatically generating a video frame template, the template can be displayed to the user, and the user can actively adjust the template as needed. Specifically, during the user's use of the terminal device, the video frame template can be displayed based on the terminal device's graphical user interface; the user can modify the video frame template by triggering a modification operation through the graphical user interface.

[0146] Users can adjust the shot order (e.g., swapping shot 1 and shot 2), modify shot duration (e.g., increasing shot duration), and modify shot labels (e.g., changing shooting labels, such as changing an overhead shot to a close-up). Furthermore, if some shots are not needed, users can actively delete them, that is, delete the storyboard template corresponding to the shot, to ensure the quality of the video frame template.

[0147] The video frame templates that have been manually calibrated can be officially added to the library to form a universal and replicable frame template library for users to use when generating videos.

[0148] Optionally, for each shot's corresponding storyboard template, in addition to structured features such as shot number, shot duration, and shooting tags, the content intent information and constraints of the content intent can also be generated.

[0149] The content intent information is a summary of the content effect goals derived from corresponding shots in multiple reference videos. It is a description of the content intent based on natural language, that is, what content effect the shot needs to achieve. For example, the content intent information could be: evoking user resonance (targeting pain point scenarios), highlighting core selling points (ingredients / processes / differentiated advantages), creating a usage atmosphere (incorporating everyday scenarios), creating conflict and contrast (before and after effects), establishing social recognition (multiple users / word-of-mouth endorsement), presenting usage effects (results display), and guiding action decisions (limited-time offers / purchase incentives), etc.

[0150] Constraints represent the rules that must be followed when generating storyboard content. For example, constraints may include: "must match the core selling points of this product", "must match the target audience tags of this product", "must conform to the scene tags of this product", "must comply with advertising laws", etc.

[0151] In this process, large AI models can be used for planning to determine the content intent information and constraints of each shot. For example, by summarizing and analyzing corresponding shots in multiple reference videos within the same video cluster, it can be identified that although all shots at that position have different specific content, they have the same function (such as highlighting the core selling point), or although they have different forms of expression, they all evoke user resonance. The large model can then extract these common contents to form the content intent information for that position.

[0152] The above descriptions of the scene are objective descriptions of the shot, generally containing more detailed information; while the content intent information is a summary of the information, generally more concise. For example, the description of a shot might be: "The female protagonist takes two gold canned foods out of a box, shows them to the camera one by one, and places them on the table; she explains as she operates them, with a natural expression." The summarized content intent information could be: "Highlighting the actual appearance of the product to enhance visual appeal." The video processing method provided in this embodiment utilizes objective transaction data such as GMV to identify reference videos for viral content. This allows the learning direction of video creation patterns to directly correspond to resource transaction results, thereby improving the return on investment (ROI) of content. By using shot segmentation and multimodal segment-by-segment analysis, the video content is decomposed into a structured tag sequence, achieving accurate capture of the basic visual framework and ensuring accurate differentiation of reference videos with similar features after clustering. Semantic segmentation ensures the semantic integrity of each shot segment, avoiding the fragmentation of a complete spoken line into multiple segments or the mixing of irrelevant visual content into a single segment, providing an accurate data foundation for subsequent processing. The use of tag sequences for frame clustering and manual calibration significantly reduces the workload of manual analysis while maintaining frame quality, supporting the large-scale frame summarization of viral videos.

[0153] This embodiment provides another video processing method, which can be applied to the aforementioned terminal devices, such as smartphones and tablets. Figure 6 This is a flowchart of a video processing method according to an embodiment of this application, such as... Figure 6 As shown, the process includes the following steps.

[0154] Step S601: In response to the selection instruction for the video frame template, obtain the target video frame template corresponding to the target resource category. This video frame template is obtained based on the video processing method provided in any of the above embodiments.

[0155] In this embodiment, based on the above... Figure 2 , Figure 3 The embodiments shown can generate multiple video frame templates, thereby forming a corresponding template library. When a user needs to generate their desired video based on a template, they can select a suitable template, i.e., the target video frame template, according to their needs.

[0156] For example, the user's terminal device provides a system interface that displays various available video frame templates. Through human-computer interaction, the user can trigger the corresponding selection command and choose the target video frame template they need.

[0157] Understandably, the target video framework template also includes storyboard templates for each shot, such as shot number, suggested duration, shooting tags, and shot function descriptions.

[0158] Step S602: In response to the video generation instruction for the resource to be processed, obtain the resource description information of the resource to be processed. The resource to be processed is a resource belonging to the target resource category.

[0159] When generating a video, the corresponding resource, i.e., the resource to be processed, needs to be specified. For example, if a user needs to promote a brand's product, the product can be used as the resource to be processed. Furthermore, for any resource, associated resource description information can be maintained. This resource description information refers to structured information describing the characteristics of the resource to be processed, which may include resource attributes, selling points, target audience, scene tags, and other information.

[0160] Specifically, a corresponding structured tag configuration (Spec) can be pre-established for each resource, including the resource description information. For example, when the resource to be processed is a facial care product, the resource description information may include: product attributes (such as "suitable for sensitive skin", "repairs the skin barrier"), selling points (such as "contains XX ingredients", "effective in 7 days"), target audience (such as "women aged 25-35"), and scenario tags (such as "daily skincare", "overnight repair"), etc.

[0161] In this embodiment, the resource category corresponding to the target video frame template selected by the user is the target resource category. To ensure the video generation effect, the category corresponding to the resource to be processed should also be the target resource category. It can be understood that this embodiment does not limit the execution order of the above steps S601 and S602; if the user selects the video frame template first and then determines the resource to be processed (i.e., executes step S601 first and then step S602), then when determining the resource to be processed, the user can be prohibited from selecting resources that are not in the target resource category.

[0162] Step S603: Based on the resource description information, adjust the storyboard templates of at least some shots in the target video frame template to obtain the corresponding target storyboards.

[0163] In this embodiment, the storyboard template in the target video frame template is a framework content applicable to various resources and needs to be adaptively adjusted based on the actual situation. Specifically, after obtaining the resource description information, the storyboard template of a certain shot can be adjusted to obtain the script used subsequently for that shot, i.e., the target storyboard script.

[0164] In some optional implementations, the above step S603, "adjusting the storyboard template of at least some shots in the target video frame template according to the resource description information to obtain the corresponding target storyboard," may specifically include the following steps c1 to c2.

[0165] Step c1: If the storyboard template contains value feature tags, determine the resource value feature information corresponding to the value feature tags based on the resource description information; the value feature tags are included in the shot tags obtained by parsing and processing the shot segments.

[0166] Step c2: Replace the value feature labels in the storyboard template with the resource value feature information to obtain the target storyboard corresponding to the storyboard template.

[0167] As shown in the above embodiments, lens tags can include not only shooting tags but also value feature tags. In this embodiment, value feature tags refer to product selling point tags that serve as placeholders in the video frame template, such as "ingredient efficacy," "usage scenarios," and "price advantage." Correspondingly, the resource description information generally contains information corresponding to some of the value feature tags, namely resource value feature information. This resource value feature information is the specific selling point content of the resource to be processed, such as "contains XX ingredients, effective in 7 days" and "suitable for late-night scenarios."

[0168] The value feature labels in the storyboard template serve as replaceable placeholders. These labels can be replaced based on resource value feature information, thereby obtaining the target storyboard that can be used subsequently.

[0169] In this case, resource value characteristic information may only be a general description of the resource's selling points. At this point, it can also be converted into text that is more easily accepted by consumers based on AI big data models. For example, the selling point (resource value characteristic information) of cat food is: 99% fresh meat rate. The converted consumer language is: high-quality ingredients, high meat content and easy to digest. This converted content can be used as spoken text, and then the target storyboard can be generated.

[0170] The target storyboard can include: shot number, suggested duration, shooting tags, and spoken text, which can be directly used as input instructions for the video generation module without the need for manual content translation.

[0171] Step S604: Generate the target video corresponding to the resource to be processed based on the prompt information containing the storyboards for each target.

[0172] In this embodiment, as shown above, the target storyboard contains the relevant information required for each shot. Therefore, based on this, prompt information, such as prompt words, can be generated for input into the large language model. This prompt information serves as the generation instruction for each target storyboard and can be directly input into the large language model, thereby generating short video content adapted to this product for the resource to be processed, i.e., the target video.

[0173] The video processing method provided in this embodiment allows users to directly utilize pre-summarized video framework templates to generate the videos they need, automatically replicating the creation patterns of viral videos and increasing the likelihood of the generated target videos becoming viral hits. Furthermore, by using the Spec tag mapping mechanism, value feature tags can be directly replaced based on resource value characteristic information, thereby directly converting viral frameworks into storyboards for this product, eliminating the manual translation step of framework understanding and significantly improving framework replication efficiency.

[0174] This embodiment also provides a video processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0175] This embodiment provides a video processing device, such as... Figure 7 As shown, the device includes: The video acquisition module 701 is used to acquire multiple reference videos related to a preset resource category; The shot segmentation module 702 is used to perform shot segmentation processing on any of the reference videos to obtain various shot segments of the reference video. Processing module 703 parses each of the shot segments according to a preset tag library corresponding to the preset resource category to determine the structured information of each shot segment; the structured information includes the description information and shot tags of the shot segment, and the shot tags match the tags in the preset tag library; based on the shot tags of each shot segment in the reference video, a corresponding tag sequence is formed; clustering is performed on the tag sequences of each reference video to obtain multiple video clusters; each video cluster includes multiple reference videos; The template generation module 704 is used to generate a video frame template corresponding to the preset resource category for any video cluster based on the structured information corresponding to each reference video in the video cluster; the video frame template includes storyboard templates for multiple shots, and is used to generate multi-shot videos according to each storyboard template.

[0176] In some optional implementations, acquiring multiple reference videos related to a preset resource category includes: Acquire multiple original videos related to a preset resource category; the original videos are used to display original resources belonging to the preset resource category; Obtain transaction data corresponding to each of the original videos; the transaction data includes the transaction volume and / or transaction amount of the original videos; The original video containing the transaction data that meets the preset conditions is used as the reference video.

[0177] In some optional implementations, using the original video that meets preset conditions as the reference video includes: For any of the original videos, time-series features are extracted based on the transaction data of the original videos to determine time-series features that represent the changes in the transaction data; Based on the time-series characteristics, determine the growth potential indicators of the transaction data in the future; Original videos whose growth potential index is greater than a preset threshold are used as reference videos.

[0178] In some optional implementations, the step of performing shot segmentation on the reference video to obtain individual shot segments of the reference video includes: The reference video is semantically segmented to obtain multiple semantic segmentation points; The reference video is divided into shots based on multiple semantic segmentation points to obtain shot segments between two adjacent semantic segmentation points.

[0179] In some optional implementations, the semantic segmentation of the reference video to obtain multiple semantic segmentation points includes: The reference video is subjected to semantic segmentation of video channels to obtain multiple candidate video segmentation points; The reference video is subjected to audio channel semantic segmentation to obtain multiple candidate audio segmentation points; The step of dividing the reference video into shots based on multiple semantic segmentation points to obtain shot segments between two adjacent semantic segmentation points includes: Based on the candidate screen segmentation points and the candidate audio segmentation points, determine the corresponding target screen segmentation points and target audio segmentation points; The image data in the reference video is segmented according to the target image segmentation points to obtain multiple image segments; The audio data in the reference video is segmented according to the target audio segmentation point to obtain multiple audio segments; Each of the aforementioned video clips and each of the aforementioned audio clips is matched to obtain a corresponding shot clip; the shot clip includes a video clip and a corresponding audio clip.

[0180] In some optional implementations, determining the corresponding target video segmentation point and target audio segmentation point based on the candidate video segmentation points and the candidate audio segmentation points includes: For any candidate screen segmentation point, determine the minimum distance between the candidate screen segmentation point and the nearest candidate audio segmentation point; If the minimum spacing is less than a preset threshold, the candidate image segmentation point is taken as the target image segmentation point; If the minimum spacing is greater than a preset threshold, a first candidate audio segmentation point and a second candidate audio segmentation point adjacent to the candidate image segmentation point are determined. Obtain the text content between the first candidate audio segmentation point and the second candidate audio segmentation point in the reference video; The text content is divided into a first sub-text and a second sub-text based on the candidate screen segmentation points; If the semantics of the first sub-text and / or the second sub-text are incomplete, delete the candidate screen segmentation point; If both the first subtext and the second subtext are semantically complete, the candidate screen segmentation point is used as the target screen segmentation point, and a target audio segmentation point corresponding to the position of the candidate screen segmentation point is generated. After traversing each of the candidate screen segmentation points, each of the candidate audio segmentation points is taken as the corresponding target audio segmentation point.

[0181] In some optional implementations, the template generation module 704 is further configured to: After generating the video frame template, in response to a modification operation on the video frame template, the video frame template is modified; the modification operation includes at least one of adjusting the shot order, modifying the shot duration, modifying the shot label, and deleting the shot.

[0182] This embodiment provides a video processing device, such as... Figure 8 As shown, the device includes: The acquisition module 801 is configured to acquire a target video frame template corresponding to a target resource category in response to a selection instruction for a video frame template; the video frame template is obtained based on the video processing method of the first aspect described above or any corresponding embodiment thereof; and to acquire resource description information of the resource to be processed in response to a video generation instruction for a resource to be processed; the resource to be processed is a resource belonging to the target resource category. The adjustment module 802 is used to adjust the storyboard template of at least some shots in the target video frame template according to the resource description information to obtain the corresponding target storyboard. The video generation module 803 is used to generate the target video corresponding to the resource to be processed based on the prompt information containing each of the target storyboards.

[0183] In some optional implementations, adjusting the storyboard templates for at least some shots in the target video frame template based on the resource description information to obtain corresponding target storyboards includes: If the storyboard template contains value feature tags, the resource value feature information corresponding to the value feature tags is determined based on the resource description information; the value feature tags are included in the shot tags obtained by parsing the shot segments. The value feature labels in the storyboard template are replaced according to the resource value feature information to obtain the target storyboard corresponding to the storyboard template.

[0184] The video processing apparatus provided in this disclosure can execute the video processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0185] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0186] The following is a detailed reference. Figure 9 This diagram illustrates a suitable structural schematic for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 902 or a program loaded from memory 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device. The processor 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0187] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 9 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0188] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a memory 908, or installed from a ROM 902. When the computer program is executed by the processor 901, it performs the functions defined in the video processing method of embodiments of this application.

[0189] Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0190] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the video processing method shown in the above embodiments is implemented.

[0191] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0192] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A video processing method, characterized in that, The method includes: Retrieve multiple reference videos related to a preset resource category; For any of the aforementioned reference videos, the reference video is divided into shots to obtain individual shot segments of the reference video; Based on the preset tag library corresponding to the preset resource category, each of the shot segments is parsed to determine the structured information of each shot segment; the structured information includes the description information and shot tags of the shot segment, and the shot tags match the tags in the preset tag library; Based on the shot tags of each shot segment in the reference video, a corresponding tag sequence is formed; Clustering is performed on the tag sequences of each of the reference videos to obtain multiple video clusters; each video cluster includes multiple of the reference videos. For any of the video clusters, a video frame template corresponding to the preset resource category is generated based on the structured information corresponding to each of the reference videos in the video cluster; the video frame template includes multiple shot storyboard templates, used to generate multi-shot videos according to each of the shot storyboard templates.

2. The method according to claim 1, characterized in that, The acquisition of multiple reference videos related to a preset resource category includes: Acquire multiple original videos related to a preset resource category; the original videos are used to display original resources belonging to the preset resource category; Obtain transaction data corresponding to each of the original videos; the transaction data includes the transaction volume and / or transaction amount of the original videos; The original video containing the transaction data that meets the preset conditions is used as the reference video.

3. The method according to claim 2, characterized in that, The step of using the original video that meets the preset conditions for the transaction data as a reference video includes: For any of the original videos, time-series features are extracted based on the transaction data of the original videos to determine time-series features that represent the changes in the transaction data; Based on the time-series characteristics, determine the growth potential indicators of the transaction data in the future; Original videos whose growth potential index is greater than a preset threshold are used as reference videos.

4. The method according to claim 1, characterized in that, The step of dividing the reference video into shot segments to obtain individual shot fragments of the reference video includes: The reference video is semantically segmented to obtain multiple semantic segmentation points; The reference video is divided into shots based on multiple semantic segmentation points to obtain shot segments between two adjacent semantic segmentation points.

5. The method according to claim 4, characterized in that, The semantic segmentation of the reference video yields multiple semantic segmentation points, including: The reference video is subjected to semantic segmentation of video channels to obtain multiple candidate video segmentation points; The reference video is subjected to audio channel semantic segmentation to obtain multiple candidate audio segmentation points; The step of dividing the reference video into shots based on multiple semantic segmentation points to obtain shot segments between two adjacent semantic segmentation points includes: Based on the candidate screen segmentation points and the candidate audio segmentation points, determine the corresponding target screen segmentation points and target audio segmentation points; The image data in the reference video is segmented according to the target image segmentation points to obtain multiple image segments; The audio data in the reference video is segmented according to the target audio segmentation point to obtain multiple audio segments; Each of the aforementioned video clips and each of the aforementioned audio clips is matched to obtain a corresponding shot clip; the shot clip includes a video clip and a corresponding audio clip.

6. The method according to claim 5, characterized in that, The step of determining the corresponding target video segmentation point and target audio segmentation point based on the candidate video segmentation point and the candidate audio segmentation point includes: For any candidate screen segmentation point, determine the minimum distance between the candidate screen segmentation point and the nearest candidate audio segmentation point; If the minimum spacing is less than a preset threshold, the candidate image segmentation point is taken as the target image segmentation point; If the minimum spacing is greater than a preset threshold, a first candidate audio segmentation point and a second candidate audio segmentation point adjacent to the candidate image segmentation point are determined. Obtain the text content between the first candidate audio segmentation point and the second candidate audio segmentation point in the reference video; The text content is divided into a first sub-text and a second sub-text based on the candidate screen segmentation points; If the semantics of the first sub-text and / or the second sub-text are incomplete, delete the candidate screen segmentation point; If both the first subtext and the second subtext are semantically complete, the candidate screen segmentation point is used as the target screen segmentation point, and a target audio segmentation point corresponding to the position of the candidate screen segmentation point is generated. After traversing each of the candidate screen segmentation points, each of the candidate audio segmentation points is used as the corresponding target audio segmentation point.

7. A video processing method, characterized in that, The method includes: In response to a selection instruction for a video frame template, a target video frame template corresponding to the target resource category is obtained; the video frame template is obtained based on the video processing method as described in any one of claims 1 to 6; In response to a video generation instruction for a resource to be processed, resource description information of the resource to be processed is obtained; the resource to be processed is a resource belonging to the target resource category. Based on the resource description information, the storyboard templates for at least some shots in the target video frame template are adjusted to obtain the corresponding target storyboards; Based on the prompts containing the respective target storyboards, the target video corresponding to the resource to be processed is generated.

8. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the video processing method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the video processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the video processing method according to any one of claims 1 to 7.

Images