A high photosynthetic yield generation method and system

Abstract

The present application relates to the technical field of image processing, and more particularly to a highlight collection generation method and system, wherein the method first acquires video materials, then inputs the video materials into a multi-modal model after segment division to extract highlights, obtains multiple highlight candidate segments, then performs timestamp verification on the multiple highlight candidate segments based on preset space-time constraints, then performs forced alignment of visual content and text summary semantics on the highlight candidate segments that pass the timestamp verification, and finally merges the highlight candidate segments that pass the forced alignment in chronological order to output a highlight collection. Compared with the prior art, the method of the present application overcomes the technical problems of fragmented integration of video logic and inaccurate grasp of core information in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method and system for generating a highlight collection. Background Art

[0002] In the digital marketing era, advertising editing technology has become a core means of enhancing communication effectiveness. With the explosive growth of short video platforms and social media, advertising content must capture audience attention in a fraction of the time. Editing technology significantly improves conversion rates and retention by optimizing pacing, narrative logic, and audiovisual impact. Current technology has evolved from traditional linear editing to intelligent editing systems that integrate AI, multimodal analysis, and automated tools.

[0003] Existing intelligent editing technologies are mainly implemented by the following solutions:

[0004] 1. Keyframe detection-based editing: By analyzing the saliency of video frames (such as faces, motion, and color contrast), keyframes are extracted as candidate segments. Models such as YOLO and Faster R-CNN are used to identify high-attention areas (such as celebrity close-ups and goal moments).

[0005] 2. Action / Motion Analysis-Based Editing: This approach uses optical flow analysis, pose estimation, and motion trajectory detection to identify high-dynamic clips (e.g., sports goals and dance climaxes). It also detects the intensity of movement and selects high-energy clips (e.g., basketball dunks and car race overtaking).

[0006] 3. Editing solution based on emotion / audio analysis: Combine audio waveforms (cheers, background music climaxes) and audience reactions (bullet comments, expression recognition) to extract emotionally charged clips.

[0007] 4. Intelligent editing solution based on multimodal fusion: Combines multimodal information such as vision, audio, and text (subtitles / bullet comments) to comprehensively judge highlight clips.

[0008] However, due to the limitations of existing model kernels or constraints, the above solution has the following technical defects:

[0009] First, fragmented clip logic makes it difficult to maintain a consistent narrative flow in short videos. In sports or variety show mashups, existing technology only extracts individual highlights (e.g., shots, laughs), but ignores the causal relationships between events (e.g., the tactical coordination leading up to a goal). This creates a chaotic rhythm, making it difficult for viewers to understand the main narrative thread, leading to lower completion rates.

[0010] Second, character relationship recognition in multi-player scenes fails, causing mashups to deviate from the core content. For short videos featuring multiple people, such as variety shows and vlogs, existing models fail to distinguish the interaction weights between the main and supporting characters (e.g., incorrectly selecting close-ups of extras). For example, in a travel mashup, frequent cuts to secondary characters weaken the core character's experience and storyline, reducing user resonance.

[0011] In summary, the existing intelligent editing technology has the problems of logical fragmentation before and after the integration of videos and inaccurate grasp of core information. Summary of the Invention

[0012] In order to solve the technical problems of the existing technology of video integration logic fragmentation and inaccurate grasp of core information, the present invention discloses a highlight collection generation method and system.

[0013] In a first aspect, the present invention discloses a method for generating a highlight collection, comprising:

[0014] Get video material;

[0015] The video material is segmented and then input into the multimodal model for highlight extraction to obtain multiple highlight candidate segments;

[0016] Based on the preset spatiotemporal constraints, multiple highlight candidate segments are timestamp-checked;

[0017] Force alignment of the visual content of the highlight candidate segments that have passed timestamp verification with the semantics of the text summary;

[0018] The highlight candidate segments that have passed forced alignment are merged in chronological order and a highlight collection is output.

[0019] Beneficial effects: The method of the present invention obtains highlight candidate segments through multimodal model extraction. This method can combine multimodal information such as vision, audio and text for fusion analysis, so as to accurately grasp the core information. By pre-configuring the spatiotemporal constraints of the multimodal model, the highlight candidate segments can be automatically timestamp-checked. The entire process does not require manual operation, which improves the verification efficiency. On this basis, after the highlight candidate segments pass the verification, the method of the present invention also performs forced alignment of the visual content and text summary semantics of the highlight candidate segments to overcome the problem of logical separation before and after the integrated video.

[0020] Preferably, the spatiotemporal constraints include:

[0021] Check whether the duration of the current highlight candidate segment is greater than or equal to the preset minimum time interval;

[0022] Check whether the overlap between the current highlight candidate segment and the next highlight candidate segment is less than or equal to the overlap threshold;

[0023] Check whether the saliency of the current highlight candidate segment is greater than or equal to the saliency threshold;

[0024] If all the above conditions are met, the current highlight candidate segment is determined to have passed the timestamp verification; if any of the above conditions is not met, the current highlight candidate segment is determined to have failed the timestamp verification.

[0025] Beneficial effect: When performing timestamp verification on highlight candidate segments, by verifying the overlap between the current highlight candidate segment and the next highlight candidate segment, the overlap between temporally adjacent highlight candidate segments can be reasonably regulated, reducing the probability of the overlap being too high or too low. Among them, too high an overlap will lead to information redundancy, while too low an overlap will lead to incoherence of the previous and subsequent information, exacerbating the logical separation before and after the video. The above method can overcome both the problem of information redundancy and the problem of logical separation. In addition, the above-mentioned spatiotemporal constraints introduce a judgment of significance, which can more quickly screen out core information, thereby further overcoming the problem of inaccurate grasp of core information in the prior art.

[0026] Furthermore, the expression of the space-time constraint is:

[0027]

[0028] Where, Indicates dependence on the start time and end time Timestamp verification condition, Indicates the minimum time interval that meets the timestamp verification conditions. represents the overlap operation function, Indicates calculation of the current highlight candidate fragment and the next highlight candidate segment The degree of overlap, represents the overlap threshold, Indicates calculation of the current highlight candidate fragment The significance of Represents the preset training coefficient, Indicates calculation of video material The maximum saliency value of all highlight candidate segments in .

[0029] Preferably, if all highlight candidate segments fail to pass the timestamp verification or the number of highlight candidate segments that pass the timestamp verification does not reach the expected number, the preset backtracking mechanism is triggered, the multimodal model is called, and the highlight candidate segments are re-extracted.

[0030] Preferably, after outputting the highlight collection, the method of the present invention further comprises:

[0031] Score the merge quality of the highlight collection to obtain an evaluation value;

[0032] If the evaluation value is greater than or equal to the expected score, the highlight collection will be used as the final integrated video;

[0033] If the evaluation value is less than the expected score, return to the highlight extraction step.

[0034] Beneficial effect: After obtaining the highlight collection, the method of the present invention will also evaluate the merging quality of the highlight collection. If the merging quality of the highlight collection does not meet expectations, it means that the information extracted from the highlights is not important enough. The method returns to the highlight extraction step and performs iterative extraction again, thereby further solving the problem of inaccurate grasp of core information in the existing technology.

[0035] Preferably, the merge quality of the highlight collection is scored, and the expression for obtaining the evaluation value is:

[0036]

[0037] Where, Indicates the evaluation value, represents the MLLM (Multimodal Large Language) model, Indicates a collection of highlights. Indicates expected information for quality assessment.

[0038] Preferably, the quality assessment expected information includes rhythm expected parameters, plot logic expected description and attractiveness expected description.

[0039] Preferably, forcibly aligning the visual content of the highlight candidate segments that pass the timestamp verification with the semantics of the text summary includes:

[0040] The preset CLIP (Contrastive Language–Image Pre-training) model is used to enforce alignment of the visual content and text summary semantics of highlight candidate segments that have passed timestamp verification.

[0041] Preferably, the CLIP model is provided with a coherence loss function for guiding model training. The coherence loss function is specifically:

[0042]

[0043] Where, represents the alignment loss value, Indicates from arrive The summation symbol, Indicates the total number of segmented segments, Indicates the CLIP model for A calculation function that forces alignment of visual content in fragments. Indicates the CLIP model for the The semantics of text summarization in fragments Calculation function for forced alignment, represents the L2 norm.

[0044] Beneficial effects: The method of the present invention designs a set of coherence loss functions to guide the model to adjust parameters so that the difference between video features and text features is gradually reduced.

[0045] In a second aspect, the present invention further discloses a highlight collection generation system, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the highlight collection generation method described in the first aspect is implemented.

[0046] The beneficial effects of the present invention are:

[0047] 1. Compared to existing technologies, the method of the present invention extracts highlight candidate segments through a multimodal model. This approach can integrate multimodal information such as visual, audio, and text for fusion analysis, thereby accurately grasping the core information. After the highlight candidate segments pass verification, the method of the present invention also performs forced alignment of the visual content and text summary semantics of the highlight candidate segments to overcome the problem of logical disconnection before and after the integrated video.

[0048] 2. Compared with the existing technology, the multimodal model of the method of the present invention can automatically generate spatiotemporal constraints according to the timestamp verification rules, and reasonably regulate the overlap between temporally adjacent highlight candidate segments, which not only solves the problem of information redundancy but also further solves the problem of logical fragmentation.

[0049] 3. Compared with the existing technology, the method of the present invention designs a set of coherence loss functions to guide the model to adjust parameters so that the difference between video features and text features gradually decreases. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an illustrative and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0051] Figure 1 This is a flow chart of the method for generating a highlight collection in Example 1 of the method of the present invention;

[0052] Figure 2 It is a structural diagram of the highlight collection generation system in Example 3 of the method of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0054] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment discloses a method for generating a highlight collection, including:

[0057] S10: Obtain video material.

[0058] In this embodiment, the video material can be any of a sports video, a TV series, an advertisement video, a public welfare promotion video, or an online recorded video. The video material can be imported into the computer system corresponding to the method of the present invention by a professional, or the computer system can automatically obtain it according to a preset action response time.

[0059] S20: Segment the video material into segments and input the segments into a multimodal model for highlight extraction to obtain multiple highlight candidate segments.

[0060] In this embodiment, during the segmentation process, two adjacent segments in the time series may have some overlapping content. The multimodal model can adopt the VideoLLM multimodal model, which is a high-precision extraction model that can extract visual features, audio source features, and text features, and fuse the above features.

[0061] Specifically, the multimodal fusion features of the segmented segments are extracted through the VideoLLM multimodal model, and the multimodal fusion features are analyzed and calculated. Then, the segmented segments corresponding to the multimodal fusion features calculated through analysis are used as highlight candidate segments.

[0062] S30: Based on the preset spatiotemporal constraints, timestamp verification is performed on the multiple highlight candidate segments.

[0063] S40: Forcibly align the visual content of the highlight candidate segments that have passed the timestamp verification with the semantics of the text summary.

[0064] In this embodiment, the model involved in the forced alignment may adopt the CLIP model.

[0065] It needs to be explained that the CLIP (Contrastive Language-Image Pre-training) model is a contrastive language-image pre-training model developed by OpenAI. It aims to learn the semantic association between images and texts by pre-training on a large number of image-text pairs, so that the model can understand the semantic content of images and texts, and realize the mutual matching and retrieval between images and texts.

[0066] S50: Perform segment merging on the high-light candidate segments in chronological order by forced alignment, and output a high-light collection.

[0067] Through the above technical solution, the method of the present application can combine visual, audio and text and other multi-modal information for fusion analysis, and accurately grasp the core information. After the high-light candidate segment is verified, the method of the present application can also perform forced alignment of the visual content and the semantic of the text summary of the high-light candidate segment through step S40, so as to overcome the problem of logical fragmentation before and after the integrated video.

[0068] Compared with the prior art, the method of the present application grasps the core information more accurately, and the generated high-light collection is logically coherent in time, improving the viewing experience of the audience.

[0069] Embodiment Two

[0070] Based on embodiment one, the present embodiment mainly further explains the method of the present application from the aspect of core algorithm.

[0071] For example, in step S20, the algorithm expression of the above-mentioned VideoLLM multi-modal model for obtaining fusion features is:

[0072]

[0073] In the formula, represents the th multi-modal fusion feature of the video material, represents a multi-modal large model, represents the th segmented segment of the video material, represents a cross-modal attention fusion operator, represents the visual modal feature of the th segmented segment of the video material, represents the audio modal feature of the th segmented segment of the video material, represents the text or subtitle modal feature of the th segmented segment of the video material.

[0074] Exemplarily, in the real-time analysis and processing of the multi-modal fusion feature, historical segment features and current segment features need to be extracted from the multi-modal fusion feature, and the current processing segment is scored, and the specific algorithm expression is as follows:

[0075]

[0076] In the formula, denotes a score weight matrix; denotes historical segment features; denotes current segment features; denotes a query vector, which is generated based on information at a current time step , and is used for querying in the historical and current segment features; denotes a splicing operation of the historical segment features and the current segment features, forming a more comprehensive feature representation; denotes the latitude of the features. When the weight score of the segmented segment reaches the expected value, it is indicated that the segmented segment belongs to the core information, and the segmented segment is taken as a highlight candidate segment.

[0077] Further, the space-time constraint of the method step S20 is specifically as follows:

[0078] Condition one: verifying whether the length of the current highlight candidate segment is greater than or equal to a preset minimum time interval;

[0079] Condition two: verifying whether the overlap degree between the current highlight candidate segment and the next highlight candidate segment is less than or equal to an overlap degree threshold.

[0080] Condition three: verifying whether the saliency of the current highlight candidate segment is greater than or equal to a saliency threshold.

[0081] If the above conditions one to three are all met, the current highlight candidate segment is determined to pass the time stamp verification. If any of the above conditions is not met, the current highlight candidate segment is determined to fail the time stamp verification.

[0082] Specifically, the specific expression of the above space-time constraint is as follows:

[0083]

[0084] In the formula, denotes a time stamp verification condition depending on a start time and an end time , denotes a minimum time interval meeting the time stamp verification condition, denotes an overlap degree operation function, denotes a calculation of the current highlight candidate segment and the next highlight candidate segment The degree of overlap, represents the overlap threshold, Indicates calculation of the current highlight candidate fragment The significance of Represents the preset training coefficient, Indicates calculation of video material The maximum saliency value of all highlight candidate segments in .

[0085] Through the design of the above-mentioned spatiotemporal constraints, condition two can reasonably regulate the overlap between temporally adjacent highlight candidate segments by verifying the overlap between the current highlight candidate segment and the next highlight candidate segment, thereby reducing the probability of too high or too low overlap. More specifically, too high an overlap will lead to information redundancy and waste of resources, thereby reducing data processing efficiency. Too low an overlap will lead to incoherence of previous and subsequent information, thereby exacerbating the logical separation before and after the video. Verification through the above-mentioned condition two can overcome both the problem of information redundancy and the problem of logical separation. In addition, condition three in the above-mentioned spatiotemporal constraints also introduces a judgment of significance, which can quickly screen out core information, thereby further overcoming the problem of inaccurate grasp of core information in the existing technology.

[0086] Furthermore, if all highlight candidate segments fail to pass the timestamp verification or the number of highlight candidate segments that pass the timestamp verification does not reach the expected number, it means that the coherence of the context logic of the highlight candidate segments extracted by the multimodal model does not meet the requirements. At this time, it is necessary to trigger the backtracking mechanism, return to step S20, re-call the multimodal model, and re-extract the highlight candidate segments.

[0087] Furthermore, the model for forcibly aligning the visual content of the highlight candidate segments with the semantics of the text summary in step S40 above adopts the CLIP model. For this model, this embodiment designs a more targeted coherence loss function, specifically:

[0088]

[0089] Where, represents the alignment loss value, Indicates from arrive The summation symbol, Indicates the total number of segmented segments, Indicates the CLIP model for the A calculation function that forces alignment of visual content in fragments. Indicates the CLIP model for the The semantics of text summarization in fragments Calculation function for forced alignment, represents the L2 norm.

[0090] It should be explained that the above coherence loss function can quantify the semantic difference between the video feature and the text feature. The greater the alignment loss value is, the greater the semantic difference between the two is, and the lower the alignment degree is. The smaller the alignment loss value is, the smaller the semantic difference between the two is, and the higher the alignment degree is. The CLIP model takes the alignment loss value as a basis for judgment to check the semantic difference between the video feature and the text feature. If the alignment loss value is greater than or equal to the alignment expected value, it means that the forced alignment is successful, and the highlight candidate segment that is successfully aligned enters the data processing process of step S50. If the alignment loss value is less than the alignment expected value, it means that the forced alignment fails, the hyperparameters in the CLIP model are adjusted again and the forced alignment is performed again. If the forced alignment fails for multiple times, the highlight candidate segment is discarded.

[0091] Compared with the prior art, the above coherence loss function can guide the CLIP model to adjust the parameters, so that the difference between the video feature and the text feature gradually decreases in the process of multiple iteration training.

[0092] Further, after step S50, the method of the present application further comprises:

[0093] S60: score the merging quality of the highlight set to obtain an evaluation value.

[0094] S70: if the evaluation value is greater than or equal to an expected score, the highlight set is taken as the final integrated video.

[0095] S80: if the evaluation value is less than the expected score, return to step S20.

[0096] Specifically, the calculation expression of the above step S60 is:

[0097]

[0098] In the formula, represents the evaluation value, represents the MLLM (multi-modal large language) model, represents the highlight set, represents the quality evaluation expected information.

[0099] In the present embodiment, the quality evaluation expected information includes a rhythm expected parameter, a plot logic expected description and an attraction expected description.

[0100] It should be noted that in this embodiment, the expected quality assessment information can be input by the user or retrieved from a pre-set database. Expected rhythm parameters include specific parameters such as video tempo parameters, video multiples, and the duration of the plot's climax. Expected plot logic descriptions are relatively vague and can include phrases such as "first deprecation, then exaltation," "first division, then generalization," "first generalization, then division," "expanding the description with a single character (single-portrait plot)" or "depicting multiple characters separately, then presenting them together in a cohesive plot segment (group plot)," "flashback plot," "intercalation plot," or "sequential plot," among other related or similar descriptions. Expected attraction descriptions are also relatively vague and can include phrases such as "focusing on popular celebrities, internet celebrities, or influencers," "focusing on current hot topics, memes, or popular topics," or "focusing on currently popular electronics, automotive, or sports products." These vague language descriptions will be parsed through a language understanding model, and then keywords or sentences will be extracted. Then, network information will be traversed and searched on the online platform to calculate the traffic level of the keywords or sentences. The traffic level will be quantitatively integrated with the rhythm expectation parameters as expected information for quality assessment, thereby achieving multi-dimensional scoring of highlight collections.

[0101] Through the above steps S60 to S80, after obtaining the highlight collection, the method of the present invention will also evaluate the merging quality of the highlight collection. If the merging quality of the highlight collection does not meet expectations, it means that the information extracted from the highlights is not important enough. Return to step S20 and iterate again to extract the highlight candidate segments that meet the expectations, thereby further improving the accuracy of the method of the present invention in grasping the core information.

[0102] Example 3

[0103] like Figure 2 As shown, this embodiment discloses a highlight collection generation system, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the highlight collection generation method described in Example 1 or Example 2 is implemented.

[0104] While several embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous modifications, variations, and alternatives will occur to those skilled in the art without departing from the concept and spirit of the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed in practicing the present invention.

Claims

1. A method for generating a highlight collection, characterized in that: include: Get video material; Segmenting the video material into segments and then inputting the segments into a multimodal model for highlight extraction to obtain a plurality of highlight candidate segments; Based on the preset spatiotemporal constraints, multiple highlight candidate segments are timestamp-checked; Force alignment of the visual content of the highlight candidate segments that have passed timestamp verification with the semantics of the text summary; Merge the highlight candidate segments that have passed forced alignment in chronological order and output a highlight collection; The spatiotemporal constraints are specifically: Condition 1: Check whether the duration of the current highlight candidate segment is greater than or equal to the preset minimum time interval; Condition 2: Check whether the overlap between the current highlight candidate segment and the next highlight candidate segment is less than or equal to the overlap threshold; Condition 3: Check whether the saliency of the current highlight candidate segment is greater than or equal to the saliency threshold; If the above conditions 1 to 3 are all met, the current highlight candidate segment is determined to have passed the timestamp verification; if any of the above conditions is not met, the current highlight candidate segment is determined to have failed the timestamp verification. The specific expression of the above spatiotemporal constraints is: Where, Indicates dependence on the start time and end time Timestamp verification condition, Indicates the minimum time interval that meets the timestamp verification conditions. represents the overlap operation function, Indicates calculation of the current highlight candidate fragment and the next highlight candidate segment The degree of overlap, represents the overlap threshold, Indicates calculation of the current highlight candidate fragment The significance of Represents the preset training coefficient, Indicates calculation of video material The maximum saliency value of all highlight candidate segments in ; Condition 2 can reasonably regulate the overlap between temporally adjacent highlight candidate segments by checking the overlap between the current highlight candidate segment and the next highlight candidate segment, thus reducing the probability of the overlap being too high or too low. The mandatory alignment of the visual content of the highlight candidate segments verified by timestamps with the semantics of the text summary includes: Using the preset CLIP model, the visual content of the highlight candidate segments that have passed the timestamp verification is forced to align with the semantics of the text summary; The CLIP model is provided with a coherence loss function for guiding model training. The coherence loss function is specifically: Where, represents the alignment loss value, Indicates from arrive The summation symbol, Indicates the total number of segmented segments, Indicates the CLIP model for the A calculation function that forces alignment of visual content in fragments. Indicates the CLIP model for The semantics of text summarization in fragments Calculation function for forced alignment, represents the L2 norm.

2. The method for generating a highlight collection according to claim 1, wherein: If all highlight candidate segments fail to pass the timestamp verification or the number of highlight candidate segments that pass the timestamp verification does not reach the expected number, the preset backtracking mechanism is triggered, the multimodal model is called, and the highlight candidate segments are re-extracted.

3. The method for generating a highlight collection according to claim 1, wherein: After outputting the highlight collection, the method further includes: Scoring the merging quality of the highlight collection to obtain an evaluation value; If the evaluation value is greater than or equal to the expected score, the highlight collection is used as the final integrated video; If the evaluation value is less than the expected score, return to the highlight extraction step.

4. The method for generating a highlight collection according to claim 3, wherein: The merging quality of the highlight collection is scored, and the expression for obtaining the evaluation value is: Where, Indicates the evaluation value, represents the MLLM model, Indicates a collection of highlights. Indicates expected information for quality assessment.

5. The method for generating a highlight collection according to claim 4, wherein: The quality assessment expected information includes rhythm expected parameters, plot logic expected description and attractiveness expected description.

6. A highlight collection generation system, characterized in that: It includes a processor and a memory, the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the highlight collection generation method described in any one of claims 1 to 5 is implemented.

Images