A method for intelligent extraction and automatic image matching of multimodal content for news media

Abstract

This invention discloses a method for intelligent extraction and automatic image matching of multimodal content for news media. Belonging to the field of image data processing technology, the method includes the following steps: segmenting a video into basic segments; extracting four types of modal feature vectors from each basic segment and fusing them into multimodal features; dividing the video into several semantic units based on comprehensive consistency scoring and insertion tolerance judgment; generating content cards and candidate libraries for each semantic unit; calculating the matching score between the semantic unit and each image in its candidate library; and obtaining the optimal reordering weight through editing feedback and ranking learning, which is used for image matching of the video to be processed. This invention introduces multimodal fusion modeling and cross-modal retrieval optimization methods to improve the completeness and accuracy of semantic expression; enhances the stability and continuity of news content structure through semantic unit construction and insertion tolerance mechanisms; and significantly improves the semantic matching degree and visual harmony of the image matching results through the combination of cross-modal recall and multi-constraint fine ranking.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method for intelligent extraction and automatic image matching of multimodal content for news media. Background Technology

[0002] With the rapid development of multimodal content understanding and cross-modal retrieval technologies, intelligent image matching methods based on multimodal fusion and semantic unit modeling have shown great potential in fields such as news media, video content production, and intelligent editing. In the news media field, ensuring semantic consistency and visual plausibility between news content and accompanying images is crucial for automated editing and content distribution. Practical applications often require simultaneous fulfillment of multiple constraints, including accurate semantic matching, consistency of persons, reasonable scene setting, and stylistic harmony, to meet the demands of high-quality news production. However, existing technologies generally suffer from problems such as reliance on single-modal retrieval, insufficient cross-modal semantic alignment capabilities, difficulty in simultaneously ensuring consistency of persons, reasonable scene setting, and stylistic harmony, and poor stability and controllability of image matching results.

[0003] For example, the invention patent with application number 202210124470.3 focuses on solving the problem of cross-modal image and text retrieval in a unified public space. Essentially, it optimizes the retrieval model, but lacks features such as segmentation, semantic merging, structured card generation, and editorial feedback loops specific to news operations. Therefore, it is not applicable to automatic image matching in the news media field. Another example is the invention patent with application number 202110118655.9, which emphasizes the combination of early and late-stage fusion to improve image and text retrieval performance. However, it still primarily focuses on image-text alignment and does not specifically address the temporal continuity of news content or interstitial scenarios.

[0004] Definitions: ASR (Automatic Speech Recognition): Converts speech signals into text, suitable for scenarios such as meeting recording, video subtitles, and customer service dialogues.

[0005] OCR (Optical Character Recognition): Extracts text from images, scanned documents, and screenshots, supporting the recognition of ID cards, invoices, PPT presentations, handwritten notes, etc.

[0006] Entity extraction, also known as Named Entities Recognition (NER), aims to identify and extract predefined categories of entities from unstructured text, such as names of people, places, organizations, and time expressions. Summary of the Invention

[0007] The purpose of this invention is to provide a solution to the above-mentioned problems. By introducing a cross-modal alignment model and a semantic unit construction mechanism, it can not only significantly improve the accuracy and efficiency of content understanding and image matching, but also ensure semantic consistency and visual style coordination, thereby promoting the development of intelligent news production and automated editing technology. This invention is a method for intelligent extraction and automatic image matching of multimodal content for news media.

[0008] To achieve the above objectives, the technical solution adopted by this invention is as follows: a method for intelligent extraction and automatic image matching of multimodal content for news media, comprising the following steps: S1, divide the video into N basic segments based on an adaptive thresholding method; S2, for each basic segment, extract feature vectors from four modalities and fuse them into multimodal features using a lightweight gating network. The four modalities are visual (v), textual (t), person (p), and style (c). The i-th basic segment u i The modal m eigenvector is For m∈{v,t,p,c}, the multimodal feature is z. i ; S3, based on the comprehensive consistency score and the insertion tolerance judgment, divides the video into several semantic units according to the basic segments. The comprehensive consistency score and the insertion tolerance judgment are constructed based on the modal m feature vector and multimodal features, respectively. S4 uses a large language model to generate content cards for each semantic unit, including a summary, a set of keywords, and a visual requirement description, and then combines the three into a text description. S5, obtain a resource library containing source images. Based on the text description, select source images as candidate libraries for each semantic unit. The candidate library for semantic unit k is D. k ; S6, calculate the matching score between the semantic unit and each image in its candidate library, where the matching score S between semantic unit k and image j is... kj The calculation methods include S61~S62; S61, obtain the feature vectors of all basic segments in semantic unit k in mode m and calculate the mean, to obtain the mean vector of mode m. Extract D k The modal eigenvector of the inner material image j is ; S62, for D k For the inner material image j, S is calculated according to the following formula. kj ; , In the formula, μ (m) To match the weights of modality m in the score, sim(∙,∙) is the cosine similarity, x kjFor the negative constraint score, μ x For x kj The weights; S7, optimizes the rearrangement weight μ=(μ) through edit feedback and sorting learning. (v) ,μ (t) ,μ (p) ,μ (c) ,μ x ), thus obtaining the optimal rearrangement weight μ*; S8: Obtain the video to be processed, process it into semantic units and generate a candidate library for each semantic unit. For each semantic unit, calculate the matching score of the material images in the candidate library based on μ*, and use the material image with the highest matching score as the matching image for that semantic unit.

[0009] Preferably, in S1, the adaptive threshold method includes S11 to S13; S11, split the video into single-frame images, calculate the inter-frame difference between two adjacent frames, where the inter-frame difference between the nth frame and the (n+1)th frame is b. n b n =1-sim(v n ,v n+1 ), v n v n+1 These are the visual features of the nth frame and the (n+1)th frame, respectively; S12, Calculate the mean difference μ among all frames. b and standard deviation σ b Calculate the adaptive segmentation threshold θ b θ b =μ b +σ b ; S13, compare each b in turn. n If b n >θ b Then, the data is segmented between frame n and frame n+1 to generate a basic segment.

[0010] As a preferred option, in S2, the i-th basic segment u i Style feature vectors The extraction method is: from u i Select K keyframes and extract the color distribution feature h of the k-th keyframe. i,k Brightness characteristics i,k Temperature index w i,k and texture complexity feature t i,k Generate style vector c i,k c i,k =[h i,k ,l i,k ,w i,k ,t i,kThen, the K style vectors are averaged and aggregated to obtain... c i,k Chinese w i,k We obtain the following formula: , In the formula, , The percentages of pixels in the warm color range and the percentages of pixels in the cool color range in the HSV color space for the k-th keyframe are respectively. The hue angles corresponding to the warm color range are 0°~60° and 300°~360°, and the hue angles corresponding to the cool color range are 180°~240°. ε is a very small constant to prevent the denominator from being zero.

[0011] As a preferred option, in S2, the i-th basic fragment u is generated. i Multimodal features z i Includes steps S21 to S23; S21, extract u i Visual feature vectors Text feature vectors Character feature vectors Style feature vector Duration Δt i and subtitle density η i , constitute u i The gate vector g i ; , In the formula, It is a 2-norm; S22, g i Input a lightweight gated network, output the importance score of mode m, and then normalize it to obtain the weights of mode m. ; S23, generate multimodal feature z according to the following formula. i ; , In the formula, W (m) Let be the linear mapping matrix of mode m.

[0012] As a preferred option, in S3, the (i+1)th basic segment u i+1 The merging method includes steps S31 to S34; S31, Preset merge threshold θ g Interruption tolerance threshold θ r Interruption tolerance length W; S32, calculate u i Overall consistency score i ; , In the formula, For the (i+1)th basic segment u i+1 The modal eigenvectors m, λ m The weights of the terms being multiplied; S33, if s i ≥θ g Then u i+1 Merge to u i The semantic unit in which it is located; If s i <θ g Define the central representation of semantic units To obtain u, we take the mean of the multimodal features of all the basic segments. i Multimodal features of the last W basic segments z i+1 ~z i+W For the δ-th multimodal feature z i+δ ,calculate With z i+δ cosine similarity ρ i+δ , 1≤δ≤W; S34, sequentially check if ρ exists. i+δ ≥θ r ; If so, then u i+1 ~u i+δ Merge to u i The semantic unit in which it is located; If not, then the video will be in u i and u i+1 Interval truncation.

[0013] Preferably, in S4, the content card is generated by a large language model guided by structured prompts in a fixed JSON format.

[0014] As a preferred option, in S5, the candidate library D for generating semantic unit k is... k The methods include S51~S53; S51, use the CLIP model image encoder to generate the visual vector m of the source image j. j 1≤j≤J, where J is the total number of material images in the material library; S52, the text description of semantic unit k is used to generate query vector q using the text encoder of the CLIP model. k ; S53, calculate the recall score r of semantic unit k for source image j. kj r kj =sim(q k ,m j The candidate library D is composed of the top-scoring images from the recall pool. k .

[0015] As a preferred option, in S62, the negative constraint score x kj Calculate according to the following formula: , This is the copyright risk value of image j. If there is a copyright risk, the value is 1; otherwise, it is 0. Let d be the historical rejection rate of image j. topic (k,j) represents the topic compatibility between semantic unit k and source image j, with values ​​of 0, 0.5, and 1 depending on whether their event labels are the same, of the same type, or of different types; α1, α2, and α3 are the weights of the corresponding multiplication terms.

[0016] Preferably, step S7 includes S71 to S73; S71, marked D k If a source image j can be adopted by semantic unit k, it is marked as a positive sample j. + and S kj Marked as S kj+ Otherwise, it is marked as a negative sample j. - and S kj Marked as S kj- ; S72, Construct the sorting loss L; , In the formula, Δ is the expected interval between positive and negative sample scores, and max(∙,∙) is the maximum value. S73, Construct rearrangement weights μ=(μ (v) ,μ (t) ,μ (p) ,μ (c) ,μ x The optimal rearrangement weight μ* is obtained by minimizing L and updating μ until the preset number of iterations.

[0017] Regarding S1: The video is segmented into basic segments, but these basic segments are not uniformly cut as in existing technologies. Instead, they take into account the inter-frame differences between adjacent single-frame images to generate an adaptive segmentation threshold θ. b The method involves segmentation, where the segmentation threshold adaptively adjusts to changes in the overall video level. This avoids over-segmentation or under-segmentation issues caused by a fixed threshold, improving the semantic consistency and boundary accuracy of the basic segments. Here, the basic segments are not the final business units, but rather the smallest computational units for subsequent semantic organization.

[0018] Regarding S2: Four modal feature vectors are extracted from each basic segment and fused into multimodal features. The visual feature vector represents the semantics of the scene and objects; the text feature vector represents the content of the report; the person feature vector obtains the set of people within the segment and their identities through face detection and recognition; and the style feature vector describes the overall visual quality of the image and directly affects the visual harmony of the matching image. This invention uses these four dimensions to form the basis for subsequent semantic organization and image matching. Taking the i-th basic segment u... i For example, the four types of modal features are visual feature vectors. Text feature vectors Character feature vectors Style feature vector The following methods can be used for extraction: (1) From u i Several keyframes are uniformly extracted from the image, and visual vectors are generated for each keyframe using the CLIP model image encoder. The average of the visual vectors from all keyframes is then calculated to obtain u. i Visual feature vectors ; (2) Using ASR and OCR technologies, u i Speech recognition and caption recognition are performed, text is generated, and then the text encoder of the CLIP model generates text feature vectors. ; (3) For u i Keyframes use a face detection model to identify people and extract corresponding feature vectors. ; (4) From u i K keyframes are selected, relevant features are extracted from each frame to generate a style vector, and then averaged and aggregated to obtain the style feature vector. .

[0019] Regarding S3: The distinction between different semantic units lies in the consistency differences between their multimodal semantic representations. When adjacent segments differ significantly in visual content, textual semantics, character composition, or scene style, and do not meet the insertion tolerance condition, they are determined to be different semantic units. The purpose of this step is to define two judgment rules to merge the basic segments into independent semantic units. The two judgment rules are comprehensive consistency judgment and insertion tolerance judgment. Generally, if the differences between two basic segments are not significant, they can be classified into the same semantic unit. If the differences are large, it is determined whether it is a "semantic interruption problem caused by short-term insertion." If it is a short-term insertion, the inserted content and the basic segments before and after it can be classified into the same semantic unit. Therefore, this invention first uses comprehensive consistency for a judgment and then uses insertion tolerance judgment for a second judgment.

[0020] Regarding overall consistency: (using u)i Overall consistency score i For example, When m = v, t, p, c, respectively characterize u i and u i+1 Consistency across modalities: visual, textual, persona, and style. i This represents the overall consistency score of the four modalities; θ g Determine u i and u i+1 Does it meet the consistency criteria? If s i ≥θ g Then it satisfies the condition that u i+1 Merge into u i The semantic unit in question is then processed, and the next basic judgment u is processed. i+2 If s i <θ g This indicates u i and u i+1 Inconsistency / failure to meet consistency criteria requires entering the insertion tolerance judgment.

[0021] Regarding the insertion tolerance determination: A maximum of W basic segments will be judged subsequently. W is set based on the length of the inserted content, generally W≤3. In this invention, the average multimodal features of all basic segments already merged into the semantic unit are used... This indicates that the multimodal features of the subsequent W basic segments are z i+1 ~z i+W Calculate separately With z i+1 ~z i+W cosine similarity ρ i+1 ~ρ i+W and in turn with θ r In comparison, among them, The multimodal features z of the subsequent δ-th basic fragment i+δ The cosine similarity is ρ i+δ , 1≤δ≤W; Compare ρ in turn i+δ With θ r Size, when ρ appears i+δ ≥θ r When, it means u i with u i+δ The basic segment between them is an interstitial segment, which will be u i+1 ~u i+δ All merged into u i The semantic unit in which it is located, and the next basic segment u is analyzed. i+δ+1 When W ρ i+δ <θ r This indicates that it's not an inserted segment, but rather a problem with the underlying segments that differ too much and need to be split. Therefore, the video will be divided into segments in u...i and u i+1 Interval truncation, to u i+1 As the starting point for the next semantic unit.

[0022] Regarding S4, the purpose is to use a large model to generate content cards according to a fixed format, which should include at least three fields: summary, keyword set, and visual requirement description. These are used to concatenate the text descriptions of semantic units and assist in generating a candidate library of semantic units in the subsequent step S5. Additionally, two fields, event tag and person tag, can be added as needed to perform factual backtracking and verification on the content cards, ensuring the high requirement for factual accuracy in news scenarios.

[0023] Regarding S5, it is actually based on a semantic similarity-based recall method. Multiple images are selected from the resource library for each semantic unit, forming a resource library that corresponds one-to-one with each semantic unit. The calculation of semantic similarity relies on the cross-modal CLIP model, which includes an image encoder and a text encoder. Taking semantic unit k as an example, the text encoder of the CLIP model edits its corresponding text description into a query vector q. k All images in the resource library are encoded into visual vectors using the CLIP model image encoder; the query vector and the visual vector can be directly used to calculate the cosine similarity, and q k The visual vector m of the source image j j The cosine similarity is used as the semantic unit k to score the recall of the source image j. kj Then, the N1 images with the highest recall scores are used to form the candidate library D. k This method enables rapid filtering of the candidate library.

[0024] Regarding S6, its purpose is to calculate a matching score for the candidate image library for each semantic unit based on various constraints, facilitating subsequent descending sorting and image recommendation for the semantic unit. The matching score is based on the following formula: , The formula contains 5 weights: μ (v) μ (t) μ (p) μ (c) μ x This facilitates the subsequent construction of rearranged weights μ=(μ (v) ,μ (t) ,μ (p) ,μ (c) ,μ x The subsequent training process also involves iteratively updating μ.

[0025] Regarding S7, the purpose is to manually annotate D. kThe internal source images are used to label positive and negative samples, thereby calculating the ranking loss L and updating the rearrangement weights μ. This ranking loss L requires that the score of the accepted source material in the same news article should be significantly higher than that of the rejected source material; otherwise, the system will be penalized and the parameters will be adjusted. During the update, μ is updated by minimizing L, in the following way: In the formula, η is the learning rate, ∇L is the gradient of the ranking loss L with respect to μ, and ← represents the parameter update operation. In implementation, an "online recording, offline updating" mechanism can be adopted, that is, accumulating editing behavior logs in real time, and then updating μ and the quality score of the source images within a fixed period to avoid excessive disturbance to the system caused by short-term noise. Furthermore, the μ update here only applies to the re-ranking stage and can be updated based on editing feedback to achieve adaptive optimization of the image matching strategy. After training, the optimal re-ranking weight μ* is obtained.

[0026] S8 processes the video to be processed based on μ*, thereby generating an image for each semantic unit of the video.

[0027] Compared with the prior art, the advantages of the present invention are as follows: (1) This invention introduces a multimodal feature fusion and dynamic weight allocation mechanism, which can effectively improve the utilization efficiency of information from different modalities based on a multimodal representation learning model. This method not only improves the integrity of semantic expression, but also ensures the stability and consistency of multimodal information fusion, providing stronger data support for news content understanding and retrieval.

[0028] (2) Introducing semantic unit construction and insertion tolerance mechanisms as key steps in the content organization process enables a more continuous and stable semantic structure for news. This method not only focuses on overall semantic coherence but also precisely handles inserted footage and short-term semantic disturbances, significantly improving the structured expression of news content and subsequent retrieval results, effectively enhancing the stability and continuity of the news content structure. Especially in the automatic editing of news videos and the generation of accompanying images, this technology significantly improves the reliability and usability of practical applications.

[0029] (3) A cross-modal retrieval coarse recall and matching score secondary ranking mechanism is introduced to generate a candidate library for each semantic unit and recommend and rank the material images in the candidate library. The cross-modal retrieval coarse recall is based on the text description of the semantic unit and the cosine similarity of the material images in the semantic space to quickly find material images that are more relevant to the semantic unit. Then, based on multiple constraints (visual, text, character, style, negative constraints), the matching score of each material image in the candidate library is calculated to facilitate subsequent recommendation and ranking. This invention can significantly improve the semantic matching degree and visual coordination of the matching results.

[0030] In summary, this invention not only significantly improves the accuracy and efficiency of content understanding and image matching, but also ensures semantic consistency and visual style coordination. It enhances the quality and application value of intelligent news processing and automatic image matching technologies, providing an innovative solution for the field of intelligent media and content production, thereby promoting the development of intelligent news production and automated editing technologies. Attached Figure Description

[0031] Figure 1 This is a flowchart of the present invention; Figure 2 This is a flowchart for generating semantic units based on basic fragments. Detailed Implementation

[0032] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0033] Example 1: See Figure 1 and Figure 2 A method for intelligent extraction and automatic image matching of multimodal content for news media includes the following steps: S1, divide the video into N basic segments based on an adaptive thresholding method; S2, for each basic segment, extract feature vectors from four modalities and fuse them into multimodal features using a lightweight gating network. The four modalities are visual (v), textual (t), person (p), and style (c). The i-th basic segment u i The modal m eigenvector is For m∈{v,t,p,c}, the multimodal feature is z. i ; S3, based on the comprehensive consistency score and the insertion tolerance judgment, divides the video into several semantic units according to the basic segments. The comprehensive consistency score and the insertion tolerance judgment are constructed based on the modal m feature vector and multimodal features, respectively. S4 uses a large language model to generate content cards for each semantic unit, including a summary, a set of keywords, and a visual requirement description, and then combines the three into a text description. S5, obtain a resource library containing source images. Based on the text description, select source images as candidate libraries for each semantic unit. The candidate library for semantic unit k is D. k ; S6, calculate the matching score between the semantic unit and each image in its candidate library, where the matching score S between semantic unit k and image j is... kj The calculation methods include S61~S62; S61, obtain the feature vectors of all basic segments in semantic unit k in mode m and calculate the mean, to obtain the mean vector of mode m. Extract D k The modal eigenvector of the inner material image j is ; S62, for D k For the inner material image j, S is calculated according to the following formula. kj ; , In the formula, μ (m) To match the weights of modality m in the score, sim(∙,∙) is the cosine similarity, x kj For the negative constraint score, μ x For x kj The weights; S7, optimizes the rearrangement weight μ=(μ) through edit feedback and sorting learning. (v) ,μ (t) ,μ (p) ,μ (c) ,μ x ), thus obtaining the optimal rearrangement weight μ*; S8: Obtain the video to be processed, process it into semantic units and generate a candidate library for each semantic unit. For each semantic unit, calculate the matching score of the material images in the candidate library based on μ*, and use the material image with the highest matching score as the matching image for that semantic unit.

[0034] Example 2: See Figures 1-2 Based on Example 1, the adaptive threshold method in S1 includes S11 to S13; S11, split the video into single-frame images, calculate the inter-frame difference between two adjacent frames, where the inter-frame difference between the nth frame and the (n+1)th frame is b. n b n =1-sim(v n ,v n+1 ), v n v n+1 These are the visual features of the nth frame and the (n+1)th frame, respectively; S12, Calculate the mean difference μ among all frames. b and standard deviation σ b Calculate the adaptive segmentation threshold θ b θ b =μ b +σ b ; S13, compare each b in turn. n If b n >θ b Then, the data is segmented between frame n and frame n+1 to generate a basic segment.

[0035] In S2, the i-th basic segment u i Style feature vectors The extraction method is: from u iSelect K keyframes and extract the color distribution feature h of the k-th keyframe. i,k Brightness characteristics i,k Temperature index w i,k and texture complexity feature t i,k Generate style vector c i,k c i,k =[h i,k ,l i,k ,w i,k ,t i,k Then, the K style vectors are averaged and aggregated to obtain... c i,k Chinese w i,k We obtain the following formula: , In the formula, , The percentages of pixels in the warm color range and the percentages of pixels in the cool color range in the HSV color space for the k-th keyframe are respectively. The hue angles corresponding to the warm color range are 0°~60° and 300°~360°, and the hue angles corresponding to the cool color range are 180°~240°. ε is a very small constant to prevent the denominator from being zero.

[0036] In S2, the i-th basic fragment u is generated. i Multimodal features z i Includes steps S21 to S23; S21, extract u i Visual feature vectors Text feature vectors Character feature vectors Style feature vector Duration Δt i and subtitle density η i , constitute u i The gate vector g i ; , In the formula, It is a 2-norm; S22, g i Input a lightweight gated network, output the importance score of mode m, and then normalize it to obtain the weights of mode m. ; S23, generate multimodal feature z according to the following formula. i ; , In the formula, W (m) Let be the linear mapping matrix of mode m.

[0037] In S3, the (i+1)th basic segment ui+1 The merging method includes steps S31 to S34; S31, Preset merge threshold θ g Interruption tolerance threshold θ r Interruption tolerance length W; S32, calculate u i Overall consistency score i ; , In the formula, For the (i+1)th basic segment u i+1 The modal eigenvectors m, λ m The weights of the terms being multiplied; S33, if s i ≥θ g Then u i+1 Merge to u i The semantic unit in which it is located; If s i <θ g Define the central representation of semantic units To obtain u, we take the mean of the multimodal features of all the basic segments. i Multimodal features of the last W basic segments z i+1 ~z i+W For the δ-th multimodal feature z i+δ ,calculate With z i+δ cosine similarity ρ i+δ , 1≤δ≤W; S34, sequentially check if ρ exists. i+δ ≥θ r ; If so, then u i+1 ~u i+δ Merge to u i The semantic unit in which it is located; If not, then the video will be in u i and u i+1 Interval truncation.

[0038] In S4, the content cards are generated by a large language model in a fixed JSON format, guided by structured prompt words.

[0039] In S5, candidate libraries D for generating semantic unit k are generated. k The methods include S51~S53; S51, use the CLIP model image encoder to generate the visual vector m of the source image j. j 1≤j≤J, where J is the total number of material images in the material library; S52, the text description of semantic unit k is used to generate query vector q using the text encoder of the CLIP model. k ; S53, calculate the recall score r of semantic unit k for source image j. kj r kj =sim(q k ,m j The candidate library D is composed of the top-scoring images from the recall pool. k .

[0040] In S62, the negative constraint score x kj Calculate according to the following formula: , This is the copyright risk value of image j. If there is a copyright risk, the value is 1; otherwise, it is 0. Let d be the historical rejection rate of image j. topic (k,j) represents the topic compatibility between semantic unit k and source image j, with values ​​of 0, 0.5, and 1 depending on whether their event labels are the same, of the same type, or of different types; α1, α2, and α3 are the weights of the corresponding multiplication terms.

[0041] S7 includes S71~S73; S71, marked D k If a source image j can be adopted by semantic unit k, it is marked as a positive sample j. + and S kj Marked as S kj+ Otherwise, it is marked as a negative sample j. - and S kj Marked as S kj- ; S72, Construct the sorting loss L; , In the formula, Δ is the expected interval between positive and negative sample scores, and max(∙,∙) is the maximum value. S73, Construct rearrangement weights μ=(μ (v) ,μ (t) ,μ (p) ,μ (c) ,μ x The optimal rearrangement weight μ* is obtained by minimizing L and updating μ until the preset number of iterations.

[0042] Example 3: See Figures 1-2 Based on Example 1, the material images in the material library can be obtained through the following three methods: 1. Historical news material library, containing historical news photos and keyframes from historical videos, such as photos of fire scenes, meeting photos, and medical scene photos; 2. General visual resource libraries, such as publicly available news image libraries and related datasets; 3. Current video keyframes, extracted from the current news.

[0043] In addition, in step S4, the content card includes a summary, keyword set, visual requirements description, event tags, and person tags, for a total of 5 fields. Based on the event tags and person tags, factual backtesting is performed on the content card. See Table 1 below for explanations of each field of the content card: Table 1. Description of each field in the content card summary Describe the core content of the news in one sentence Keyword set Core term collection Visual requirements description Description of visual elements required for illustration Event Tags Event type classification Character Tags List of people appearing on screen All fields above are generated by the large language model. To ensure stable output of content cards, the system uses structured prompts to require the large model to output in a fixed JSON format. If the initial output is missing fields, has an incorrect field type, or cannot be parsed, a format correction and retry mechanism is triggered, retrying a maximum of two times. If the requirements are still not met, an empty field is used as a fallback. After this processing, the content cards are no longer freely generated natural language, but a standardized intermediate representation that can be parsed, indexed, and searched.

[0044] When the field contains event tags and person tags, these two types of tags can also be used to perform fact backtracking verification on the content card. Taking the fact backtracking verification of the content card corresponding to semantic unit k as an example, the specific method includes steps S41~S46. S41, Preset maximum number of verifications N max =3, let the current number of checks be N. now =1; S42, extract all entities from event tags and person tags, and convert them into entity vectors respectively; For example, the summary of this semantic unit is "A fire occurred in a certain place in 2026, and firefighters quickly arrived at the scene to put it out". The corresponding event label is "fire" and the person label is "firefighter". All extracted entities are fire and firefighter. The text encoder of the CLIP model encodes fire and firefighter into entity vectors e1 and e2 respectively. S43, based on ASR and OCR technologies, extracts text from semantic unit k, converts each text into a text vector, and forms a set T. k Suppose three texts are "A fire has occurred in a certain place", "Firefighters are putting out the fire", and "There is a lot of smoke at the scene", which can be converted into text vectors t1, t2, and t3, respectively. k ={t1,t2,t3}; S44, sequentially verify the entities "fire" and "firefighter". The verification method for "fire" is as follows: Calculate the cosine similarity between e1 and t1~t3 sequentially: sim(e1,t1)=0.92, sim(e1,t2)=0.65, sim(e1,t3)=0.58, with a maximum value of 0.92. Due to the threshold θ... v =0.8, and 0.92>0.8, then the entity "Fire" passes the verification; the verification of the entity "Firefighter" is the same. S45. If both "fire" and "firefighter" verifications pass, output the above content card; otherwise, proceed to S46. S46, N now =N now +1 and N now <N max Repeat steps S42 to S45; otherwise, directly input the verification results of each entity.

[0045] Furthermore, the purpose of the verification in this embodiment is to suppress illusions and avoid illusions generated by large models. For example, in S42, the entity "earthquake" is extracted and converted into entity vector e3. Then, according to S44, the cosine similarity between e3 and t1~t3 is calculated: sim(e3,t1)=0.12, sim(e3,t2)=0.08, sim(e3,t3)=0.10, and the maximum value is 0.12 < θ. v If the value is 0.8, then the entity "earthquake" fails the verification.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent extraction and automatic image matching of multimodal content for news media, characterized in that, Includes the following steps: S1, divide the video into N basic segments based on an adaptive thresholding method; S2, for each basic segment, extract feature vectors from four modalities and fuse them into multimodal features using a lightweight gating network. The four modalities are visual (v), textual (t), person (p), and style (c). The i-th basic segment u i The modal m eigenvector is For m∈{v,t,p,c}, the multimodal feature is z. i ; S3, based on the comprehensive consistency score and the insertion tolerance judgment, divides the video into several semantic units according to the basic segments. The comprehensive consistency score and the insertion tolerance judgment are constructed based on the modal m feature vector and multimodal features, respectively. S4 uses a large language model to generate content cards for each semantic unit, including a summary, a set of keywords, and a visual requirement description, and then combines the three into a text description. S5, obtain a resource library containing source images. Based on the text description, select source images as candidate libraries for each semantic unit. The candidate library for semantic unit k is D. k ; S6, calculate the matching score between the semantic unit and each image in its candidate library, where the matching score S between semantic unit k and image j is... kj The calculation methods include S61~S62; S61, obtain the feature vectors of all basic segments in semantic unit k in mode m and calculate the mean, to obtain the mean vector of mode m. Extract D k The modal eigenvector of the inner material image j is ; S62, for D k For the inner material image j, S is calculated according to the following formula. kj ; ， In the formula, μ (m) To match the weights of modality m in the score, sim(∙,∙) is the cosine similarity, x kj For the negative constraint score, μ x For x kj The weights; S7, optimizes the rearrangement weight μ=(μ) through edit feedback and sorting learning. (v) ,μ (t) ,μ (p) ,μ (c) ,μ x ), thus obtaining the optimal rearrangement weight μ*; S8: Obtain the video to be processed, process it into semantic units and generate a candidate library for each semantic unit. For each semantic unit, calculate the matching score of the material images in the candidate library based on μ*, and use the material image with the highest matching score as the matching image for that semantic unit. In S3, the (i+1)th basic segment u i+1 The merging method includes steps S31 to S34; S31, Preset merge threshold θ g Interruption tolerance threshold θ r Interruption tolerance length W; S32, calculate u i Overall consistency score i ; ， In the formula, For the (i+1)th basic segment u i+1 The modal m eigenvector, λ m The weights of the terms being multiplied; S33, if s i ≥θ g Then u i+1 Merge to u i The semantic unit in which it is located; If s i <θ g Define the central representation of semantic units To obtain u, we take the mean of the multimodal features of all the basic segments. i Multimodal features of the last W basic segments z i+1 ~z i+W For the δ-th multimodal feature z i+δ ,calculate With z i+δ cosine similarity ρ i+δ , 1≤δ≤W; S34, sequentially check if ρ exists. i+δ ≥θ r ; If so, then u i+1 ~u i+δ Merge to u i The semantic unit in which it is located; If not, then the video will be in u i and u i+1 Interval truncation.

2. The method for intelligent extraction and automatic image matching of multimodal content for news media according to claim 1, characterized in that, In S1, the adaptive threshold method includes S11 to S13; S11, split the video into single-frame images, calculate the inter-frame difference between two adjacent frames, where the inter-frame difference between the nth frame and the (n+1)th frame is b. n b n =1-sim(v n ,v n+1 ), v n v n+1 These are the visual features of the nth frame and the (n+1)th frame, respectively; S1 2. Calculate the mean difference μ among all frames. b and standard deviation σ b Calculate the adaptive segmentation threshold θ b θ b =μ b +σ b ; S13, compare each b in turn. n If b n >θ b Then, the data is segmented between frame n and frame n+1 to generate a basic segment.

3. The method for intelligent extraction and automatic image matching of multimodal content for news media according to claim 1, characterized in that, In S2, the i-th basic segment u i Style feature vectors The extraction method is: from u i Select K keyframes and extract the color distribution feature h of the k-th keyframe. i,k Brightness characteristics i,k Temperature index w i,k and texture complexity feature t i,k Generate style vector c i,k c i,k =[h i,k ,l i,k ,w i,k ,t i,k Then, the K style vectors are averaged and aggregated to obtain... c i,k Chinese w i,k We obtain the following formula: ， In the formula, , These represent the percentage of pixels in the warm color range and the percentage of pixels in the cool color range in the HSV color space for the k-th keyframe, respectively. The hue angles corresponding to the warm color range are 0°~60° and 300°~360°, and the hue angles corresponding to the cool color range are 180°~240°. ε is a very small constant to prevent the denominator from being zero.

4. The method for intelligent extraction and automatic image matching of multimodal content for news media according to claim 1, characterized in that, In S2, the i-th basic fragment u is generated. i Multimodal features z i Includes steps S21 to S23; S21, extract u i Visual feature vectors Text feature vectors Character feature vectors Style feature vector Duration Δt i and subtitle density η i , constitute u i The gate vector g i ; ， In the formula, It is a 2-norm; S22, g i Input a lightweight gated network, output the importance score of mode m, and then normalize it to obtain the weights of mode m. ; S23, generate multimodal feature z according to the following formula. i ; ， In the formula, W (m) Let be the linear mapping matrix of mode m.

5. The method for intelligent extraction and automatic image matching of multimodal content for news media according to claim 1, characterized in that, In S4, the content cards are generated by a large language model in a fixed JSON format, guided by structured prompt words.

6. The method for intelligent extraction and automatic image matching of multimodal content for news media according to claim 1, characterized in that, In S5, candidate libraries D for generating semantic unit k are generated. k The methods include S51~S53; S51, use the CLIP model image encoder to generate the visual vector m of the source image j. j 1≤j≤J, where J is the total number of material images in the material library; S52, the text description of semantic unit k is used to generate query vector q using the text encoder of the CLIP model. k ; S53, calculate the recall score r of semantic unit k for source image j. kj r kj =sim(q k ,m j The candidate library D is composed of the top-scoring images from the recall pool. k .

7. The method for intelligent extraction and automatic image matching of multimodal content for news media according to claim 1, characterized in that, In S62, the negative constraint score x kj Calculate according to the following formula: ， This is the copyright risk value of image j. If there is a copyright risk, the value is 1; otherwise, it is 0. Let d be the historical rejection rate of image j. topic (k,j) represents the topic compatibility between semantic unit k and source image j, with values ​​of 0, 0.5, and 1 depending on whether their event labels are the same, of the same type, or of different types; α1, α2, and α3 are the weights of the corresponding multiplication terms.

8. The method for intelligent extraction and automatic image matching of multimodal content for news media according to claim 1, characterized in that, Step S7 includes S71 to S73; S71, marked D k If a source image j can be adopted by semantic unit k, it is marked as a positive sample j. + and S kj Marked as S kj+ Otherwise, it is marked as a negative sample j. - and S kj Marked as S kj- ; S72, Construct the sorting loss L; ， In the formula, Δ is the expected interval between positive and negative sample scores, and max(∙,∙) is the maximum value. S73, Construct rearrangement weights μ=(μ (v) ,μ (t) ,μ (p) ,μ (c) ,μ x The optimal rearrangement weight μ* is obtained by minimizing L and updating μ until the preset number of iterations.

Images