Advertisement placement information processing method, related device and medium

Abstract

The present disclosure provides a method, related device and medium for processing advertisement implantation information. The method comprises: aggregating video frames belonging to the same storyboard in a target video into storyboard segments, wherein the target video is a video to be implanted with target advertisement information; aggregating storyboard segments belonging to the same plot in the target video into plot segments; inputting the plot segments into a segment representation prediction model to obtain a first representation representing the plot segments; obtaining advertisement demand information; determining a first matching degree between the advertisement demand information and the first representation of each plot segment, and determining a target segment from a plurality of plot segments based on the first matching degree; and implanting target advertisement information in the target segment. The embodiments of the present disclosure can improve the flexibility and accuracy of advertisement implantation information processing. The embodiments of the present disclosure can be applied to scenes such as film and television drama playback, video websites, video software, live advertisement insertion, and media information processing.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to an advertisement implantation information processing method, a related device and a medium. Background Art

[0002] On video websites, it is often necessary to insert advertisements. For example, in a certain video, there is a scene where the heroine applies skin care products. At this time, a floating layer appears in the picture, showing an advertisement for lotion A. In the prior art, when inserting advertisements, video features and audio features are often extracted from the target video, and deep learning models are used to perform element recognition on the video features and audio features, such as target detection, face recognition, behavior recognition, and place recognition. The identified objects, people, behaviors, places and other elements are labeled to form labels. Then, the advertising demand information is compared with the labels. If a label that matches the advertising demand information appears, the advertising information is implanted.

[0003] The disadvantages of this solution are insufficient flexibility and poor accuracy. The lack of flexibility is reflected in the difficulty in covering flexible and changeable advertising needs. Once there is an advertising demand that cannot be covered by the existing label system (for example, the existing label system has hundreds of foods, but the advertising demand is for "snacks"), it is difficult to find the location for advertising in the video. The poor accuracy is reflected in the fact that sometimes although a certain label appears in the video, the location where the label appears may not be suitable for inserting advertising information, resulting in poor advertising insertion effect. Summary of the invention

[0004] The embodiments of the present disclosure provide an advertisement placement information processing method, a related device, and a medium, which can improve the flexibility and accuracy of advertisement placement information processing.

[0005] According to one aspect of the present disclosure, a method for processing advertisement placement information is provided, comprising:

[0006] Aggregate video frames belonging to the same storyboard in a target video into storyboard segments, wherein the target video is a video to be embedded with target advertising information;

[0007] Aggregate the storyboard segments belonging to the same plot in the target video into a plot segment;

[0008] Inputting the plot segment into a segment representation prediction model to obtain a first representation representing the plot segment;

[0009] Obtain advertising demand information;

[0010] Determining a first matching degree between the advertisement demand information and the first representation of each of the plot segments, and determining a target segment from among the plurality of plot segments based on the first matching degree;

[0011] The target advertisement information is embedded in the target segment.

[0012] According to one aspect of the present disclosure, there is provided an advertisement placement information processing device, the device comprising:

[0013] A storyboard segment acquisition unit, used for aggregating video frames belonging to the same storyboard in a target video into storyboard segments, wherein the target video is a video to be embedded with target advertising information;

[0014] A plot segment acquisition unit, used for aggregating the storyboard segments belonging to the same plot in the target video into a plot segment;

[0015] a segment representation prediction unit, configured to input the plot segment into a segment representation prediction model to obtain a first representation representing the plot segment;

[0016] An advertisement demand acquisition unit, used for acquiring advertisement demand information;

[0017] a target segment matching unit, configured to determine a first matching degree between the advertisement demand information and the first representation of each of the plot segments, and determine a target segment from among the plurality of plot segments based on the first matching degree;

[0018] The target advertisement embedding unit is used to embed the target advertisement information into the target segment.

[0019] Optionally, the segment representation prediction unit is specifically used for:

[0020] Acquire text information corresponding to the plot segment from the target video;

[0021] The plot segment and the text information are input into the segment representation prediction model to obtain the first representation representing the plot segment.

[0022] Optionally, the segment representation prediction unit is specifically used for:

[0023] Using a subtitle recognition model and a speech recognition model, identifying dialogue information from the plot segment;

[0024] Using a face recognition model, identifying character information corresponding to the dialogue information from the plot segment;

[0025] The dialogue information and the character information corresponding to the dialogue information are integrated into the text information.

[0026] Optionally, the segment representation prediction unit is specifically used for:

[0027] Using the subtitle recognition model, performing subtitle recognition in the plot segment to obtain first probabilities corresponding to a plurality of candidate dialogue information respectively;

[0028] Using the speech recognition model, performing speech recognition in the plot segment to obtain second probabilities corresponding to a plurality of candidate dialogue information respectively;

[0029] The speech information is determined from the plurality of candidate speech information based on the first probability and the second probability respectively corresponding to each of the candidate speech information.

[0030] Optionally, the segment representation prediction unit is specifically used for:

[0031] Obtaining a first weight of the subtitle recognition model and a second weight of the speech recognition model;

[0032] Determining a weighted sum of the first probability and the second probability based on the first weight and the second weight;

[0033] Based on the weighted sum, the speech information is determined from among the plurality of candidate speech information.

[0034] Optionally, the segment indicates that the prediction model is a multimodal large model;

[0035] The segment representation prediction unit is specifically used for:

[0036] Combining the text information with the first prompt to obtain a first combined text;

[0037] The plot segment and the first combined text are input into the multimodal large model to obtain the first representation representing the plot segment.

[0038] Optionally, the segment representation prediction unit is specifically used for:

[0039] Extracting a plurality of first representative frames from the plot segment, and generating a first representative frame vector based on the first representative frames;

[0040] converting a plurality of said first representative frame vectors into first features suitable for inputting into said multimodal macro model;

[0041] Based on the first combined text, generating a second feature suitable for input into the multimodal large model;

[0042] The first feature and the second feature are input into the multimodal large model to obtain the first representation representing the plot segment.

[0043] Optionally, the first representation is a first text description of the plot segment;

[0044] The target segment matching unit is specifically used for:

[0045] extracting a plurality of first keywords from the advertising demand information;

[0046] For each of the first keywords, determining a first similarity between the first keyword and the first text description;

[0047] The first matching degree between the advertisement demand information and the first text description is determined based on the first similarity between each of the first keywords and the first text description.

[0048] Optionally, the target segment matching unit is specifically used to:

[0049] Segmenting the advertising demand information;

[0050] Identifying stop words in the separated words and removing the stop words;

[0051] Among the words after removing the stop words, non-entity words are removed to obtain the multiple first keywords.

[0052] Optionally, the target segment matching unit is specifically used to:

[0053] For each of the first keywords, determining a first word frequency of the first keyword appearing in the first text description;

[0054] determining a first inverse document frequency of occurrence of the first keyword in a plurality of the first text descriptions;

[0055] The first similarity is determined based on the first term frequency and the first inverse document frequency.

[0056] Optionally, the target segment matching unit is specifically used to:

[0057] For each of the first keywords, determining the number of first occurrences of the first keyword in the first text description;

[0058] extracting second keywords from the first text description, and determining a second keyword number of the second keywords;

[0059] The first word frequency is determined based on the first number of times and the second number of keywords.

[0060] Optionally, the target segment matching unit is specifically used to:

[0061] determining a first number of the first text descriptions that include the first keyword;

[0062] Get the total number of first text descriptions;

[0063] The first inverse document frequency is determined based on the first number and the first total number of text descriptions.

[0064] Optionally, the target segment matching unit is specifically used to:

[0065] extracting second keywords from each of the first text descriptions;

[0066] Taking each of the extracted second keywords as a first index, the first index and the first text description identifier of each of the first text descriptions containing the second keyword are stored in a first index comparison table in correspondence;

[0067] The first index comparison table is searched using the first keyword as the first index to obtain the first text description identifier stored corresponding to the first index, thereby determining the first number of the first text descriptions containing the first keyword.

[0068] Optionally, the first representation is a first abstract feature of the plot segment;

[0069] The target segment matching unit is specifically used for:

[0070] Inputting the advertisement demand information into the segment representation prediction model to obtain a second abstract feature of the advertisement demand information;

[0071] The first matching degree is calculated based on the second abstract feature and the first abstract feature of each of the plot segments.

[0072] Optionally, the segment representation prediction unit is specifically used for:

[0073] Inputting the storyboard segment into the segment representation prediction model to obtain a second representation representing the storyboard segment;

[0074] The target segment matching unit is specifically used for:

[0075] Determining a second matching degree between the advertisement demand information and the second representation of each of the storyboard segments;

[0076] Based on the first matching degree and the second matching degree, the target segment is determined from among the plurality of plot segments and the plurality of storyboard segments.

[0077] Optionally, the target segment matching unit is specifically used to:

[0078] Weighting the first matching degree according to a first weight value to obtain a first weighted matching degree, and weighting the second matching degree according to a second weight value to obtain a second weighted matching degree;

[0079] Determining the first weighted matching degree or the second weighted matching degree as the matching degree to be examined;

[0080] The plot segment or the storyboard segment whose matching degree to be examined satisfies the first condition is determined as the target segment.

[0081] Optionally, the target segment matching unit is specifically used to:

[0082] Determine the plot segment whose first matching degree satisfies the second condition as a target plot segment;

[0083] The storyboard segment in the target plot segment whose second matching degree satisfies the third condition is determined as the target segment.

[0084] Optionally, the segment indicates that the prediction model is a multimodal large model;

[0085] The segment representation prediction unit is specifically used for:

[0086] The storyboard segment and the second prompt are inputted into the segment representation prediction model together to obtain a second representation representing the storyboard segment.

[0087] Optionally, the fragment represents that the prediction model is specifically used for:

[0088] Extracting a plurality of second representative frames from the storyboard segments, and generating a second representative frame vector based on the second representative frames;

[0089] converting a plurality of said second representative frame vectors into third features suitable for input into said multimodal large model;

[0090] Based on the second prompt, generating a fourth feature suitable for input into the multimodal large model;

[0091] The third feature and the fourth feature are inputted into the multimodal large model together to obtain the second representation representing the storyboard segment.

[0092] Optionally, the second representation is a second text description of the storyboard segment;

[0093] The target segment matching unit is specifically used for:

[0094] extracting a plurality of first keywords from the advertising demand information;

[0095] For each of the first keywords, determining a second similarity between the first keyword and the second text description;

[0096] The second matching degree between the advertisement demand information and the second text description is determined based on the second similarity between each of the first keywords and the second text description.

[0097] Optionally, the target segment matching unit is specifically used to:

[0098] Segmenting the advertising demand information;

[0099] Identifying stop words in the separated words and removing the stop words;

[0100] Among the words after removing the stop words, non-entity words are removed to obtain the multiple first keywords.

[0101] Optionally, the target segment matching unit is specifically used to:

[0102] For each of the first keywords, determining a second word frequency of the first keyword appearing in the second text description;

[0103] determining a second inverse document frequency of occurrence of the first keyword in a plurality of the second text descriptions;

[0104] The second similarity is determined based on the second term frequency and the second inverse document frequency.

[0105] Optionally, the target segment matching unit is specifically used to:

[0106] For each of the first keywords, determining a second number of times the first keyword appears in the second text description;

[0107] extracting third keywords from the second text description, and determining a third keyword number of the third keywords;

[0108] The second word frequency is determined based on the second number of times and the third number of keywords.

[0109] Optionally, the target segment matching unit is specifically used to:

[0110] determining a second number of the second text descriptions containing the first keyword;

[0111] Get the total number of second text descriptions;

[0112] Based on the second number and the second total number of text descriptions, the second inverse document frequency is determined.

[0113] Optionally, the target segment matching unit is specifically used to:

[0114] extracting third keywords from each of the second text descriptions;

[0115] Taking each of the extracted third keywords as a second index, the second index and the second text description identifier of each of the second text descriptions containing the third keyword are stored in a second index comparison table in correspondence with each other;

[0116] The second index comparison table is searched using the first keyword as the second index to obtain the second text description identifier stored corresponding to the second index, thereby determining the second number of the second text descriptions containing the first keyword.

[0117] Optionally, the second representation is a third abstract feature of the storyboard segment;

[0118] The target segment matching unit is specifically used for:

[0119] Inputting the advertisement demand information into the segment representation prediction model to obtain a second abstract feature of the advertisement demand information;

[0120] The second matching degree is calculated based on the second abstract feature and the third abstract feature of each of the storyboard segments.

[0121] Optionally, the segment representation prediction unit is specifically used for:

[0122] From the target video, identifying a public figure in the plot segment;

[0123] Obtaining public figure information of the public figure;

[0124] The plot segment and the public figure information are input into the segment representation prediction model to obtain the first representation representing the plot segment.

[0125] Optionally, the segment indicates that the prediction model is a multimodal large model;

[0126] The segment representation prediction unit is specifically used for:

[0127] Combining the public figure information with the second prompt to obtain a second combined text;

[0128] The plot segment and the second combined text are input into the multimodal large model to obtain the first representation representing the plot segment.

[0129] According to one aspect of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method for processing advertisement implantation information when executing the computer program.

[0130] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for processing advertisement implantation information is implemented.

[0131] According to one aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer program. The computer program is read and executed by a processor of a computer device, so that the computer device executes the above-mentioned method for processing advertisement placement information.

[0132] In the disclosed embodiment, the video frames belonging to the same storyboard in the target video to be implanted with advertising information are aggregated into storyboard segments, and then the storyboard segments belonging to the same plot in the target video are aggregated into plot segments. Such plot segments can have common complete semantics because they belong to the same plot. The plot segments are input into the segment representation prediction model to obtain a first representation representing the plot segments. This first representation expresses the complete semantics of the plot segments. The advertising demand information is compared with the first representation of each plot segment to obtain the first matching degree of the advertising demand information and the first representation of each plot segment. This matching degree represents the precise matching degree between the semantics of the entire advertising demand information and the semantics of each plot segment. The video segments implanted with advertising information are determined by the matching degree. Due to the deep semantic analysis, the video segments implanted with advertising information have higher accuracy than the simple label comparison of the prior art. At the same time, since the disclosed embodiment relies on deep semantic analysis and matching to identify the advertising implantation position, it does not rely on a fixed label system, which alleviates the disadvantage of poor flexibility brought by the fixed label system and has higher flexibility in advertising implantation processing.

[0133] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0134] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0135] Figure 1 This is a schematic diagram of extracting tags from videos to generate a tag library;

[0136] Figure 2 is a system architecture diagram applied by the advertisement placement information processing method according to an embodiment of the present disclosure;

[0137] FIG. 3A to FIG. 3DIt is a schematic diagram of obtaining a matched target segment based on an input advertisement demand in a Ruyite advertisement scenario in which an embodiment of the present disclosure is applied;

[0138] Figure 4 is a flow chart of a method for processing advertisement implantation information according to an embodiment of the present disclosure;

[0139] Figure 5 yes Figure 4 In step 430, the plot segment is input into the segment representation prediction model to obtain a first flow chart representing a first representation of the plot segment;

[0140] Figure 6 yes Figure 5 Obtaining a schematic diagram representing a first representation of the plot segment;

[0141] Figure 7 yes Figure 5 Step 510 is a flowchart of obtaining text information corresponding to the plot segment from the target video;

[0142] Figure 8 yes Figure 7 A schematic diagram of obtaining text information corresponding to a plot segment;

[0143] Fig. 9 yes Figure 7 A flowchart of step 710 using a subtitle recognition model and a speech recognition model to identify line information from a plot segment;

[0144] Fig.10 yes Fig. 9 A schematic diagram of using subtitle recognition model and speech recognition model to recognize dialogue information;

[0145] Fig.11 yes Fig. 9 A flowchart of determining speech information from a plurality of candidate speech information based on the first probability and the second probability respectively corresponding to each candidate speech information in step 930;

[0146] Fig.12 yes Figure 5 In step 520, the plot segment and the text information are input into the segment representation prediction model to obtain a flow chart representing a first representation of the plot segment;

[0147] Fig.13 yes Fig.12 Obtaining a schematic diagram representing a first representation of the plot segment;

[0148] Fig.14 yes Fig.12 In step 1220, the plot segment and the first combined text are input into the multimodal macro model to obtain a flow chart representing a first representation of the plot segment;

[0149] Fig.15 yes Fig.14 A schematic diagram of a first representation of a plot segment obtained based on the multimodal large model;

[0150] Fig.16 yes Figure 4 In step 430, the plot segment is input into the segment representation prediction model to obtain a second flow chart representing a first representation of the plot segment;

[0151] Fig.17 yes Fig.16 Combined with the public figure information, a schematic diagram representing a first representation of the plot segment is obtained;

[0152] Fig.18 yes Fig.16 In step 1630, the plot segment and the public figure information are input into the segment representation prediction model to obtain a flow chart representing the first representation of the plot segment;

[0153] Fig.19 yes Fig.18 Based on the second combined text, a schematic diagram representing a first representation of the plot segment is obtained;

[0154] Fig. 20 yes Figure 4 A first flow chart of determining a first degree of matching between the advertisement demand information and the first representation of each plot segment in step 450;

[0155] Fig.21 yes Fig. 20 A schematic diagram of determining a first matching degree between the advertisement demand information and the first text description;

[0156] Fig. 22 yes Fig. 20 A flow chart of extracting a plurality of first keywords from the advertisement demand information in step 2010;

[0157] Fig.23 yes Fig. 22 A schematic diagram for extracting the first keyword;

[0158] Fig.24 yes Fig. 20 A flowchart of step 2020 for determining a first similarity between the first keyword and the first text description for each first keyword;

[0159] Fig.25 yes Fig.24 A flowchart of step 2410 for determining, for each first keyword, a first word frequency of the first keyword appearing in the first text description;

[0160] Fig.26 yes Fig.25 A schematic diagram for determining the first word frequency of each first keyword;

[0161] Fig. 27 yes Fig.24 A flowchart of step 2420 determining a first inverse document frequency of a first keyword appearing in a plurality of first text descriptions;

[0162] Fig.28 yes Fig. 27 A schematic diagram of determining a first inverse document frequency of each first keyword;

[0163] Fig.29 yes Fig. 27 A flowchart of step 2710 determining a first number of first text descriptions containing a first keyword;

[0164] Fig.30 yes Fig. 27 Constructing a first index comparison table to determine a schematic diagram of the first number;

[0165] Fig.31 yes Figure 4 A second flow chart of determining a first degree of matching between the advertisement demand information and the first representation of each plot segment in step 450;

[0166] Fig.32 yes Fig.31 A schematic diagram of determining a first matching degree based on a first abstract feature and a second abstract feature;

[0167] Fig.33 yes Figure 4 A flowchart of determining a first matching degree between the advertisement demand information and the first representation of each plot segment in step 450, and determining a target segment from a plurality of plot segments based on the first matching degree;

[0168] Fig.34 yes Fig.33 A schematic diagram of segment matching based on a first representation and a second representation;

[0169] Fig.35 yes Fig.33 In step 3310, the storyboard segment is input into the segment representation prediction model to obtain a flow chart representing a second representation of the storyboard segment;

[0170] Fig.36 yes Fig.35 Determine a schematic diagram of a second representation based on the storyboard segment and the second prompt;

[0171] Fig.37 yes Fig.35In step 3510, the storyboard segment and the second prompt are inputted into the segment representation prediction model to obtain a flow chart representing the second representation of the storyboard segment;

[0172] Fig.38 yes Fig.37 A schematic diagram of obtaining a second representation based on the fragment representation prediction model;

[0173] Fig.39 yes Fig.33 A first flow chart of determining a second degree of matching between the advertisement demand information and the second representation of each storyboard segment in step 3320;

[0174] Fig.40 yes Fig.39 A schematic diagram of determining a second matching degree between the advertisement demand information and the second text description;

[0175] Fig.41 yes Fig.39 A flow chart of extracting a plurality of first keywords from the advertisement demand information in step 3910;

[0176] Fig.42 yes Fig.39 A flowchart of step 3920 determining, for each first keyword, a second similarity between the first keyword and the second text description;

[0177] Fig.43 yes Fig.42 A flowchart of step 4210 determining, for each first keyword, a second word frequency of the first keyword appearing in the second text description;

[0178] Fig.44 yes Fig.43 A schematic diagram for determining the second word frequency of each first keyword;

[0179] Fig.45 yes Fig.42 A flowchart of step 4220 determining a second inverse document frequency of a first keyword appearing in a plurality of second text descriptions;

[0180] Fig.46 yes Fig.45 A schematic diagram of determining a second inverse document frequency of each first keyword;

[0181] Fig.47 yes Fig.45 A flowchart of step 4510 determining a second number of second text descriptions containing the first keyword;

[0182] Fig.48 yes Fig.47 Based on the second index comparison table, determine a schematic diagram of the second number;

[0183] Fig.49 yes Fig.33 A second flow chart of determining a second degree of matching between the advertisement demand information and the second representation of each storyboard segment in step 3320;

[0184] Fig.50 yes Fig.49 A schematic diagram of determining a second matching degree based on a second abstract feature and a third abstract feature;

[0185] Fig.51 yes Fig.33 Step 3330 is a first flowchart of determining a target segment from among a plurality of plot segments and a plurality of storyboard segments based on the first matching degree and the second matching degree;

[0186] Fig.52 yes Fig.33 Step 3330 is a second flowchart of determining a target segment from a plurality of plot segments and a plurality of storyboard segments based on the first matching degree and the second matching degree;

[0187] Fig.53 yes Fig.52 A schematic diagram for determining the target fragment;

[0188] Fig.54 is a module diagram of an advertisement placement information processing device according to an embodiment of the present disclosure;

[0189] Fig.55 According to the embodiment of the present disclosure Figure 4 The terminal structure diagram of the advertisement embedding information processing method shown;

[0190] Fig.56 According to the embodiment of the present disclosure Figure 4 The server structure diagram of the advertisement embedding information processing method shown. DETAILED DESCRIPTION

[0191] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0192] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0193] Scene advertising: Scene advertising is a type of advertising in which brands look for implanted scenes that match the product's tone based on the plot context or video images, accurately combine the elements in the scene with the brand / product, and appear in the form of a floating, non-interrupted video screen.

[0194] Automatic Speech Recognition (ASR): technology that converts speech signals into text information.

[0195] Optical Character Recognition (OCR): refers to the technology of converting text images into machine-readable text information.

[0196] Transformer model: It is a deep learning model structure based on the self-attention mechanism, which has a good effect in natural language processing tasks. In the self-attention mechanism, the Transformer model calculates the degree of association between each token in the input sequence and other tokens, thereby capturing the long-distance dependencies within the sequence. This mechanism enables the Transformer to process the entire sequence in parallel, improving computational efficiency.

[0197] Token: A token in Transformer refers to a word or subword in a text (such as a single character in Chinese), which is the basic unit used by the model when processing sequences. In a multimodal model, visual information is also mapped to a token and input into the language model.

[0198] Term Frequency-Inverse Document Frequency (TF-IDF): is a statistical method used to evaluate the importance of a word in a document collection or corpus. The core idea is to measure the importance of a word to a document by calculating the frequency of occurrence of a word in a specific document (i.e., term frequency, TF) and the sparsity of the word in the entire corpus (i.e., inverse document frequency, IDF).

[0199] On video websites, it is often necessary to insert advertisements. For example, in a certain video, there is a scene where the heroine applies skin care products. At this time, a floating layer appears in the picture, showing an advertisement for lotion A. In related technologies, when inserting advertisements, video features and audio features are often extracted from the target video, and deep learning models are used to perform element recognition on the video features and audio features, such as target detection, face recognition, behavior recognition, and place recognition. The identified objects, people, behaviors, places, and other elements are labeled to form labels, such as Figure 1Then, the advertisement demand information is compared with the tag. If a tag matching the advertisement demand information appears, the advertisement information is implanted.

[0200] The disadvantages of this solution are insufficient flexibility and poor accuracy. The lack of flexibility is reflected in the difficulty in covering flexible and changeable advertising needs. Once there is an advertising demand that cannot be covered by the existing label system (for example, the existing label system has hundreds of foods, but the advertising demand is for "snacks"), it is difficult to find the location for advertising in the video. The poor accuracy is reflected in the fact that sometimes although a certain label appears in the video, the location where the label appears may not be suitable for inserting advertising information, resulting in poor advertising insertion effect.

[0201] System architecture and scenario description of the application of the embodiments of the present disclosure

[0202] Figure 2 2 is a system architecture diagram of the method for processing advertisement placement information according to an embodiment of the present disclosure, which includes a server 210, the Internet 220, and an object terminal 230.

[0203] The server 210 refers to a computer system that can provide advertising insertion information processing services. For example, it can match and output the video clip that best matches the advertising demand information based on the advertising demand information from the target terminal 230. Compared with the target terminal 230, the server 210 has higher requirements in terms of stability, security, performance, etc. The server 210 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. The server 210 can also communicate with the Internet 220 in a wired or wireless manner to exchange data.

[0204] The object terminal 230 is a device that displays a video based on the embedded advertisement to the object, or provides an operation interface for the object to input the advertisement demand information, select / upload the video to be embedded with the advertisement. It includes various forms such as desktop computers, laptop computers, PDAs (personal digital assistants), mobile phones, car terminals, home theater terminals, and dedicated terminals. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network and share a display device to work together, forming a terminal. The terminal can also communicate with the Internet 220 in a wired or wireless manner to exchange data.

[0205] The embodiments of the present disclosure can be applied in various scenarios, such as FIG. 3A to FIG. 3D The scene shown in the Ruyi Tie advertisement.

[0206] like Figure 3AThe content management platform shown has an interface that can display the content library to advertising users. The content library provides advertising users with a list of optional drama titles, and advertising users select the dramas for which scene advertising is required by clicking or selecting a box.

[0207] After the user completes the selection of the episodes to be placed, the user will be redirected to the advertising demand input interface. Figure 3B In this interface, advertising users can enter specific advertising needs in the advertising demand input box according to specific advertising needs, and click "Search" to start segment matching. For example, the product to be advertised, the advertising words, the target scene of the advertisement, etc.

[0208] In the background server of the content management platform, multiple episodes that need to be advertised are respectively used as target videos. For each target video, the background server aggregates the video frames belonging to the same storyboard in the target video into storyboard segments; aggregates the storyboard segments belonging to the same plot in the target video into plot segments; inputs the plot segments into the segment representation prediction model to obtain a first representation representing the plot segments. The background server then determines the first matching degree between the advertising demand information and the first representation of each plot segment based on the advertising demand information input by the advertising delivery user, and determines the target segment from the multiple plot segments based on the first matching degree.

[0209] After the backend server determines the target segments corresponding to each episode that needs to be advertised, it will generate an advertisement matching list based on the title of the target segment, the corresponding time (hour: minute: second), the matching text description (first representation), and the specific thumbnail corresponding to the target segment. It will also be displayed to the advertising user on the interface of the content management platform for confirmation or further screening by the advertising user, such as Figure 3C shown.

[0210] After the user who places the advertisement confirms that the target segment that the advertisement matches is correct, the backend server embeds the target advertisement information in the target segment. In this example, the advertisement floating layer is "pasted" in the target segment of the drama in the form of a Ruyi post. Figure 3D As shown, when a video viewing user watches a corresponding episode on a terminal (such as a mobile phone), he or she will see a corresponding advertisement floating layer in a scene related to the advertisement.

[0211] In other examples, when matching advertisements with target clips, the backend server will also extract the storyboard clips of the target video and input the storyboard clips into the clip representation prediction model to obtain the corresponding second representation. The advertisement demand information is matched with both the first representation and the second representation to obtain the matched plot clips or storyboard clips. In this way, advertisement matching can be performed based on a variety of different segmentation granularities, thereby improving the matching accuracy.

[0212] In summary, the video frames belonging to the same storyboard in the target video to be implanted with advertising information are aggregated into storyboard segments, and then the storyboard segments belonging to the same plot in the target video are aggregated into plot segments. Such plot segments can have common complete semantics because they belong to the same plot. The plot segments are input into the segment representation prediction model to obtain a first representation representing the plot segment. This first representation expresses the complete semantics of the plot segment. The advertising demand information is compared with the first representation of each plot segment to obtain the first matching degree of the advertising demand information and the first representation of each plot segment. This matching degree represents the precise matching degree between the semantics of the entire advertising demand information and the semantics of each plot segment. The video segment to be implanted with advertising information is determined by this matching degree. Due to the deep semantic analysis, it has higher accuracy than the simple label comparison of the related technology. At the same time, since this example relies on deep semantic analysis and matching to identify the advertising implantation location, it does not rely on a fixed label system, which alleviates the disadvantage of poor flexibility brought by the fixed label system and has higher flexibility in advertising implantation processing.

[0213] It should be understood that the above content only illustrates some application scenarios of the present disclosure. The business scenarios to which the present disclosure can be applied may include but are not limited to the specific embodiments listed above.

[0214] General description of the disclosed embodiments

[0215] It should be emphasized that the embodiments of the present disclosure can be applied to a variety of application scenarios, such as film and television drama playback, video websites, video software, live broadcast advertising insertion, media information processing and other scenarios. In the related art, by labeling the elements identified from the video, matching the labels with the advertising demand information to implant advertising information, there are problems of insufficient flexibility and poor accuracy. Some embodiments of the present disclosure provide an advertising implantation information processing method, related devices and media, which can improve the flexibility and accuracy of advertising implantation information processing.

[0216] The method for inserting advertisement information is to obtain the first representation of each plot segment of the video through a segment representation prediction model, and to accurately match the semantics of the first representation with the advertisement demand, thereby determining the video segment suitable for inserting advertisement information.

[0217] The advertisement placement information processing method of the disclosed embodiment may be executed on the server 110 , or on the target terminal 130 , or may be partially executed on the server 110 and the other partially executed on the target terminal 130 .

[0218] like Figure 4 As shown, according to one embodiment of the present disclosure, the method for processing advertisement implantation information includes:

[0219] Step 410: Aggregate video frames belonging to the same storyboard in the target video into storyboard segments, wherein the target video is the video to be embedded with target advertising information;

[0220] Step 420, aggregating the storyboard segments belonging to the same plot in the target video into a plot segment;

[0221] Step 430: input the plot segment into the segment representation prediction model to obtain a first representation representing the plot segment;

[0222] Step 440: Obtain advertising demand information;

[0223] Step 450: determining a first matching degree between the advertisement demand information and the first representation of each plot segment, and determining a target segment from the plurality of plot segments based on the first matching degree;

[0224] Step 460: embed target advertisement information into the target segment.

[0225] In step 410, the target video refers to the video in which the target advertisement information needs to be embedded. For example, the target video can be a video selected by the advertisement delivery user as required. The target video can be a certain episode or several episodes, or a certain film and television drama, and a film and television drama includes multiple episodes.

[0226] Video frames of the same storyboard refer to video frames with the same or similar scenes and the same shot layout in the video. Storyboard segments refer to segments obtained by aggregating video frames belonging to the same storyboard and used to represent a storyboard. Each storyboard segment contains relatively coherent video screen content.

[0227] For example, the target video is segmented into multiple storyboard segments by using a shot segmentation tool (PySceneDetect) based on screen statistics. PySceneDetect is an open source tool for detecting scene changes in videos and automatically segmenting videos. It mainly identifies shot switching by analyzing pixel changes in each frame and segments the video into multiple segments.

[0228] PySceneDetect performs scene change detection on the target video, identifies the transition points of different scenes by analyzing the video frames of the target video; then analyzes the target video frame by frame, and counts the information of each video frame, such as the average pixel value of each video frame; based on the preset threshold value and the counted information of each video frame, performs scene cutting. PySceneDetect ultimately outputs multiple video clips with clear scene change points, i.e., storyboard clips. It should be noted that PySceneDetect is a specific model example provided in this example. In addition, other shot segmentation models can also be used, which are not specifically limited in this disclosure.

[0229] In step 420, the storyboard segments of the same plot refer to the storyboard segments that occur in the same scene and can form a continuous story.

[0230] A plot clip is a video clip that is aggregated from multiple storyboard clips and is used to represent a continuous plot story.

[0231] Exemplarily, the plot segments in this embodiment can be obtained by aggregating storyboard segments using a pre-trained plot segmentation model.

[0232] In this example, the plot segmentation model is a model that uses a recurrent neural network (RNN). For example, the plot segmentation model can be trained by inputting a training set of shot features with segmentation point labels, so that the trained plot segmentation model can automatically identify the segmentation points in the continuous multiple shot features input to it, and aggregate the shot features between each two adjacent segmentation points into a plot segment.

[0233] In this example, the multiple storyboard segments obtained in step 410 are input into the trained plot segmentation model, and the plot segmentation model automatically identifies the segmentation points in the multiple storyboard segments. Finally, the plot segmentation model aggregates the storyboard segments between each two adjacent segmentation points into one plot segment, thereby aggregating the multiple storyboard segments into at least one plot segment. It should be noted that using the plot segmentation model to obtain the plot segment is only a specific model example provided in this example. In addition, models with other neural network structures can also be used, which is not specifically limited in this disclosure.

[0234] In step 430, the segment representation prediction model refers to a model that can analyze the plot segment, obtain semantic information corresponding to the plot segment, and output it in various forms. For example, the segment representation prediction model can be a multimodal large model. The semantic information corresponding to the plot segment refers to information that can characterize the specific content of the plot segment.

[0235] The first representation refers to the information of the summary description of the content corresponding to the plot segment, which is output by analyzing and predicting the input plot segment through the segment representation prediction model, and is used to characterize the semantic information corresponding to the plot segment. For example, the first representation can be a text description composed of natural language, or an abstract expression composed of feature vectors.

[0236] In step 440, the advertisement demand information refers to information used to characterize the expectations / demands of the advertisement placement user for the location of the advertisement in the video. For example, the advertisement demand information may include specific advertisement words, the product corresponding to the advertisement, the celebrity endorsed by the advertisement, and the scene applicable to the advertisement product (such as family gatherings, outdoor sports, etc.).

[0237] For example, the advertisement demand information can be obtained by the advertisement delivery user through FIG. 3A to FIG. 3C The provided content management platform is input into the system so that the system can obtain the advertisement demand information when executing the advertisement placement information processing method. The advertisement demand information can also be extracted from an advertisement demand information table pre-stored in the system, which is obtained by statistically analyzing the needs of users who place advertisements.

[0238] In step 450, the first matching degree refers to the semantic similarity between the advertisement demand information and the first representation of the plot segment. The higher the similarity, the higher the first matching degree.

[0239] After determining the first matching degree between each plot segment and the advertising demand information, the plot segment with the corresponding first matching degree meeting the requirement is used as the target segment. For example, the plot segment with the first matching degree greater than a preset matching degree threshold can be used as the target segment. Alternatively, the plot segments are sorted from large to small according to the first matching degree, and the plot segments with the highest sorting, such as the top 10%, are selected as the target segments. Alternatively, the plot segment with the largest first matching degree is selected as the target segment.

[0240] In step 460, the target advertising information refers to the advertising information corresponding to the advertising demand information and expected to be embedded in the target segment of the target video. For example, the target advertising information may include product name, advertising words, advertising pictures, and advertising forms (such as floating layer, video top bullet screen, video bottom bullet screen, etc.).

[0241] For example, the target advertisement information can be embedded in the form of a floating layer. Specifically, the product name, advertisement words, and advertisement pictures corresponding to the target advertisement are edited to obtain a completed floating layer image. The floating layer image is set on the corresponding target segment. Thus, when the video plays to these target segments, the corresponding floating layer image will appear to display the advertisement, such as Figure 3D shown.

[0242] In the above-mentioned embodiment of step 410 to step 460, the video frames belonging to the same storyboard in the target video to be implanted with advertising information are aggregated into storyboard segments, and then the storyboard segments belonging to the same plot in the target video are aggregated into plot segments. Such plot segments can have common complete semantics because they belong to the same plot. The plot segments are input into the segment representation prediction model to obtain a first representation representing the plot segments. This first representation expresses the complete semantics of the plot segments. The advertising demand information is compared with the first representation of each plot segment to obtain the first matching degree of the advertising demand information and the first representation of each plot segment. This matching degree represents the precise matching degree between the semantics of the entire advertising demand information and the semantics of each plot segment. The video segments implanted with advertising information are determined by the matching degree. Due to the deep semantic analysis, the simple label comparison of the related technology has higher accuracy. At the same time, since the embodiment of the present disclosure relies on deep semantic analysis and matching to identify the advertising implantation position, it does not rely on a fixed label system, which alleviates the disadvantage of poor flexibility brought by the fixed label system and has higher flexibility in advertising implantation processing.

[0243] The above is a general description of step 410 to step 460. Step 410, step 420, step 440, and step 460 have been described in detail above, so only the specific implementation process of step 430 and step 450 will be described in detail below.

[0244] Detailed description of step 430

[0245] In step 430 , the plot segment is input into a segment representation prediction model to obtain a first representation representing the plot segment.

[0246] In one embodiment, referring to Figure 5 , step 430 includes:

[0247] Step 510: Obtain text information corresponding to the plot segment from the target video;

[0248] Step 520: Input the plot segment and the text information into a segment representation prediction model to obtain a first representation representing the plot segment.

[0249] The steps 510 and 520 are described in detail below.

[0250] In step 510, the text information corresponding to the plot segment refers to information expressed in text form, including lines, characters, etc. in the plot segment.

[0251] In some examples, the text information may include only dialogue information, or only character information, or both dialogue and character information based on demand.

[0252] In other examples, the character information may be character information corresponding to a character in a plot, or may be information of an actor who plays the character.

[0253] In step 520, the first representation refers to information representing the plot summary of the plot segment, which is output by extracting semantics from the plot segment and the text information respectively through the segment representation prediction model and combining the semantics of the two to predict and output. In this embodiment, the first representation can be a summary description in text form or summary information expressed by a feature vector.

[0254] Combine the following Figure 6 , the process of obtaining a first representation of a plot segment through a segment representation prediction model is explained.

[0255] like Figure 6 As shown, each target video that needs to be placed for advertising is preprocessed. The preprocessing includes: segmenting each target video to obtain multiple plot segments, such as segment 1 to segment n in the figure; performing text extraction on each target video to obtain text information containing line information, and segmenting the text information to obtain text 1 to text n corresponding to segment 1 to segment n. In other examples, text extraction can also be performed on each plot segment to obtain text information corresponding to each plot segment.

[0256] Segments 1 to segment n are input into the segment representation prediction model in pairs with the corresponding texts 1 to text n. The segment representation prediction model outputs the first representation corresponding to each segment, such as plot 1 to plot n corresponding to segments 1 to segment n in the figure. In this example, the first representation of the plot segment can be a descriptive text of the plot summary of the plot segment, or a feature vector corresponding to the plot summary of the plot segment.

[0257] In the related art, text extraction and video feature extraction are performed on the video to obtain text content and video feature content, and then the text content and video feature content are labeled to obtain various labels. In the above steps 510 and 520, the plot segment and the corresponding text information are input into the segment representation prediction model to obtain the first representation, so that the semantics of the plot segment corresponding to the first representation is more complete, which can improve the accuracy of matching the first representation with the advertising demand information.

[0258] In one embodiment, referring to Figure 7 , step 510 includes:

[0259] Step 710: using a subtitle recognition model and a speech recognition model to identify lines from the plot segment;

[0260] Step 720: using a face recognition model, identifying character information corresponding to the lines from the plot clips;

[0261] Step 730: Integrate the dialogue information and the character information corresponding to the dialogue information into text information.

[0262] Steps 710 to 730 are described in detail below.

[0263] In step 710, the subtitle recognition model uses optical character recognition technology to perform text recognition and extraction on each video frame of the plot segment to obtain the line information in the plot segment. In other examples, other subtitle recognition models based on image processing can also be selected, which are not limited here.

[0264] Exemplarily, the speech recognition model uses automatic speech recognition ASR technology to recognize and convert the speech information of the characters in the plot segment into text information, thereby obtaining the line information in the plot segment. In other examples, other models that can convert speech information into text information can also be selected, which are not specifically limited here.

[0265] Line information refers to the lines spoken by each character in the plot segment. Lines are an important element to promote the development of the plot and explain the plot to the audience. Therefore, extracting the line information in the plot segment as one of the text information is conducive to the semantic analysis of the plot segment.

[0266] In step 720, the face recognition model is a model that can recognize the characters appearing in the plot segment and obtain the character's name, appearance time, location, etc. The character information includes the name, appearance time, appearance location, etc. of the character corresponding to the line information.

[0267] In step 730, the text information integrates the dialogue information and the character information. At this time, the text information includes the specific dialogue of the plot segment and the character corresponding to the dialogue.

[0268] Combine the following Figure 8 , explains the process of obtaining text information from plot clips. Figure 8 As shown, the target video is segmented to obtain multiple plot segments, such as segment 1 to segment n in the figure. Segments 1 to segment n are respectively input into the speech recognition model, the subtitle recognition model, and the face recognition model.

[0269] The speech recognition model recognizes the dialogue information in segments 1 to n from the speech level. The subtitle recognition model recognizes the dialogue information in segments 1 to n from the image (subtitle) level. The recognition results of the speech recognition model and the subtitle recognition model are combined to obtain dialogue 1 to dialogue n corresponding to segments 1 to n.

[0270] The face recognition model performs face recognition on segments 1 to n respectively to obtain character information of each plot segment, as shown in Figure 1 to character n.

[0271] Integrate lines 1 to lines n with characters 1 to characters n one by one, for example, integrate line 1 with character 1, integrate line 2 with character 2, and so on. Finally, the text information corresponding to each plot segment is obtained, as shown in text 1 to text n.

[0272] If text information is directly extracted from the plot clips, the amount of calculation is large, and the model requirements are high, and the extraction results are usually difficult to meet the requirements. In the above steps 710 to 730, the speech recognition model and the subtitle recognition model are used to extract the line information in the speech dimension and the image dimension, and the face recognition model is used to extract the character information. Different elements (lines, characters) in the plot clips are extracted respectively by different models, and then integrated to obtain the final text information. Compared with directly extracting all the information from the plot clips, this embodiment uses different models to extract the required information in a targeted manner and then integrates it to obtain text information. It can obtain various types of required information more accurately, and has low model requirements and a small computational load.

[0273] In one embodiment, referring to Fig. 9 , step 710 includes:

[0274] Step 910: Using a subtitle recognition model, perform subtitle recognition in the plot segment to obtain first probabilities corresponding to a plurality of candidate dialogue information;

[0275] Step 920: Perform speech recognition in the plot segment using the speech recognition model to obtain second probabilities corresponding to a plurality of candidate dialogue information.

[0276] Step 930: Determine the speech information from the plurality of candidate speech information based on the first probability and the second probability respectively corresponding to each candidate speech information.

[0277] Steps 910 to 930 are described in detail below.

[0278] In step 910, when using the subtitle recognition model to recognize the subtitles in the plot segment and output the line information, while generating the line information, the subtitle recognition model generates probability information corresponding to the line information, that is, the first probability. The candidate line information is the line information corresponding to the plot segment output by the subtitle recognition model. Since the line information at this time is not the final result, it is the candidate line information. There is a first probability corresponding to one candidate line information.

[0279] In step 920, when the speech information in the plot segment is recognized and outputted using the speech recognition model, while generating the speech information, the speech recognition model will generate probability information corresponding to the speech information, i.e., the second probability. The candidate speech information is the speech information corresponding to the plot segment outputted by the speech recognition model. Since the speech information at this time is not the final result, it is the candidate speech information. There is one second probability corresponding to one candidate speech information.

[0280] In step 930, when the same plot segment is identified, for each line therein, since the subtitle recognition model and the speech recognition model are used for recognition respectively, there will be two output results as candidate line information, and the candidate line information corresponds to a first probability and a second probability respectively.

[0281] For example, based on the magnitude of the first probability and the second probability, the candidate speech information corresponding to the larger probability may be selected as the final speech information.

[0282] Combine the following Fig.10 , explains the process of using subtitle recognition models and speech recognition models to recognize dialogue information.

[0283] The subtitle recognition model recognizes the plot segment and outputs candidate lines 1_1, candidate lines 2_1, ..., candidate lines n_1, each of which corresponds to a first probability. The speech recognition model recognizes the plot segment and outputs candidate lines 1_2, candidate lines 2_2, ..., candidate lines n_2, each of which corresponds to a second probability. Among them, candidate lines 1_1 and candidate lines 1_2 are a pair of candidate line information 1, and correspond to the first probability and the second probability; similarly, candidate lines 2_1 and candidate lines 2_2 are a pair of candidate line information 2, and correspond to the first probability and the second probability, and so on.

[0284] For each pair of candidate speech information, based on the larger probability of the first probability and the second probability, the candidate speech corresponding to the larger probability is used as the speech information. For example, in candidate speech information 1, the first probability is greater than the second probability, so candidate speech 1_1 is selected as the speech information, that is, speech information 1_1 is obtained; in candidate speech information 2, the second probability is greater than the first probability, so candidate speech 2_2 is selected as the speech information, that is, speech information 2_2 is obtained. And so on, finally, speech information 1_1, speech information 2_2, ..., speech information n_1 are obtained.

[0285] In the related art, a single model is usually used to extract subtitles from videos. Due to the existence of noisy ambient sounds, pronunciation problems, or inaccurate subtitles, a single speech recognition model or subtitle recognition model may have inaccurate determination problems. In the above steps 910 to 930, the subtitle recognition model and the speech recognition model are used simultaneously to identify the dialogue information of the plot clips to obtain multiple candidate dialogue information, and the candidate dialogue information with a higher probability is selected as the final dialogue information based on the first probability and the second probability. The dialogue information obtained in this way is determined by the candidate dialogue information with a higher probability (confidence) in the two models, which can improve the accuracy of dialogue information recognition.

[0286] In one embodiment, referring to Fig.11 , step 930 includes:

[0287] Step 1110, obtaining a first weight of a subtitle recognition model and a second weight of a speech recognition model;

[0288] Step 1120: Determine a weighted sum of the first probability and the second probability based on the first weight and the second weight;

[0289] Step 1130: Determine speech information from a plurality of candidate speech information based on the weighted sum.

[0290] Steps 1110 to 1130 are described in detail below.

[0291] In step 1110, the first weight is an indicator for characterizing the reliability of the recognition result of the subtitle recognition model. The second weight is an indicator for characterizing the reliability of the recognition result of the speech recognition model. Exemplarily, the first weight and the second weight can be pre-set based on the actual situation of the subtitles and the character's speech in the video. For example, in a certain video, the subtitles of the video are clear and complete, but the character's speech contains a large number of dialects or languages ​​from different countries. At this time, the recognition complexity of the speech recognition model is high and it is easy to produce deviations. At this time, the first weight is set to be greater than the second weight, and the first weight is set to be larger, and the second weight is set to be smaller.

[0292] Exemplarily, the subtitle recognition model and the speech recognition model are trained with the same type of video as samples, and the first weight and the second weight of the subtitle recognition model and the speech recognition model for the recognition of the type are set based on the accuracy of the output results of the subtitle recognition model and the speech recognition model during training. The video type refers to whether the video has a dialect, multiple languages, multiple language subtitles, missing / unclear subtitles, etc.

[0293] In step 1120, the weighted sum is the sum of the product of the first probability and the first weight and the product of the second probability and the second weight.

[0294] In step 1130, illustratively, when the first weight of the subtitle recognition model is greater than the second weight of the speech recognition model, the weighted sum is compared with a preset threshold. If the weighted sum is greater than the threshold, the candidate speech information output by the subtitle recognition model is used as the speech information, that is, the candidate speech information corresponding to the first probability is selected as the speech information. If the weighted sum is less than the threshold, the candidate speech information output by the speech recognition model is used as the speech information, that is, the candidate speech information corresponding to the second probability is selected as the speech information.

[0295] When the second weight of the speech recognition model is greater than the first weight of the subtitle recognition model, the weighted sum is compared with a preset threshold. If the weighted sum is greater than the threshold, the candidate speech information output by the speech recognition model is used as the speech information, that is, the candidate speech information corresponding to the second probability is selected as the speech information. If the weighted sum is less than the threshold, the candidate speech information output by the subtitle recognition model is used as the speech information, that is, the candidate speech information corresponding to the first probability is selected as the speech information.

[0296] For different videos, the difficulty of subtitle recognition or speech recognition is often different. If the selection is made only based on the probability corresponding to the candidate dialogue information in the respective models, the factor of the video type cannot be taken into account. In the above steps 1110 to 1130, the first weight of the subtitle recognition model and the second weight of the speech recognition model are obtained. When determining the dialogue information from multiple candidate dialogue information, the weighted sum of the first probability and the second probability is used as the basis for determining the dialogue information. For the same target video or plot clip, the larger the weight, the more accurate and reliable the recognition result of the corresponding model. Therefore, based on the weighted sum of the first probability and the second probability to determine the dialogue information, in addition to considering the probability of the model itself, the contribution of the model to the video recognition (first weight, second weight) is also comprehensively considered to improve the accuracy and credibility of the dialogue information.

[0297] In one embodiment, the segment representation prediction model is a multimodal large model. Fig.12 , step 520 includes:

[0298] Step 1210: Combine the text information with the first prompt to obtain a first combined text;

[0299] Step 1220: Input the plot segment and the first combined text into the multimodal large model to obtain a first representation of the plot segment.

[0300] Steps 1210 and 1220 are described in detail below.

[0301] In step 1210, the first prompt refers to text information input by the user for indicating that the segment representation prediction model outputs a first representation of the plot segment that meets the user's needs. In some examples, the first prompt may be a pre-set templated prompt. In other examples, the first prompt may also be a prompt input by the user in real time based on the plot segment.

[0302] For example, the first prompt is "Please combine the picture information and the line information to summarize the plot of this segment." At this time, the segment representation prediction model will output a summary description of the corresponding plot segment as the first representation under the guidance of the first prompt.

[0303] The first combined text refers to the text information obtained by combining the text information with the first prompt in a natural language. For example, when the text information includes dialogue information and character information, the specific dialogue and character included in the text information are combined with the first prompt "summarize the plot summary of this segment". The first combined text is "The characters in the plot segment include... (character information), and the character's dialogue is... (dialogue information). Please combine the picture information and dialogue information to summarize the plot summary of this segment."

[0304] In step 1220, the multimodal large model can integrate and process input data from different modalities (such as text, images, audio, etc.) and output corresponding content based on user needs.

[0305] After the plot segment and the first combined text are input into the multimodal large model, the multimodal large model will perform feature extraction and feature conversion on the plot segment, and map the plot segment to features that the multimodal large model can process. The multimodal large model will also perform feature extraction on the first combined text to obtain corresponding features. At this time, the features corresponding to the plot segment and the features corresponding to the first combined text belong to the features of the same modality.

[0306] The multimodal large model then analyzes and processes the plot fragment and the first combined text based on the semantics corresponding to the first prompt in the first combined text, and outputs the content required by the user (i.e., the first representation of the plot fragment).

[0307] Combine the following Fig.13 , the process of obtaining the first representation by inputting the plot fragment and the first combined text into the multimodal large model is explained. Fig.13 As shown, the plot fragment is as follows Fig.13 As shown in the six-grid image in the input screen.

[0308] Correspondingly, the text information of the plot segment includes the line information: [Everyone please stay|I have a suggestion|Today is a good day|Our two families are together|Let's take some photos as a souvenir, okay|That's right|This is a meeting of art students|It's worth remembering|Let's take photos of their two children first|They are busy|When we can't see them in the future|Let's look at the photos|Keep a memory|Okay|Come here|Come here|That's not right|Why are you two so unfamiliar|Get closer|Be more intimate|It seems that we have to teach by example|Sister Lu|Let's show them a demonstration|You|This way, this way|Yes|This way].

[0309] The first prompt is "Please combine the screen information and the lines information to summarize the plot of this clip." Combining the first prompt with the text information, we get Fig.13 The combined text (first combined text) is shown.

[0310] The plot clip and the combined text are input into the multimodal big model. Based on the first prompt, the multimodal big model determines that the specific content it needs to output should be the summary description corresponding to the plot clip. The multimodal big model extracts and analyzes the features of the plot clip and the lines in the combined text, obtains the semantic features corresponding to the image data and text data, and generates the target content based on the semantic features. The final output plot clip summary is: "This clip may be two families gathering in a restaurant, taking photos as a souvenir, Sister Lu and others try to get the two children to be closer, but the interaction between them is very unfamiliar."

[0311] In the related technologies, different models are used to extract features from plot segments and convert them into corresponding labels. Fig.13 After extracting the plot fragment, specific labels such as round table, chair, party, and Sister Lu (character) can be obtained. However, this does not allow the complete semantics of the plot fragment, which is not conducive to subsequent matching with advertising demand information. In the above steps 1210 and 1220, the multimodal large model can directly process the image data of the plot fragment and the text data of the first combined text. And the multimodal large model is taught by the first combined text combined with the first prompt and text information, so that the multimodal large model can accurately output the first representation of the plot fragment required by the user. Compared with the prior art that extracts a single label from the plot fragment, this embodiment uses the first prompt to guide the multimodal large model to perform high-level semantic extraction of the plot fragment in the dimension of plot segmentation, and obtains the first representation containing the overall semantic information of the plot fragment. Since the first representation contains the overall semantic information of the plot fragment, when it is subsequently matched with the advertising demand information, the matching accuracy can be improved, and a target fragment that is more adapted to the advertising demand information can be obtained.

[0312] In one embodiment, referring to Fig.14 , step 1220 includes:

[0313] Step 1410: extract multiple first representative frames from the plot segment, and generate first representative frame vectors based on the first representative frames;

[0314] Step 1420, converting the plurality of first representative frame vectors into first features suitable for input into a multimodal large model;

[0315] Step 1430: Generate a second feature suitable for input into a multimodal large model based on the first combined text;

[0316] Step 1440: Input the first feature and the second feature into the multimodal large model to obtain a first representation representing the plot segment.

[0317] Steps 1410 to 1440 are described in detail below.

[0318] In step 1410, the first representative frame refers to a video frame extracted from the plot segment. Exemplarily, N video frames are uniformly extracted from the plot segment as N first representative frames, where N is a natural number. In another example, the plot segment can be preliminarily divided by hierarchical clustering, and then the histogram features of the preliminarily extracted key frames are compared in combination with the k-means clustering algorithm to remove redundant frames, and finally determine the first representative frame sequence of the plot segment.

[0319] The first representative frame vector is a vector obtained by extracting features from the first representative frame. For example, a plurality of first representative frames can be converted into a plurality of d-dimensional feature vectors, i.e., a plurality of first representative frame vectors, through a video image feature model, such as Contrastive Language-Image Pre-training (CLIP).

[0320] In step 1420, the first feature can be obtained by converting the first representative frame vector through the cross-modal matching module. The first feature is data that can be understood by the multi-modal large model.

[0321] Exemplarily, the cross-modal matching module may adopt a transformer-based network or a multilayer perceptron (MLP).

[0322] In step 1430, the second feature refers to feature data that can be understood by the multimodal large model, obtained by extracting features from the first combined text.

[0323] In step 1440, the multimodal large model may adopt models such as LLaVa and InternVL, wherein LLaVA (Large Language and Vision Assistant) is a new type of end-to-end trained large multimodal model, which aims to combine visual encoders and large language models to achieve comprehensive understanding of images and texts and generate content. InternVL is a multimodal large language model (MLLM) that aims to achieve efficient processing of multiple visual-language tasks by extending the visual base model and aligning it with the large language model.

[0324] Combine the following Fig.15 , a process of obtaining a first representation by inputting the plot fragment and the first combined text into the multimodal large model is explained.

[0325] like Fig.15 As shown, a pair of plot clips and corresponding first combined texts are taken as an example. First, multiple video frames are uniformly extracted from the plot clips as first representative frames. Multiple first representative frames are input into the video image feature model for feature extraction to obtain multiple first representative frame vectors. Multiple first representative frame vectors are input into the cross-modal matching module for modal conversion, and semantic mapping of multiple first representative frame vectors is completed to obtain a more compact first feature.

[0326] While processing the first representative frame, feature extraction is performed based on the first combined text to obtain a second feature suitable for input into the multimodal large model. It should be noted that in this example, the multimodal large model can understand the semantics of the text modality data, and therefore maps the image features corresponding to the plot segment to feature data of the text modality.

[0327] The first feature and the second feature are input into the multimodal large model. The multimodal large model can combine the semantics corresponding to the plot segment and the semantics corresponding to the first combined text, perform high-level semantic extraction on the plot of the plot segment, and obtain a first representation of the plot segment.

[0328] The related technology is to use multiple models to train separately on subtasks, and extract different labels from the target video based on different models to match the advertising demand information. There are problems such as weak generalization ability and insufficient understanding of the plot of the target video. In the above steps 1410 to 1440, the multimodal large model is used to combine the input plot segment and the first combined text to output the first representation of the plot segment. Since the multimodal large model has a strong semantic understanding ability, under the prompt of the first combined text, it can better understand the plot context, emotion, context and other information in the plot segment, and can have a deeper understanding of the semantics of the plot segment. Therefore, the first representation output by the multimodal large model can fully characterize the deep semantics of the corresponding plot segment. Matching with the advertising demand information based on such a first representation is conducive to finding a target segment that better meets the advertising demand information and improves the effect of advertising implantation.

[0329] In one embodiment, referring to Fig.16 , step 430 includes:

[0330] Step 1610: identifying public figures in the plot clips from the target video;

[0331] Step 1620: Obtain public figure information of the public figure;

[0332] Step 1630: Input the plot segment and the public figure information into a segment representation prediction model to obtain a first representation representing the plot segment.

[0333] Steps 1610 to 1630 are described in detail below.

[0334] In step 1610, public figures refer to actors who play roles in the plot segment. Exemplarily, a face recognition model may be used to identify public figures in the plot segment.

[0335] In step 1620, the public figure information refers to information such as the actor's name, appearance time, and location.

[0336] In step 1630, the first representation in this embodiment includes semantic information of the plot summary corresponding to the plot segment and semantic information corresponding to the public figure.

[0337] Below, reference Fig.17 , explaining the process of obtaining the first representation by combining the public figure information. Fig.17As shown, each target video that needs to be placed for advertising is preprocessed separately. The preprocessing includes: segmenting each target video to obtain multiple plot segments, such as segment 1 to segment n in the figure; identifying public figures for each target video to obtain public figure information, such as public figures 1 to public figures n in the figure. In other examples, public figure identification can also be performed on each plot segment to obtain public figure information corresponding to each plot segment.

[0338] Segments 1 to n are respectively matched with corresponding public figures 1 to public figures n, and are input into the segment representation prediction model in pairs. The segment representation prediction model outputs the first representation corresponding to each segment, such as plot 1 to plot n corresponding to segments 1 to segment n in the figure. In this example, the first representation of the plot segment can be a descriptive text of the plot summary of the plot segment, or it can be a feature vector corresponding to the plot summary of the plot segment. In this example, plot 1 to plot n also carry public figure information. Therefore, in the subsequent process of matching the plot segment with the advertising demand information, the public figure can also be matched as a matching factor.

[0339] If the first representation of the plot segment is predicted directly through the segment representation prediction model, the segment representation prediction model may not be able to accurately identify the public figures appearing in the plot segment, so that the relevant information of the public figures may not exist in the first representation. In the above steps 1610 to 1630, the public figures in the plot are first identified from the target video, and the corresponding information of the public figures is obtained. Then the public figure information and the plot segment are input into the segment representation prediction model to obtain the first representation of the plot segment, so that the first representation can accurately carry the relevant information of the public figures appearing in the plot segment. When the advertising demand information of the designated public figure (such as endorsement advertising) is involved in the subsequent process, the public figure can also be involved in the matching process as a matching factor. Thereby accurately identifying the target segment corresponding to the advertising demand information of the designated public figure, improving the advertising implantation effect.

[0340] In one embodiment, the segment representation prediction model is a multimodal large model; Fig.18 , step 1630 includes:

[0341] Step 1810: Combine the public figure information into the second prompt to obtain a second combined text;

[0342] Step 1820: input the plot segment and the second combined text into the multimodal large model to obtain a first representation representing the plot segment.

[0343] Step 1810 and step 1820 are described in detail below.

[0344] In step 1810, the second prompt refers to text information input by the user to indicate that the segment representation prediction model outputs a first representation of the plot segment that meets the user's needs. In this embodiment, the second prompt includes at least information indicating that the first representation output by the segment representation prediction model needs to include public figure information. In some examples, the second prompt may be a pre-set templated prompt. In other examples, the second prompt may also be a prompt input by the user in real time based on the plot segment.

[0345] For example, the second prompt is "Please summarize the plot of this segment." At this time, the segment representation prediction model will output a summary description containing the public figure information as the second representation under the guidance of the second prompt.

[0346] The second combined text refers to the text information obtained by combining the public figure information with the second prompt in a natural language. For example, the public figure information including the specific actor name is combined with the second prompt "Please summarize the plot of this clip." The second combined text is "The public figure information in the plot clip includes actress A, actor A, middle-aged actress B, and middle-aged actor B. Please summarize the plot of this clip by combining the picture information and the public figure information."

[0347] In step 1820, the multimodal large model can integrate and process input data from different modalities (such as text, images, audio, etc.) and output corresponding content based on user needs.

[0348] After the plot segment and the second combined text are input into the multimodal large model, the multimodal large model will extract and convert the plot segment, and map the plot segment into features that the multimodal large model can process. The multimodal large model will also extract features from the second combined text to obtain corresponding features. At this time, the features corresponding to the plot segment and the features corresponding to the second combined text belong to the same modality.

[0349] The multimodal large model then analyzes and processes the plot fragment and the second combined text based on the semantics corresponding to the second prompt in the second combined text, and outputs the content required by the user (i.e., the first representation containing the public figure information).

[0350] Combine the following Fig.19 , explaining the process of obtaining the first representation by combining the public figure information. Fig.19 As shown, the plot fragment is as follows Fig.19 As shown in the six-grid image in the input screen.

[0351] Correspondingly, the public figure information of the plot segment includes actress A, actor A, middle-aged actress B, and middle-aged actor B.

[0352] The second prompt is "Please combine the screen information and the public figure information to summarize the plot of this clip." Combining the second prompt with the public figure information, we get the following: Fig.19 The combined text (second combined text) shown is: "This is a plot segment in a film or TV series. The public figures in the segment are: actress A, actor A, middle-aged actress B, and middle-aged actor B. \nPlease summarize the plot of this segment by combining the screen information and the public figure information."

[0353] The plot clip and the second combined text are input into the multimodal large model. Based on the second prompt, the multimodal large model determines that the specific content it needs to output should be a summary description corresponding to the plot clip, and it needs to include public figure information. The multimodal large model extracts and analyzes the features of the plot clip and the public figure information in the second combined text to obtain the semantic features corresponding to the image data and text data, and generates the target content based on the semantic features. The final output plot clip summary is: "This clip may be two families gathering in a restaurant and taking photos as a souvenir. Middle-aged actress B, middle-aged actor B and others try to make actress A and actor B closer, but the interaction between them is very unfamiliar."

[0354] In the existing technology of matching advertisements through video pre-labeling, the public figures in the video are usually directly matched as a separate label, that is, as long as the character appears in the video, the match is considered successful. However, in practice, there may be a scene in a certain video clip where the character exists that is not suitable for advertisement implantation. In the above steps 1810 and 1820, the public figure information is combined with the second prompt to generate a second combined text. The second combined text and the plot clip are input into the clip representation prediction model together, so that the output first representation includes the public figure information. In this way, when matching the clips of the advertising demand information (such as endorsement advertisements) of the specified public figure, it is possible to match the public figure together based on the plot semantics in the first representation, avoiding the situation where the public figure is matched but the plot of the corresponding target clip is not suitable for implanted advertising, thereby improving the user's acceptance of implanted advertising and avoiding discomfort.

[0355] Detailed description of step 450

[0356] In step 450, a first matching degree between the advertisement demand information and the first representation of each plot segment is determined, and a target segment is determined from among the plurality of plot segments based on the first matching degree.

[0357] In one embodiment, the first representation is a first text description of the plot segment. Fig. 20 , step 450 comprises:

[0358] Step 2010: extracting a plurality of first keywords from the advertisement demand information;

[0359] Step 220: for each first keyword, determine a first similarity between the first keyword and the first text description;

[0360] Step 2030: Determine a first matching degree between the advertisement demand information and the first text description based on a first similarity between each first keyword and the first text description.

[0361] Steps 2010 to 2030 are described in detail below.

[0362] In step 210, the first keyword refers to a word that represents key information of the advertising demand information, obtained by preprocessing the advertising demand information to remove redundant words therein.

[0363] In step 220, the first similarity refers to the TF-IDF value of the first keyword relative to the first text description. The first similarity is used to characterize the importance of the corresponding first keyword to the first text description.

[0364] In step 2030, the first matching degree is calculated based on the TF-IDF value of each first keyword in the advertisement demand information, and is a value used to represent the matching degree between the advertisement demand information and the first text description.

[0365] Next, combine Fig.21 The process of determining the first matching degree is explained. Fig.21 As shown, first keyword 1 to first keyword n ​​are extracted from the advertisement demand information. The first similarity between each first keyword and the first text description is calculated to obtain first similarity 1 to first similarity n respectively. The first similarity in this example is the TF-IDF value of the first keyword relative to the first text description.

[0366] The first keywords 1 to the first keywords n and the corresponding first similarities 1 to the first similarities n of the advertising demand information are converted into vector form. Each first similarity constitutes a dimension, and the first similarities 1 to the first similarities n are combined into a document vector of the advertising demand information.

[0367] The cosine similarity between the document vector of the advertising demand information and the document vector of the first text description is calculated, and the cosine similarity is used as the first matching degree between the advertising demand information and the first text description. The first matching degree ranges from [-1, 1], and the closer the value is to 1, the more matching the advertising demand information is with the first text description.

[0368] In the related art, by tagging the target video and matching each tag with the advertising demand information, there are problems of insufficient flexibility and poor matching accuracy. In the above steps 2010 to 2030, semantic similarity matching at the document level is performed between the first text description capable of representing the semantics of a complete plot segment and the advertising demand information. When matching, more semantic details can be considered, thereby improving the accuracy of the matching. And because the matching is based on semantics rather than fixed tags, the flexibility of the matching is higher.

[0369] In one embodiment, referring to Fig. 22 , step 2010 includes:

[0370] Step 2210: Segment the advertising demand information;

[0371] Step 2220: Identify stop words in the segmented words and remove the stop words;

[0372] Step 2230: Remove non-entity words from the words after removing the stop words to obtain multiple first keywords.

[0373] The following will describe steps 2210 to 2230 in detail.

[0374] In step 2210, word segmentation refers to dividing each word in the advertising demand information to obtain multiple words.

[0375] In step 2220, stop words refer to common conjunctions, prepositions, articles, etc., such as "de", "le", "shi", "zai", etc. Stop words do not represent specific meanings for the advertising demand information, so they need to be removed to improve the accuracy of subsequent semantic matching. Exemplarily, stop words can be identified through a pre-set stop word list.

[0376] In step 2230, non-entity words refer to words that do not have actual meanings. The multiple first keywords refer to the remaining words after removing the stop words and non-entity words from the words segmented from the advertising demand information.

[0377] The following will explain the process of extracting the first keywords from the advertising demand information in conjunction with Fig.23 . As Fig.23 shown, assume that there is advertising demand information corresponding to "The product is a TV box and needs to be placed in the segment where a family watches TV.". First, segment the advertising demand information, and the segmented words are: "product", "is", "TV box", ",", "needs", "place", "in", "a family", "watch", "TV", "de", "segment", "zhong", ".".

[0378] Identify and remove stop words from the separated words, and obtain the words after removing the stop words: "product", "TV box", "," "need", "delivery", "family", "watch", "TV", "clip", ".".

[0379] Among the words after removing the stop words, punctuation marks and non-entity words with no actual meaning are removed, and multiple first keywords are obtained: “TV box”, “delivery”, “family”, “watch”, and “TV”.

[0380] It should be noted that the above examples are only for the purpose of facilitating understanding of the process of extracting multiple first keywords from the advertising demand information, and the assumed examples do not represent limitations on actual implementation.

[0381] If the advertising demand information is directly matched with the first text description, the useless words contained therein may easily interfere with the matching result. In the above steps 2210 to 2230, the advertising demand information is preprocessed to remove useless stop words and non-entity words, thereby obtaining multiple first keywords. Matching the first text description based on multiple first keywords can effectively reduce noise and redundant information and improve the accuracy of matching.

[0382] In one embodiment, referring to Fig.24 , step 2020 includes:

[0383] Step 2410: for each first keyword, determine a first word frequency of the first keyword appearing in the first text description;

[0384] Step 2420: determining a first inverse document frequency of the first keyword appearing in a plurality of first text descriptions;

[0385] Step 2430: Determine a first similarity based on the first term frequency and the first inverse document frequency.

[0386] Steps 2410 to 2430 are described in detail below.

[0387] In step 2410, the first word frequency refers to the number of times the first keyword appears in the first text description. The first word frequency is used to measure the frequency of the first keyword appearing in the first text description. The higher the first word frequency, the more important the first keyword is in the first text description.

[0388] In step 2420, the first inverse document frequency is used to characterize the prevalence of the first keyword in the plurality of first text descriptions. The higher the first inverse document frequency, the fewer the corresponding first keywords appear in the plurality of first text descriptions, and similarly, the higher the importance of the first keyword.

[0389] In step 2430, the first similarity is the TF-IDF value of the corresponding first keyword. The first similarity is equal to the product of the first term frequency and the first inverse document frequency.

[0390] Exemplarily, assume that a certain first keyword appears three times in a certain first text description, and the first text description has a total of 30 words; at this time, there are a total of 20 first text descriptions, of which the first keyword appears in 4 first text descriptions.

[0391] Correspondingly, the first word frequency of the first keyword relative to the first text description is equal to the number of times the first keyword appears divided by the total number of words in the first text description, that is, the first word frequency=3 / 30=0.1.

[0392] The first inverse document frequency of the first keyword is equal to the total number of first text descriptions divided by the number of first text descriptions in which the first keyword appears, that is, the first inverse document frequency=20 / 4=5.

[0393] Finally, the first similarity of the first keyword is equal to the first word frequency multiplied by the first inverse document frequency, that is, the first similarity=0.1*5=0.5.

[0394] It should be noted that the above example is to illustrate the process of calculating the first similarity through assumed numerical values ​​for the purpose of facilitating understanding, and does not represent a limitation on actual implementation.

[0395] Compared with the general matching of keywords of the advertising demand information with tags extracted from the target video, in the above steps 2410 to 2430, the first similarity (TF-IDF value) of each first keyword of the advertising demand information is calculated as the basis for matching the advertising demand information with the first text description. Using the TF-IDF value as the first similarity can accurately identify the relevance of the first text description with the first keyword, and can distinguish those words that appear frequently in certain documents but are not important to the overall corpus.

[0396] In one embodiment, referring to Fig.25 , step 2410 includes:

[0397] Step 2510: for each first keyword, determine the number of first occurrences of the first keyword in the first text description;

[0398] Step 2520: extracting second keywords from the first text description, and determining the second keyword number of the second keywords;

[0399] Step 2530: Determine the first word frequency based on the first number and the second number of keywords.

[0400] Steps 2510 to 2530 are described in detail below.

[0401] In step 2510, the first number refers to the number of times the corresponding first keyword appears in a first text description.

[0402] In step 2520, the second keyword refers to a keyword extracted by performing data preprocessing on the first text description. The data preprocessing includes performing word segmentation on the first text description, identifying stop words, removing stop words, removing non-entity words, extracting stems, etc.

[0403] The number of second keywords refers to the total number of second keywords extracted from the first text description.

[0404] In step 2530, the first word frequency is equal to the first number divided by the second keyword number.

[0405] Since the first keyword is also a keyword extracted by preprocessing the advertisement demand information, the words in the first text description that are the same as the first keyword should also be extracted as the second keyword.

[0406] Combine the following Fig.26 , the process of determining the first word frequency is explained. Fig.26 As shown, it is assumed that first keyword 1 to first keyword n ​​are extracted from the advertisement demand information, and 30 second keywords (second keyword 1 to second keyword 30) are extracted from the first text description.

[0407] Take the first keyword 1 as an example: determine the number of times the first keyword 1 appears in the second keywords 1 to 30, assuming that the first keyword 1 appears 3 times; correspondingly, the first word frequency 1 of the first keyword 1=3 / 30=0.1.

[0408] Similarly, the first word frequencies 2 to n of the remaining first keywords 2 to first keywords n relative to the first text description are obtained.

[0409] If the first word frequency of the first keyword is determined directly based on the first text description, the first word frequency may be too small due to the presence of stop words and non-practical words in the first text description. In the above steps 2510 to 2530, keywords are extracted from the first text description, and the first word frequency is determined based on the extracted multiple second keywords and the first keyword. Unnecessary noise words in the first text description are removed, thereby improving the numerical accuracy of the first word frequency.

[0410] In one embodiment, referring to Fig. 27 , step 2420 includes:

[0411] Step 2710: Determine a first number of first text descriptions containing a first keyword;

[0412] Step 2720, obtaining the total number of first text descriptions;

[0413] Step 2730: Determine a first inverse document frequency based on the first number and the total number of first text descriptions.

[0414] Steps 2710 to 2730 are described in detail below.

[0415] In step 2710 , the first number refers to the number of first text descriptions that contain the first keyword in the plurality of first text descriptions.

[0416] In step 2720, the total number of first text descriptions refers to the number of first representations (first text descriptions) of each plot segment output by the segment representation prediction model during the current advertising placement information processing process.

[0417] In step 2730, the first inverse document frequency is equal to the total number of first text descriptions divided by the first number.

[0418] Below, reference Fig.28 , the process of determining the first inverse document frequency is explained. Fig.28 As shown, in a certain advertisement insertion information processing process, at least one target video to be inserted with advertisements is preprocessed to obtain multiple plot segments (segment 1 to segment n) and multiple text information (text 1 to text n). Then, segments 1 to segment n and texts 1 to text n are respectively input into the segment representation prediction model in a one-to-one correspondence to obtain the first text description (plot 1 to plot n) corresponding to each plot segment.

[0419] Assume that n is 20, that is, the total number of first text descriptions is 20 (plot 1 to plot 20).

[0420] Keyword extraction is performed on the advertisement demand information to obtain first keyword 1 to first keyword m. Take first keyword 1 as an example. Assume that first keyword 1 is currently included in plot 1, plot 3, plot 5, and plot 17, that is, the first number is 4. Then the first inverse document frequency of first keyword 1 = 20 / 4 = 5.

[0421] Similarly, the first inverse document frequencies 2 to the first inverse document frequencies m of the remaining first keywords 2 to the first keywords m are obtained.

[0422] If only the word frequency of the first keyword in each first text description is used as the basis for matching the advertising demand information with the first text description, since the first keyword may be a common word in the first text description, it may appear in multiple first text descriptions, and the matching result will not be accurate enough. In the above steps 2710 to 2730, by calculating the first inverse document frequency of the first keyword, the prevalence of the first keyword in multiple first text descriptions can be accurately measured. The first inverse document frequency of the first keyword is then combined with the first word frequency to obtain the first similarity between the first keyword and the first text description. Using the first similarity as the basis for matching the advertising demand information with the first text description can improve the accuracy of the matching.

[0423] In one embodiment, referring to Fig.29 , step 2710 includes:

[0424] Step 2910: extracting second keywords from each first text description;

[0425] Step 2920: using each extracted second keyword as a first index, correspondingly storing the first index and the first text description identifier of each first text description containing the second keyword into a first index comparison table;

[0426] Step 2930: Use the first keyword as the first index to search the first index comparison table to obtain the first text description identifier stored corresponding to the first index, thereby determining the first number of first text descriptions containing the first keyword.

[0427] Step 2920 and step 2930 are described in detail below.

[0428] In step 2920, the first index is an index value of a first text description identifier for representing at least one corresponding text description. In this embodiment, the specific value of the first index is the corresponding second keyword. The first index comparison table refers to a data storage structure composed of multiple first indexes and corresponding first text description identifiers.

[0429] The first text description identifier is an identifier for indicating the first text description. Exemplarily, the first text description identifier may be represented by a specific numerical value, or may be formed by a combination of one or more of letters, numbers, and symbols.

[0430] In step 2930, using the first keyword as the first index means setting the specific value of the first index to the first keyword. Thus, the same index is found in the first index comparison table, and the first text descriptions corresponding to the same index stored in the first index comparison table are obtained. The number of the first text descriptions found is the first number.

[0431] Below, reference Fig.30 , a process of determining a first number of first text descriptions containing a first keyword is explained.

[0432] like Fig.30 As shown, assuming that there are n first text descriptions, m second keywords (second keyword 1 to second keyword m) are extracted from first text description 1 to first text description n. Each second keyword is used as a first index, and first index 1 to first index m are obtained accordingly.

[0433] Assume that the first text description 1, the first text description 3, and the first text description 4 all contain the second keyword 1. Then, a mapping relationship between the first index 1 and the first text description identifier 1, the first text description identifier 3, and the first text description identifier 4 is constructed and stored in the first index comparison table. Similarly, at least one first text description identifier corresponding to each of the remaining first indexes 2 to the first index m is obtained, and finally a completed first index comparison table is obtained.

[0434] The first keyword 1 is used as the first index 1_1, and the first index comparison table is looked up. Assuming that the first index 1_1 is the same as the first index 2, the first text description identifier 6 and the first text description identifier 8 corresponding to the first index 2 are obtained based on the mapping relationship. It is thus determined that the first text description 6 and the first text description 8 include the first keyword 1, and correspondingly, the first number of the first text descriptions containing the first keyword is 2.

[0435] If each first keyword is compared with each first text description one by one to determine the number of first text descriptions containing the first keyword, the query efficiency will be low. In the above steps 2910 to 2930, a first index comparison table with an inverted index structure is constructed. Each first keyword only needs to be compared with each first index of the first index comparison table to quickly query the first text description identifiers of all first text descriptions containing the first keyword, thereby quickly determining the first number. Based on this, the matching efficiency between the advertising demand information and the first text description can be improved.

[0436] In one embodiment, the first representation is a first abstract feature of the plot segment. Fig.31 , step 450 comprises:

[0437] Step 3110: input the advertisement demand information into the segment representation prediction model to obtain the second abstract feature of the advertisement demand information;

[0438] Step 3120: Calculate a first matching degree based on the second abstract feature and the first abstract feature of each plot segment.

[0439] Step 3110 and step 3120 are described in detail below.

[0440] In step 3110, the second abstract feature refers to a feature vector obtained after feature extraction of the advertisement demand information by a segment representation prediction model.

[0441] In step 3120, the first abstract feature refers to a first representation in the form of a feature vector obtained by the segment representation prediction model based on the plot segment and the corresponding text information.

[0442] There is a first matching degree between the advertisement demand information and each plot segment. The first matching degree refers to a value calculated based on the first abstract feature of a plot segment and the second abstract feature of the advertisement demand information, and is used to characterize the matching degree between the advertisement demand information and the plot segment.

[0443] Exemplarily, the first matching degree may be obtained by calculating the cosine similarity between the first abstract feature and the second abstract feature. In other examples, the first matching degree may also be obtained by calculating the Euclidean distance between the first abstract feature and the second abstract feature.

[0444] Combine the following Fig.32 , the process of calculating the first matching degree based on the first abstract feature and the second abstract feature is explained. Fig.32 As shown, each target video that needs to be placed for advertising is preprocessed. The preprocessing includes: segmenting each target video to obtain multiple plot segments, such as segment 1 to segment n in the figure; performing text extraction on each target video to obtain text information containing line information, and segmenting the text information to obtain text 1 to text n corresponding to segment 1 to segment n. In other examples, text extraction can also be performed on each plot segment to obtain text information corresponding to each plot segment.

[0445] Segments 1 to n are respectively input into the segment representation prediction model with corresponding texts 1 to text n in pairs. The segment representation prediction model outputs the first representation corresponding to each segment, such as the first abstract feature 1 to the first abstract feature n corresponding to segments 1 to n in the figure.

[0446] The advertising demand information is input into the segment representation prediction model to obtain the corresponding second abstract feature. The second abstract feature is respectively calculated for similarity with the first abstract feature 1 to the first abstract feature n to obtain the first matching degree 1 to the first matching degree n. The similarity calculation may be to calculate the cosine similarity between the first abstract feature and the second abstract feature, or to calculate the Euclidean distance between the first abstract feature and the second abstract feature.

[0447] Finally, based on the first matching degree, at least one plot segment matching the advertisement demand information is selected as the target segment.

[0448] In other examples, each plot segment may also be input into the segment representation prediction model. The segment representation prediction model performs operations such as representative frame selection, representative frame feature vector extraction, feature mode conversion, etc. on each plot segment to obtain the first feature of each plot segment (refer to the implementation details corresponding to step 1410 and step 1420 provided in the above embodiment).

[0449] The advertising demand information is then input into the segment representation prediction model to obtain the corresponding second abstract feature. The second abstract feature is respectively calculated for similarity with the first feature of each plot segment to obtain the first matching degree between the advertising demand information and each plot segment.

[0450] In the above steps 3110 and 3120, the first matching degree is calculated by calculating the feature vector corresponding to the plot segment and the feature vector corresponding to the advertising demand information. Compared with calculating the similarity between texts by using the first text description corresponding to the plot segment and the advertising demand information, the method of directly calculating the first matching degree based on the cross-modal features can cover some detailed visual information ignored by the text description and recall more possible target segments.

[0451] In one embodiment, referring to Fig.33 , step 450 comprises:

[0452] Step 3310: input the storyboard segment into the segment representation prediction model to obtain a second representation representing the storyboard segment;

[0453] Step 3320: Determine a second matching degree between the advertisement demand information and the second representation of each storyboard segment;

[0454] Step 3330: Determine a target segment from among multiple plot segments and multiple storyboard segments based on the first matching degree and the second matching degree.

[0455] Steps 3310 to 3330 are described in detail below.

[0456] In step 3310, the second representation refers to information output by analyzing and predicting the input storyboard segment through the segment representation prediction model, which is used to characterize the detailed description of the content corresponding to the storyboard segment, that is, the semantic information corresponding to the storyboard segment mentioned above. Exemplarily, the second representation can be a text description composed of natural language, or an abstract expression composed of feature vectors.

[0457] In step 3320, the second matching degree refers to the semantic similarity between the advertising demand information and the second representation of the storyboard segment. The higher the similarity, the higher the second matching degree.

[0458] In step 3330, the corresponding plot segments that meet the first matching degree or the corresponding storyboard segments that meet the second matching degree are used as target segments. The target segments may include only plot segments that meet the requirements, or only storyboard segments that meet the requirements, or both storyboard segments that meet the requirements and plot segments that meet the requirements.

[0459] For example, a plot segment with a first matching degree greater than a preset first matching degree threshold can be used as a target segment, and a storyboard segment with a second matching degree greater than a preset second matching degree threshold can be used as a target segment. The first matching degree threshold and the second matching degree threshold can be the same or different, and are set according to actual implementation conditions.

[0460] For example, the plot segments are sorted from large to small according to the first matching degree, and the plot segments with a high ranking, such as the top 5%, are selected as the first candidate target segments. Then, the storyboard segments are sorted from large to small according to the second matching degree, and the storyboard segments with a high ranking, such as the top 5%, are selected as the second candidate target segments. The first candidate target segment and the second candidate target segment are integrated to obtain the final target candidate segment. Among them, the selection percentage can be set according to actual needs, and the selection percentages corresponding to the storyboard segments and the plot segments can be the same or different.

[0461] For example, the plot segments and the storyboard segments are sorted from large to small according to the corresponding first matching degree and second matching degree to obtain a candidate segment sequence, and the first 10% of the segments are selected from the candidate segment sequence as target segments.

[0462] Exemplarily, the plot segment with the largest first matching degree and the storyboard segment with the largest second matching degree are selected as target segments.

[0463] Combine the following Fig.34 , explaining the process of determining the target clip among multiple plot clips and multiple storyboard clips.

[0464] like Fig.34 As shown, each target video that needs to be used for advertising delivery is pre-processed. The pre-processing includes:

[0465] Each target video is segmented to obtain multiple plot segments, such as segment 1 to segment n in the figure;

[0466] Perform storyboard processing on each target video to obtain multiple storyboard segments, such as storyboard 1 to storyboard m in the figure;

[0467] Text extraction is performed on each target video to obtain text information including dialogue information, and the text information is segmented to obtain text 1 to text n corresponding to segments 1 to segment n. In other examples, text extraction can be performed on each plot segment to obtain text information corresponding to each plot segment.

[0468] Segments 1 to segment n are input into the segment representation prediction model in pairs with the corresponding texts 1 to text n. The segment representation prediction model outputs the first representation corresponding to each plot segment, such as plot summary 1 to plot summary n corresponding to segments 1 to segment n in the figure.

[0469] Input the segment representation prediction model from segment 1 to segment m. The segment representation prediction model outputs the second representation corresponding to each segment segment, such as segment details 1 to segment details m corresponding to segment 1 to segment m in the figure.

[0470] In other examples, text extraction processing may be performed on each storyboard segment to obtain text information corresponding to each storyboard segment. Storyboards 1 to storyboard m and the corresponding text information are input into the segment representation prediction model in pairs to obtain storyboard details 1 to storyboard details m (not shown in the figure).

[0471] The advertisement demand information is matched with the plot summary 1 to the plot summary n and the storyboard 1 to the storyboard m to obtain the target segment.

[0472] In the related art, text extraction and video feature extraction are performed on the video respectively to obtain text content and video feature content, and then the text content and video feature content are labeled to obtain various labels. Compared with matching video clips that can be used for advertising implantation based on labels, in the above steps 3310 to 3330, the target video is segmented in the plot dimension and the storyboard dimension to obtain multiple plot clips and multiple storyboard clips. Then, the first representation of each of the multiple plot clips and the second representation of each of the multiple storyboard clips are obtained through the clip representation prediction model. Finally, based on the semantics corresponding to the first representation and the semantics corresponding to the second representation, the clips are matched with the advertising demand information to obtain the target clip. Matching based on semantics from different dimensions can more accurately match the target clip corresponding to the advertising demand information.

[0473] In one embodiment, the segment representation prediction model is a multimodal large model. Fig.35 , step 3310 includes:

[0474] Step 3510: Input the storyboard segment and the second prompt into the segment representation prediction model to obtain a second representation representing the storyboard segment.

[0475] Step 3510 is described in detail below.

[0476] In step 3510, the second prompt refers to text information input by the user for indicating that the segment representation prediction model outputs a second representation of the storyboard segment that meets the user's needs. In some examples, the second prompt may be a pre-set templated prompt. In other examples, the second prompt may also be a prompt input by the user in real time based on the storyboard segment.

[0477] For example, the second prompt is "This is a shot in a film or TV series. Please describe the picture in detail, including the characters' clothing, actions, expressions, and positions, and describe the foreground, background, and other important elements of the picture." At this time, the segment representation prediction model will output a detailed description of the corresponding storyboard segment as the second representation under the guidance of the second prompt.

[0478] Combine the following Fig.36 , the process of obtaining the second representation by inputting the storyboard fragment and the second prompt into the multimodal large model is explained. Fig.36 As shown, the storyboard fragments are as follows Fig.36 The input screen is shown in the image in the column.

[0479] Correspondingly, the second prompt is "This is a shot in a film or TV series. Please describe the scene in detail, including the clothes, actions, expressions and positions of the characters in the picture, and describe the foreground, background and other important elements of the picture." Fig.36 As shown in the prompt column.

[0480] The storyboard clips and the prompts are input into the multimodal big model. Based on the second prompt, the multimodal big model determines that the specific content it needs to output should be a detailed description of the storyboard clips, and needs details such as clothing, actions, expressions, positions, foregrounds, and backgrounds. The multimodal big model extracts and analyzes the features of the storyboard clips and the second prompts to obtain the semantic features corresponding to the image data and text data, and generates the target content based on the semantic features. The detailed description of the final output storyboard clip is: "The picture shows the screen of a mobile phone, and the screen displays a photo being taken. There are two people in the photo: the person on the left is a man, wearing a dark suit, a white shirt, and a smile. His right hand hangs naturally, and his left hand is placed on the shoulder of the woman next to him, appearing intimate and friendly. The person on the right is a woman, wearing a dark dress, with her left hand on the waist of the man next to her, appearing intimate and friendly. The background is an indoor environment, which may be a restaurant or a family gathering place. There are dining tables and chairs in the background, with some tableware and food on the table, which looks warm and lively. There are some decorations on the walls in the background, which increase the beauty of the environment. The hands of the photographer can be seen in the foreground, and his fingers are operating the mobile phone to take pictures."

[0481] In the related technologies, different models are used to extract features from plot segments and convert them into corresponding labels. Fig.36 After extracting the storyboard fragments, specific labels such as hand, mobile phone, camera, etc. can be obtained. However, this does not allow the complete semantics of the plot fragments, which is not conducive to subsequent matching with advertising demand information. In the above step 3510, the multimodal large model can be used to directly process the image data of the storyboard fragments and the text data of the second prompt. And the multimodal large model is taught by the second prompt so that the multimodal large model can accurately output the second representation of the storyboard fragment required by the user. Compared with the related art that extracts a single label from the storyboard fragment, this embodiment uses the second prompt to guide the multimodal large model to perform high-level semantic extraction of the storyboard fragment in the dimension of the storyboard, and obtains a second representation containing detailed semantic information of the storyboard fragment. Therefore, when the second representation is used to match the advertising demand information in the future, the matching accuracy can be improved, and a target fragment that is more adapted to the advertising demand information can be obtained.

[0482] In one embodiment, referring to Fig.37 , step 3510 includes:

[0483] Step 3710: extract multiple second representative frames from the storyboard segments, and generate second representative frame vectors based on the second representative frames;

[0484] Step 3720: convert the plurality of second proxy frame vectors into third features suitable for input into the multimodal large model;

[0485] Step 3730: Based on the second prompt, generate a fourth feature suitable for input into the multimodal large model;

[0486] Step 3740: Input the third feature and the fourth feature into the multimodal large model to obtain a second representation representing the storyboard segment.

[0487] Steps 3710 to 3730 are described in detail below.

[0488] In step 3710, the second representative frame refers to a video frame extracted from the storyboard segment. Exemplarily, N video frames are uniformly extracted from the storyboard segment as N second representative frames, where N is a natural number. In another example, the storyboard segment can be preliminarily divided by hierarchical clustering, and then the k-means clustering algorithm is combined to perform histogram feature comparison on the preliminarily extracted key frames to remove redundant frames, and finally determine the second representative frame sequence of the storyboard segment.

[0489] The second representative frame vector is a vector obtained by extracting features from the second representative frame. For example, multiple second representative frames can be converted into multiple d-dimensional feature vectors, i.e., multiple second representative frame vectors, through a video image feature model, such as Contrastive Language-Image Pre-training (CLIP).

[0490] In step 3720, the third feature can be obtained by converting the second representative frame vector through the cross-modal matching module. The third feature is the abstract feature data corresponding to the storyboard segment that can be understood by the multi-modal large model.

[0491] Exemplarily, the cross-modal matching module may adopt a transformer-based network or a multilayer perceptron (MLP).

[0492] In step 3730, the fourth feature refers to feature data that can be understood by the multimodal large model, obtained by extracting features from the second prompt.

[0493] Combine the following Fig.38 , the process of obtaining the second representation of the storyboard fragment by using the storyboard fragment and the second prompt through the multimodal large model is explained.

[0494] like Fig.38 As shown, a pair of storyboard segments and corresponding second prompts are taken as an example. First, multiple video frames are uniformly extracted from the storyboard segments as second representative frames. The multiple second representative frames are input into the video image feature model for feature extraction to obtain multiple second representative frame vectors. The multiple second representative frame vectors are input into the cross-modal matching module for modal conversion, and the semantic mapping of the multiple second representative frame vectors is completed to obtain a more compact third feature.

[0495] While processing the second representative frame, feature extraction is performed based on the second prompt to obtain a fourth feature suitable for input into the multimodal large model.

[0496] The third feature and the fourth feature are input into the multimodal large model. The multimodal large model can combine the semantics corresponding to the storyboard segment and the semantics corresponding to the second prompt, perform high-level semantic extraction on the detailed content of the storyboard segment, and obtain the second representation of the storyboard segment.

[0497] In the related art, multiple models are trained separately on subtasks, and different labels are extracted from the target video based on different models to match the advertising demand information. There are problems of weak generalization ability and insufficient understanding of the plot of the target video. In the above steps 3710 to 3740, the multimodal large model is used to combine the input storyboard fragment and the second prompt to output the second representation of the storyboard fragment. Since the multimodal large model has a strong semantic understanding ability, under the guidance of the second prompt, it can better understand the details of the characters, clothing, actions, expressions, positions, foregrounds, backgrounds, etc. in the storyboard fragment, and can have a deeper understanding of the semantics of the storyboard fragment. Therefore, the second representation output by the multimodal large model can fully represent the deep semantics of the corresponding storyboard fragment. Matching with the advertising demand information based on such a second representation is conducive to finding a target fragment that better meets the advertising demand information and improves the effect of advertising implantation.

[0498] In one embodiment, the second representation is a second text description of the storyboard segment; Fig.39 , step 3320 includes:

[0499] Step 3910: extract multiple first keywords from the advertisement demand information;

[0500] Step 3920: for each first keyword, determine a second similarity between the first keyword and the second text description;

[0501] Step 3930: Determine a second matching degree between the advertisement demand information and the second text description based on a second similarity between each first keyword and the second text description.

[0502] Step 3920 and step 3930 are described in detail below.

[0503] In step 3920, the second similarity refers to the TF-IDF value of the first keyword relative to the second text description. The second similarity is used to characterize the importance of the corresponding first keyword to the second text description.

[0504] In step 3930, the second matching degree is calculated based on the TF-IDF value of each first keyword in the advertisement demand information, and is a value used to represent the matching degree between the advertisement demand information and the second text description.

[0505] Combine the following Fig.40 , the process of determining the second matching degree is explained. Fig.40As shown, first keyword 1 to first keyword n ​​are extracted from the advertising demand information. The second similarity between each first keyword and the second text description is calculated to obtain second similarity 1 to second similarity n respectively. The second similarity in this example is the TF-IDF value of the first keyword relative to the second text description.

[0506] The first keyword 1 to the first keyword n ​​and the corresponding second similarities 1 to the second similarities n of the advertising demand information are converted into a vector form. Each second similarity constitutes a dimension, and the second similarities 1 to the second similarities n are combined into a document vector of the advertising demand information.

[0507] The cosine similarity between the document vector of the advertising demand information and the document vector of the second text description is calculated, and the cosine similarity is used as the second matching degree between the advertising demand information and the second text description. The second matching degree ranges from [-1, 1], and the closer the value is to 1, the more matching the advertising demand information and the second text description are.

[0508] In the related art, by labeling the target video and matching it with the advertising demand information based on each label, there are problems of insufficient flexibility and poor matching accuracy. In the above steps 3910 to 3930, the similarity at the semantic level is calculated by using the document vector of the second text description that can characterize the detailed semantics of the storyboard clip and the document vector of the advertising demand information. When matching, more semantic details can be taken into account, thereby improving the accuracy of the matching. And because the matching is based on semantics rather than fixed labels, the matching flexibility is higher.

[0509] In one embodiment, referring to Fig.41 , step 3910 includes:

[0510] Step 4110: segment the advertisement demand information;

[0511] Step 4120, identifying stop words in the separated words and removing the stop words;

[0512] Step 4130: After removing the stop words, remove non-entity words and obtain multiple first keywords.

[0513] It should be noted that the feature explanations, implementation details and beneficial effects of steps 4110 to 4130 in this embodiment correspond to those of steps 2210 to 2230 in the above embodiment and are not elaborated here.

[0514] In one embodiment, referring to Fig.42 , step 3920 includes:

[0515] Step 4210: for each first keyword, determine a second word frequency of the first keyword appearing in the second text description;

[0516] Step 4220: Determine a second inverse document frequency of the first keyword appearing in a plurality of second text descriptions;

[0517] Step 4230: Determine a second similarity based on the second term frequency and the second inverse document frequency.

[0518] Steps 4210 to 4230 are described in detail below.

[0519] In step 4210, the second word frequency refers to the number of times the first keyword appears in the second text description. The second word frequency is used to measure the frequency of the first keyword appearing in the second text description. The higher the second word frequency, the more important the first keyword is in the second text description.

[0520] In step 4220, the second inverse document frequency is used to characterize the prevalence of the first keyword in the plurality of second text descriptions. The higher the second inverse document frequency, the fewer the corresponding first keywords appear in the plurality of second text descriptions, and correspondingly, the higher the importance of the second keyword.

[0521] In step 4230, the second similarity is the TF-IDF value of the corresponding first keyword relative to the second text description. The second similarity is equal to the product of the second term frequency and the second inverse document frequency.

[0522] For example, assume that a first keyword appears 6 times in a second text description, and the second text description has 25 words in total; at this time, there are 15 second text descriptions in total, and the first keyword appears in 3 of them.

[0523] Correspondingly, the second word frequency of the first keyword relative to the second text description is equal to the number of times the first keyword appears divided by the total number of words in the second text description, that is, the second word frequency=6 / 25=0.24.

[0524] The second inverse document frequency of the first keyword is equal to the total number of second text descriptions divided by the number of second text descriptions in which the first keyword appears, that is, the second inverse document frequency=15 / 3=5.

[0525] Finally, the second similarity of the first keyword is equal to the second term frequency multiplied by the second inverse document frequency, that is, the second similarity=0.24*5=1.2.

[0526] It should be noted that the above example is to illustrate the process of calculating the second similarity through assumed numerical values ​​for the purpose of facilitating understanding, and does not represent a limitation on actual implementation.

[0527] Compared with the general matching of keywords of the advertising demand information with tags extracted from the target video. In the above steps 4210 to 4230, the second similarity of each first keyword of the advertising demand information is calculated as the basis for matching the advertising demand information with the second text description. Using the TF-IDF value as the second similarity, the relevance of the second text description with the first keyword can be accurately identified, and words that appear frequently in certain documents but are not important to the overall corpus can be distinguished.

[0528] In one embodiment, referring to Fig.43 , step 4210 includes:

[0529] Step 4310: for each first keyword, determine a second number of times the first keyword appears in the second text description;

[0530] Step 4320: extract the third keyword from the second text description, and determine the third keyword number of the third keyword;

[0531] Step 4330: Determine the second word frequency based on the second number and the third keyword number.

[0532] Steps 4310 to 4330 are described in detail below.

[0533] In step 4310, the second number refers to the number of times the corresponding first keyword appears in a second text description.

[0534] In step 4320, the third keyword refers to a keyword extracted by performing data preprocessing on the second text description. The data preprocessing includes performing word segmentation on the second text description, identifying stop words, removing stop words, removing non-entity words, extracting stems, etc.

[0535] The number of third keywords refers to the total number of third keywords extracted from the second text description.

[0536] In step 4330, the second word frequency is equal to the second number divided by the third keyword number.

[0537] Since the first keyword is also a keyword extracted by data preprocessing of the advertisement demand information, the words in the second text description that are the same as the first keyword should also be extracted as the third keyword.

[0538] Next, combine Fig.44 , the process of determining the second word frequency is explained. Fig.44 As shown, it is assumed that first keyword 1 to first keyword n ​​are extracted from the advertisement demand information, and 25 third keywords (third keyword 1 to third keyword 25) are extracted from the second text description.

[0539] Take the first keyword 1 as an example: determine the number of words in the third keywords 1 to the third keywords 25 that are the same as the first keyword 1 as the second count. Assuming that the third keywords 3, 4, and 9 are the same as the first keyword 1, the second count is 3; correspondingly, the second word frequency of the first keyword 1 is 1=3 / 25=0.12.

[0540] Similarly, the second word frequencies 2 to n of the remaining first keywords 2 to first keywords n relative to the second text description are obtained.

[0541] In the above steps 4310 to 4330, keywords are extracted from the second text description, and the second word frequency is determined based on the extracted third keywords and the first keyword. Compared with directly determining the second word frequency of the first keyword based on the second text description, this embodiment removes unnecessary noise words in the second text description, thereby improving the numerical accuracy of the second word frequency.

[0542] In one embodiment, referring to Fig.45 , step 4220 includes:

[0543] Step 4510, determining a second number of second text descriptions containing the first keyword;

[0544] Step 4520, obtaining the total number of second text descriptions;

[0545] Step 4530: Determine a second inverse document frequency based on the second number and the total number of second text descriptions.

[0546] Steps 4510 to 4530 are described in detail below.

[0547] In step 4510 , the second number refers to the number of second text descriptions that contain the first keyword in the plurality of second text descriptions.

[0548] In step 4520, the total number of second text descriptions refers to the number of second representations (second text descriptions) of each storyboard segment output by the segment representation prediction model during the current advertising placement information processing process.

[0549] In step 4530, the second inverse document frequency is equal to the total number of second text descriptions divided by the second number.

[0550] Below, refer to Fig.46 , the process of determining the second inverse document frequency is explained. Fig.46As shown, during a certain advertisement placement information processing process, at least one target video to be placed in the advertisement is storyboarded to obtain a plurality of storyboard segments (storyboard 1 to storyboard m). Storyboards 1 to storyboard m are then input into the segment representation prediction model to obtain second text descriptions corresponding to each storyboard segment (storyboard details 1 to storyboard details m).

[0551] Assume that m is 15, that is, the total number of second text descriptions is 15 (storyboard description 1 to storyboard description m, m=15).

[0552] Keyword extraction is performed on the advertisement demand information to obtain first keyword 1 to first keyword n. Take first keyword 1 as an example. Assume that first keyword 1 is currently included in storyboard description 1, storyboard description 5, storyboard description 7, storyboard description 11, and storyboard description 12, that is, the second number is 5. Then the second inverse document frequency of first keyword 1 = 15 / 5 = 3.

[0553] Similarly, the second inverse document frequencies 2 to the second inverse document frequencies n of the remaining first keyword 2 to the first keyword n ​​are obtained.

[0554] In the above steps 4510 to 4530, by calculating the second inverse document frequency of the first keyword, the prevalence of the first keyword in multiple second text descriptions can be accurately measured. The second inverse document frequency of the first keyword is then combined with the second word frequency to obtain the second similarity between the first keyword and the second text description. Compared with only using the word frequency of the first keyword in each second text description as the basis for matching the advertising demand information with the second text description, this embodiment uses the second similarity that combines the second word frequency and the second inverse document frequency as the basis for matching the advertising demand information with the second text description, which can improve the accuracy of the matching.

[0555] In one embodiment, referring to Fig.47 , step 4510 includes:

[0556] Step 4710: extracting third keywords from each second text description;

[0557] Step 4720: using each extracted third keyword as a second index, correspondingly storing the second index and the second text description identifier of each second text description containing the third keyword into a second index comparison table;

[0558] Step 4730: Use the first keyword as the second index, search the second index comparison table, obtain the second text description identifier stored corresponding to the second index, and thereby determine the second number of second text descriptions containing the first keyword.

[0559] Step 4720 and step 4730 are described in detail below.

[0560] In step 4720, the second index is an index value of a second text description identifier for representing at least one corresponding second text description. In this embodiment, the specific value of the second index is the corresponding third keyword. The second index comparison table refers to a data storage structure composed of multiple second indexes and corresponding second text description identifiers.

[0561] The second text description identifier is an identifier for indicating the second text description. Exemplarily, the second text description identifier may be represented by a specific numerical value, or may be formed by a combination of one or more of letters, numbers, and symbols.

[0562] In step 4730, using the first keyword as the second index means setting the specific value of the second index to the first keyword. Thus, the same index is found in the second index comparison table, and the second text descriptions corresponding to the same index in the second index comparison table are obtained. The number of the second text descriptions found is the second number.

[0563] Below, refer to Fig.48 , a process of determining the second number by constructing an index is explained.

[0564] like Fig.48 As shown, assuming that there are n second text descriptions, m third keywords (third keyword 1 to third keyword m) are extracted from second text description 1 to second text description n. Each third keyword is used as a second index, and second index 1 to second index m are obtained accordingly.

[0565] Assuming that the second text description 1 and the second text description 4 both contain the third keyword 1, a mapping relationship between the second index 1 and the second text description identifier 1 and the second text description identifier 4 is constructed and stored in the second index comparison table. Similarly, at least one second text description identifier corresponding to each of the remaining second indexes 2 to the second index m is obtained, and finally a completed second index comparison table is obtained.

[0566] The first keyword 1 is used as the second index 1_2, and the second index comparison table is looked up. Assuming that the second index 1_2 is the same as the second index 3, the second text description identifier 16 to the second text description identifier 21 corresponding to the second index 3 are obtained based on the mapping relationship. It is thus determined that the second text descriptions 16 to the second text descriptions 21 all contain the first keyword 1, and correspondingly, the second number of second text descriptions containing the first keyword is 6.

[0567] In the above steps 4710 to 4730, a second index comparison table of an inverted index structure is constructed. Each first keyword only needs to be compared with each second index of the second index comparison table to quickly query the second text description identifier of all second text descriptions containing the first keyword, thereby quickly determining the second number. Compared to comparing each first keyword with the second text description one by one to determine the number of second text descriptions containing the first keyword, the method of determining the second number in this embodiment can improve the matching efficiency between the advertising demand information and the second text description.

[0568] In one embodiment, the second representation is a third abstract feature of the storyboard segment; Fig.49 , step 3320 includes:

[0569] Step 4910: input the advertisement demand information into the segment representation prediction model to obtain the second abstract feature of the advertisement demand information;

[0570] Step 4920: Calculate a second matching degree based on the second abstract feature and the third abstract feature of each storyboard segment.

[0571] Step 4910 and step 4920 are described in detail below.

[0572] In step 4910, the second abstract feature refers to a feature vector obtained after feature extraction of the advertising demand information by a segment representation prediction model.

[0573] In step 4920, the third abstract feature refers to a second representation in the form of a feature vector of the storyboard segment output by the segment representation prediction model.

[0574] There is a second matching degree between the advertisement demand information and each storyboard segment. The second matching degree is a value calculated based on the third abstract feature of a storyboard segment and the second abstract feature of the advertisement demand information, and is used to characterize the matching degree between the advertisement demand information and the storyboard segment.

[0575] Below, refer to Fig.50 , the process of calculating the second matching degree through the second abstract feature and the third abstract feature is explained. Fig.50 As shown, each target video that needs to be placed with advertisements is pre-processed. The pre-processing includes: performing storyboard processing on each target video to obtain multiple storyboard segments, such as storyboard 1 to storyboard m in the figure.

[0576] Each storyboard segment is input into the segment representation prediction model. The segment representation prediction model performs operations such as representative frame selection, representative frame feature vector extraction, feature mode conversion, etc. on each storyboard segment to obtain the third feature of each storyboard segment as the third abstract feature (refer to the implementation details corresponding to step 3710 and step 3720 provided in the above embodiment).

[0577] The advertising demand information is input into the segment representation prediction model to obtain the corresponding second abstract feature. The second abstract feature is respectively calculated for similarity with the third abstract feature 1 to the third abstract feature m to obtain the second matching degree 1 to the second matching degree m. The similarity calculation may be the calculation of the cosine similarity between the third abstract feature and the second abstract feature, or the calculation of the Euclidean distance between the third abstract feature and the second abstract feature.

[0578] Finally, based on the second matching degree, at least one storyboard segment matching the advertisement requirement information is selected as the target segment.

[0579] In other examples, each storyboard segment also corresponds to a second prompt. In this example, after obtaining storyboards 1 to m through preprocessing, storyboards 1 to m and corresponding second prompts 1 to second prompts m are input into the segment representation prediction model in pairs. The segment representation prediction model outputs the second representation corresponding to each storyboard segment, and the data form of the second representation is a feature vector that can be understood by the segment representation prediction model. Therefore, each second representation is used as the third abstract feature.

[0580] The advertisement demand information is then input into the segment representation prediction model to obtain the corresponding second abstract feature. The second abstract feature is respectively calculated for similarity with the second representation of each storyboard segment to obtain the second matching degree between the advertisement demand information and each storyboard segment.

[0581] In the above steps 4910 and 4920, the second matching degree is calculated by comparing the feature vector corresponding to the storyboard segment with the feature vector corresponding to the advertising demand information. Compared with calculating the similarity between the texts by comparing the second text description corresponding to the storyboard segment with the advertising demand information, the second matching degree is calculated directly based on the cross-modal features in this embodiment, which can cover some detailed visual information ignored by the text description and recall more possible target segments.

[0582] In one embodiment, referring to Fig.51 , step 3330 includes:

[0583] Step 5110: weighting the first matching degree according to the first weight value to obtain a first weighted matching degree, and weighting the second matching degree according to the second weight value to obtain a second weighted matching degree;

[0584] Step 5120: Determine the first weighted matching degree or the second weighted matching degree as the matching degree to be examined;

[0585] Step 5130: Determine the plot segment or storyboard segment whose matching degree satisfies the first condition as the target segment.

[0586] Steps 5110 to 5130 are described in detail below.

[0587] In step 5110, the first weight is set based on the importance of the plot segment compared to the storyboard segment, and is used to characterize the weight of the importance of the plot segment. The larger the first weight is, the greater the proportion of the plot segment in the target segment finally determined. The first weighted matching degree is the matching degree obtained by multiplying the first matching degree by the first weight.

[0588] The second weight is set based on the importance of the storyboard segment compared to the plot segment, and is used to characterize the importance of the storyboard segment. The larger the second weight is, the greater the proportion of the storyboard segment in the target segment finally determined. The second weighted matching degree is the matching degree obtained by multiplying the second matching degree by the second weight.

[0589] In step 5120, both the first weighted matching degree and the second weighted matching degree obtained in step 5110 can be used as the matching degree to be examined. The first weighted matching degree and the second weighted matching degree are all integrated to obtain a matching degree set to be examined.

[0590] In step 5130, illustratively, the first condition may be that the corresponding to-be-inspected matching degree is greater than a preset matching degree threshold. In this case, the plot segment or storyboard segment corresponding to the to-be-inspected matching degree greater than the matching degree threshold is determined as the target segment.

[0591] Exemplarily, the matching degrees to be examined in the matching degree set to be examined are sorted from large to small to obtain a matching degree sequence to be examined, and the first condition is the top 10% in the matching degree sequence to be examined. At this time, the plot segments or storyboard segments corresponding to the top 10% of the matching degrees to be examined in the matching degree sequence to be examined are determined as target segments.

[0592] For example, the first condition may also be the maximum matching degree among the matching degrees to be examined. In this case, the plot segment or storyboard segment corresponding to the maximum matching degree among the multiple matching degrees to be examined is determined as the target segment.

[0593] Different advertisements may have different preferences for the placement of plot segments or storyboard segments. In the above steps 5110 to 5130, by setting the first weight and the second weight for the first matching degree and the second matching degree respectively, the emphasis between the plot segment and the storyboard segment is also used as one of the conditions for determining the target segment. Compared with directly selecting the target segment based on the first matching degree and the second matching degree, this embodiment determines the final target segment based on the first weighted matching degree and the second weighted matching degree, which can improve the accuracy of target segment recognition, and the target segment finally matched can better meet the advertising needs or the preference for plot segments or storyboard segments in actual situations.

[0594] In one embodiment, referring to Fig.52 , step 3330 includes:

[0595] Step 5210: Determine the plot segment whose first matching degree satisfies the second condition as the target plot segment;

[0596] Step 5220: Determine the storyboard segment in the target plot segment whose second matching degree satisfies the third condition as the target segment.

[0597] Step 5210 and step 5220 are described in detail below.

[0598] In step 5210, the second condition may be that the first matching degree is greater than a preset threshold. The second condition may be the first matching degrees that rank in the top x% after sorting the first matching degrees from large to small, where x is a positive integer and can be set according to actual needs. The second condition may also be the maximum matching degree among the first matching degrees.

[0599] The target plot segment refers to a plot segment whose corresponding first matching degree satisfies the second condition.

[0600] In step 5220, the third condition is similar to the second condition and will not be elaborated here.

[0601] Below, refer to Fig.53 , the process of determining the storyboard segments as target segments from the plot segments is explained.

[0602] like Fig.53 As shown, each target video that needs to be used for advertising delivery is pre-processed. The pre-processing includes:

[0603] Perform storyboard processing on each target video to obtain multiple storyboard segments, such as storyboard 1 to storyboard m in the figure;

[0604] Based on m storyboard segments, the storyboard segments belonging to the same plot are aggregated to obtain multiple plot segments, such as segment 1 to segment n in the figure;

[0605] Text extraction is performed on each target video to obtain text information including dialogue information, and the text information is segmented to obtain text 1 to text n corresponding to segments 1 to segment n. In other examples, text extraction can be performed on each plot segment to obtain text information corresponding to each plot segment.

[0606] Segments 1 to segment n are input into the segment representation prediction model in pairs with the corresponding texts 1 to text n. The segment representation prediction model outputs the first representation corresponding to each plot segment, such as plot summary 1 to plot summary n corresponding to segments 1 to segment n in the figure.

[0607] Shots 1 to 1 are input into the segment representation prediction model. The segment representation prediction model outputs the second representation corresponding to each shot segment, such as shot details 1 to shot details m corresponding to shots 1 to 1 respectively in the figure. In other examples, text extraction processing may be performed on each shot segment to obtain text information corresponding to each shot segment. Shots 1 to 1 are input into the segment representation prediction model in pairs with the corresponding text information to obtain shot details 1 to shot details m (not shown in the figure).

[0608] The advertisement demand information is matched with plot summaries 1 to n to obtain first matching degrees 1 to n corresponding to segments 1 to n, respectively.

[0609] The advertisement demand information is matched with the storyboard descriptions 1 to m to obtain second matching degrees 1 to m corresponding to the storyboards 1 to m, respectively.

[0610] Assume that the second condition and the third condition in this example are both that the matching degree is greater than the preset threshold. Based on the first matching degree 1 to the first matching degree n, the target plot segment is determined from the n plot segments. Since the plot segments are obtained based on the aggregation of storyboards, the storyboard segments in the target plot segment are the storyboard segments from storyboard 1 to storyboard m. At this time, the second matching degree of the storyboard segments in the target plot segment has been determined. Based on the third condition, the storyboard segments whose corresponding second matching degree meets the third condition are selected from the storyboard segments in the target plot segment as the target segment.

[0611] In other examples, each target video may be segmented to obtain multiple plot segments, such as segments 1 to n in the figure. Then, segments 1 to n are respectively input into a segment representation prediction model in pairs with corresponding texts 1 to text n. The segment representation prediction model outputs the first representation corresponding to each plot segment, such as plot summary 1 to plot summary n corresponding to segments 1 to n in the figure.

[0612] Plot summaries 1 to n are matched with advertising demand information to obtain corresponding first matching degrees 1 to n. Based on the second condition and the first matching degree, a plot segment that meets the second condition is selected as a target plot segment.

[0613] After obtaining the target plot segment, the target plot segment is storyboarded to obtain a plurality of storyboard segments. Each storyboard segment is then input into the segment representation prediction model to obtain a storyboard description corresponding to each storyboard segment. Each storyboard description is matched with the advertising demand information to obtain a corresponding second matching degree. Based on the third condition and the second matching degree, a storyboard segment that meets the third condition is selected as the target segment.

[0614] In the above steps 5210 and 5220, the advertising demand information is first matched with the plot segments, and then the target plot segments that meet the conditions are matched at the storyboard level. Compared with matching directly based on plot segments, this embodiment can more accurately determine the target segments and improve the precision of advertising insertion through multi-level matching from large to small.

[0615] Description of the apparatus and device of the present disclosure

[0616] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the characterization of arrows, these steps are not necessarily executed in sequence according to the order of arrow characterization. Unless there is a clear description in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0617] Fig.54 The structure diagram of the advertisement placement information processing device 5400 provided in the embodiment of the present disclosure. The advertisement placement information processing device 5400 includes:

[0618] The storyboard segment acquisition unit 5410 is used to aggregate video frames belonging to the same storyboard in the target video into storyboard segments, wherein the target video is a video to be embedded with target advertising information;

[0619] The plot segment acquisition unit 5420 is used to aggregate the storyboard segments belonging to the same plot in the target video into a plot segment;

[0620] a segment representation prediction unit 5430, configured to input the plot segment into a segment representation prediction model to obtain a first representation representing the plot segment;

[0621] Advertisement demand acquisition unit 5440, used to acquire advertisement demand information;

[0622] a target segment matching unit 5450, configured to determine a first matching degree between the advertisement demand information and the first representation of each plot segment, and determine a target segment from among the plurality of plot segments based on the first matching degree;

[0623] The target advertisement embedding unit 5460 is used to embed target advertisement information in the target segment.

[0624] Optionally, the fragment representation prediction unit 5430 is specifically configured to:

[0625] From the target video, obtain the text information corresponding to the plot clip;

[0626] The plot segment and text information are input into the segment representation prediction model to obtain a first representation representing the plot segment.

[0627] Optionally, the fragment representation prediction unit 5430 is specifically configured to:

[0628] Use subtitle recognition models and speech recognition models to identify lines from plot clips;

[0629] Use the face recognition model to identify the character information corresponding to the lines in the plot clips;

[0630] The dialogue information and the character information corresponding to the dialogue information are integrated into text information.

[0631] Optionally, the fragment representation prediction unit 5430 is specifically configured to:

[0632] Using the subtitle recognition model, subtitle recognition is performed in the plot segment to obtain first probabilities corresponding to multiple candidate dialogue information respectively;

[0633] Using the speech recognition model, speech recognition is performed in the plot segment to obtain second probabilities corresponding to a plurality of candidate dialogue information respectively;

[0634] Based on the first probability and the second probability respectively corresponding to each piece of candidate speech information, speech information is determined from a plurality of candidate speech information.

[0635] Optionally, the fragment representation prediction unit 5430 is specifically configured to:

[0636] Obtaining a first weight of a subtitle recognition model and a second weight of a speech recognition model;

[0637] Determining a weighted sum of the first probability and the second probability based on the first weight and the second weight;

[0638] Based on the weighted sum, speech information is determined from among a plurality of candidate speech information.

[0639] Optionally, the segment representation prediction model is a multimodal large model;

[0640] The fragment representation prediction unit 5430 is specifically used for:

[0641] Combining the text information with the first prompt to obtain a first combined text;

[0642] The plot segment and the first combined text are input into the multimodal large model to obtain a first representation of the plot segment.

[0643] Optionally, the fragment representation prediction unit 5430 is specifically configured to:

[0644] Extracting a plurality of first representative frames from the plot clip, and generating a first representative frame vector based on the first representative frames;

[0645] Converting a plurality of first representative frame vectors into first features suitable for input into a multimodal large model;

[0646] Based on the first combined text, generating a second feature suitable for input into a multimodal large model;

[0647] The first feature and the second feature are input into the multimodal large model together to obtain a first representation representing the plot segment.

[0648] Optionally, the first representation is a first text description of the plot segment;

[0649] The target segment matching unit 5450 is specifically used for:

[0650] extracting a plurality of first keywords from the advertising demand information;

[0651] For each first keyword, determining a first similarity between the first keyword and the first text description;

[0652] Based on the first similarities between each first keyword and the first text description, a first matching degree between the advertisement demand information and the first text description is determined.

[0653] Optionally, the target segment matching unit 5450 is specifically configured to:

[0654] Segment the advertising demand information;

[0655] Identify stop words in the separated words and remove them;

[0656] Among the words after removing the stop words, non-entity words are removed to obtain multiple first keywords.

[0657] Optionally, the target segment matching unit 5450 is specifically configured to:

[0658] For each first keyword, determining a first word frequency of the first keyword appearing in the first text description;

[0659] determining a first inverse document frequency of occurrence of a first keyword in a plurality of first text descriptions;

[0660] Based on the first term frequency and the first inverse document frequency, a first similarity is determined.

[0661] Optionally, the target segment matching unit 5450 is specifically configured to:

[0662] For each first keyword, determining the number of first occurrences of the first keyword in the first text description;

[0663] extracting a second keyword from the first text description, and determining a second keyword number of the second keyword;

[0664] Based on the first number and the second number of keywords, a first word frequency is determined.

[0665] Optionally, the target segment matching unit 5450 is specifically configured to:

[0666] determining a first number of first text descriptions containing a first keyword;

[0667] Get the total number of first text descriptions;

[0668] Based on the first number and the first total number of text descriptions, a first inverse document frequency is determined.

[0669] Optionally, the target segment matching unit 5450 is specifically configured to:

[0670] extracting second keywords from each first text description;

[0671] Taking each extracted second keyword as a first index, the first index and the first text description identifier of each first text description containing the second keyword are stored in a first index comparison table in correspondence;

[0672] The first keyword is used as the first index, and the first index comparison table is searched to obtain the first text description identifier stored corresponding to the first index, thereby determining the first number of first text descriptions containing the first keyword.

[0673] Optionally, the first representation is a first abstract feature of the plot segment;

[0674] The target segment matching unit 5450 is specifically used for:

[0675] Inputting the advertisement demand information into the segment representation prediction model to obtain a second abstract feature of the advertisement demand information;

[0676] A first matching degree is calculated based on the second abstract feature and the first abstract feature of each plot segment.

[0677] Optionally, the fragment representation prediction unit 5430 is specifically configured to:

[0678] Inputting the storyboard segment into the segment representation prediction model to obtain a second representation representing the storyboard segment;

[0679] The target segment matching unit 5450 is specifically used for:

[0680] Determining a second matching degree between the advertisement demand information and the second representation of each storyboard segment;

[0681] Based on the first matching degree and the second matching degree, a target segment is determined from among the plurality of plot segments and the plurality of storyboard segments.

[0682] Optionally, the target segment matching unit 5450 is specifically configured to:

[0683] Weighting the first matching degree according to the first weight value to obtain a first weighted matching degree, and weighting the second matching degree according to the second weight value to obtain a second weighted matching degree;

[0684] Determining the first weighted matching degree or the second weighted matching degree as the matching degree to be examined;

[0685] The plot segment or storyboard segment whose matching degree to be examined satisfies the first condition is determined as the target segment.

[0686] Optionally, the target segment matching unit 5450 is specifically configured to:

[0687] Determine the plot segment whose first matching degree satisfies the second condition as the target plot segment;

[0688] The storyboard segment whose second matching degree in the target plot segment satisfies the third condition is determined as the target segment.

[0689] Optionally, the segment representation prediction model is a multimodal large model;

[0690] The fragment representation prediction unit 5430 is specifically used for:

[0691] The storyboard segment and the second prompt are input into the segment representation prediction model together to obtain a second representation representing the storyboard segment.

[0692] Optionally, the fragment representation prediction model is specifically used for:

[0693] Extracting a plurality of second representative frames from the storyboard fragment, and generating a second representative frame vector based on the second representative frames;

[0694] converting the plurality of second representative frame vectors into third features suitable for input into a multimodal large model;

[0695] Based on the second prompt, generating a fourth feature suitable for input into the multimodal large model;

[0696] The third feature and the fourth feature are input into the multimodal large model together to obtain a second representation representing the storyboard segment.

[0697] Optionally, the second representation is a second text description of the storyboard segment;

[0698] The target segment matching unit 5450 is specifically used for:

[0699] extracting a plurality of first keywords from the advertising demand information;

[0700] For each first keyword, determining a second similarity between the first keyword and the second text description;

[0701] Based on the second similarity between each first keyword and the second text description, a second matching degree between the advertisement demand information and the second text description is determined.

[0702] Optionally, the target segment matching unit 5450 is specifically configured to:

[0703] Segment the advertising demand information;

[0704] Identify stop words in the separated words and remove them;

[0705] Among the words after removing the stop words, non-entity words are removed to obtain multiple first keywords.

[0706] Optionally, the target segment matching unit 5450 is specifically configured to:

[0707] For each first keyword, determining a second word frequency of the first keyword appearing in the second text description;

[0708] determining a second inverse document frequency of occurrence of the first keyword in the plurality of second text descriptions;

[0709] Based on the second term frequency and the second inverse document frequency, a second similarity is determined.

[0710] Optionally, the target segment matching unit 5450 is specifically configured to:

[0711] For each first keyword, determining a second number of times the first keyword appears in the second text description;

[0712] extracting a third keyword from the second text description and determining a third keyword number of the third keyword;

[0713] Based on the second number and the third keyword number, a second word frequency is determined.

[0714] Optionally, the target segment matching unit 5450 is specifically configured to:

[0715] determining a second number of second text descriptions containing the first keyword;

[0716] Get the total number of second text descriptions;

[0717] Based on the second number and the second total number of text descriptions, a second inverse document frequency is determined.

[0718] Optionally, the target segment matching unit 5450 is specifically configured to:

[0719] extracting third keywords from each second text description;

[0720] Taking each extracted third keyword as a second index, the second index and the second text description identifier of each second text description containing the third keyword are stored in a second index comparison table in correspondence;

[0721] The first keyword is used as the second index, and the second index comparison table is searched to obtain the second text description identifier stored corresponding to the second index, thereby determining the second number of second text descriptions containing the first keyword.

[0722] Optionally, the second representation is a third abstract feature of the storyboard segment;

[0723] The target segment matching unit 5450 is specifically used for:

[0724] Inputting the advertisement demand information into the segment representation prediction model to obtain a second abstract feature of the advertisement demand information;

[0725] A second matching degree is calculated based on the second abstract feature and the third abstract feature of each storyboard segment.

[0726] Optionally, the fragment representation prediction unit 5430 is specifically configured to:

[0727] From the target video, identify public figures in the plot clips;

[0728] Obtain public figure information of public figures;

[0729] The plot segment and the public figure information are input into the segment representation prediction model to obtain a first representation representing the plot segment.

[0730] Optionally, the segment representation prediction model is a multimodal large model;

[0731] The fragment representation prediction unit 5430 is specifically used for:

[0732] Combining the public figure information with the second prompt to obtain a second combined text;

[0733] The plot segment and the second combined text are input into the multimodal large model together to obtain a first representation representing the plot segment.

[0734] Reference Fig.55 , Fig.55 The structural block diagram of the terminal part for implementing the advertising placement information processing method of the embodiment of the present disclosure includes: a radio frequency (RF) circuit 5510, a memory 5515, an input unit 5530, a display unit 5540, a sensor 5550, an audio circuit 5560, a wireless fidelity (WiFi) module 5570, a processor 5580, and a power supply 5590. Those skilled in the art can understand that Fig.55 The terminal structure shown does not constitute a limitation on the mobile phone or computer, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0735] RF circuit 5510 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information from the base station, it is sent to processor 5580 for processing; in addition, the designed uplink data is sent to the base station.

[0736] The memory 5515 may be used to store software programs and modules. The processor 5580 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 5515 .

[0737] The input unit 5530 may be used to receive input digital or character information and generate key signal input related to the setting and function control of the content terminal. Specifically, the input unit 5530 may include a touch panel 5531 and other input devices 5532.

[0738] The display unit 5540 may be used to display input information or provided information and various menus of the content terminal. The display unit 5540 may include a display panel 5541.

[0739] The audio circuit 5560, the speaker 5561, and the microphone 5562 can provide an audio interface.

[0740] In this embodiment, the processor 5580 included in the terminal can execute the advertisement implantation information processing method of the previous embodiment.

[0741] The terminals of the embodiments of the present disclosure include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The embodiments of the present disclosure can be applied to various scenarios, including but not limited to film and television drama playback, video websites, video software, live broadcast advertising insertion, media information processing, etc.

[0742] Fig.56 A block diagram of the structure of a portion of a server for implementing the advertising implantation information processing method of an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 5622 (for example, one or more processors) and a memory 5632, and one or more storage media 5630 (for example, one or more mass storage devices) storing application programs 5642 or data 5644. Among them, the memory 5632 and the storage medium 5630 can be short-term storage or permanent storage. The program stored in the storage medium 5630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 5622 can be configured to communicate with the storage medium 5630 and execute a series of instruction operations in the storage medium 5630 on the server.

[0743] The server may also include one or more power supplies 5626, one or more wired or wireless network interfaces 5650, one or more input and output interfaces 5658, and / or one or more operating systems 5641, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0744] The central processor 5622 in the server can be used to execute the advertisement implantation information processing method of the embodiment of the present disclosure.

[0745] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the advertising implantation information processing methods of the aforementioned embodiments.

[0746] The embodiment of the present disclosure also provides a computer program product, which includes a computer program. A processor of a computer device reads and executes the computer program, so that the computer device executes and implements the above-mentioned method for processing advertisement implantation information.

[0747] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar contents, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0748] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated content, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated content before and after is in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0749] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.

[0750] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0751] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0752] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0753] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0754] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0755] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A method for processing advertisement implantation information, characterized in that: include: Aggregate video frames belonging to the same storyboard in a target video into storyboard segments, wherein the target video is a video to be embedded with target advertising information; Aggregate the storyboard segments belonging to the same plot in the target video into a plot segment; Acquire text information corresponding to the plot segment from the target video; Combining the text information with a first prompt to obtain a first combined text, wherein the first prompt is text information for instructing the segment representation prediction model to output a first representation of the plot segment; Inputting the plot segment and the first combined text into the segment representation prediction model to obtain the first representation, wherein the segment representation prediction model is a multimodal large model; Obtain advertising demand information; Determining a first matching degree between the advertisement demand information and the first representation of each of the plot segments, and determining a target segment from among the plurality of plot segments based on the first matching degree; The target advertisement information is embedded in the target segment.

2. The method for processing advertisement implantation information according to claim 1, characterized in that: The step of obtaining text information corresponding to the plot segment from the target video includes: Using a subtitle recognition model and a speech recognition model, identifying dialogue information from the plot segment; Using a face recognition model, identifying character information corresponding to the dialogue information from the plot segment; The dialogue information and the character information corresponding to the dialogue information are integrated into the text information.

3. The method for processing advertisement implantation information according to claim 2, characterized in that: The step of using the subtitle recognition model and the speech recognition model to identify the line information from the plot segment includes: Using the subtitle recognition model, performing subtitle recognition in the plot segment to obtain first probabilities corresponding to a plurality of candidate dialogue information respectively; Using the speech recognition model, performing speech recognition in the plot segment to obtain second probabilities corresponding to a plurality of candidate dialogue information respectively; The speech information is determined from the plurality of candidate speech information based on the first probability and the second probability respectively corresponding to each of the candidate speech information.

4. The method for processing advertisement implantation information according to claim 3, characterized in that: The determining the speech information from the plurality of candidate speech information based on the first probability and the second probability respectively corresponding to each of the candidate speech information includes: Obtaining a first weight of the subtitle recognition model and a second weight of the speech recognition model; Determining a weighted sum of the first probability and the second probability based on the first weight and the second weight; Based on the weighted sum, the speech information is determined from among the plurality of candidate speech information.

5. The method for processing advertisement implantation information according to claim 1, characterized in that: The step of inputting the plot segment and the first combined text into the segment representation prediction model to obtain the first representation includes: Extracting a plurality of first representative frames from the plot segment, and generating a first representative frame vector based on the first representative frames; converting a plurality of said first representative frame vectors into first features suitable for inputting into said multimodal macro model; Based on the first combined text, generating a second feature suitable for input into the multimodal large model; The first feature and the second feature are input into the multimodal large model to obtain the first representation.

6. The method for processing advertisement implantation information according to claim 1, characterized in that: The first representation is a first text description of the plot segment; The determining a first matching degree between the advertisement demand information and the first representation of each of the plot segments includes: extracting a plurality of first keywords from the advertising demand information; For each of the first keywords, determining a first similarity between the first keyword and the first text description; The first matching degree between the advertisement demand information and the first text description is determined based on the first similarity between each of the first keywords and the first text description.

7. The method for processing advertisement implantation information according to claim 6, characterized in that: The extracting a plurality of first keywords from the advertisement demand information includes: Segmenting the advertising demand information; Identifying stop words in the separated words and removing the stop words; Among the words after removing the stop words, non-entity words are removed to obtain the multiple first keywords.

8. The method for processing advertisement implantation information according to claim 6, characterized in that: The step of determining, for each of the first keywords, a first similarity between the first keyword and the first text description includes: For each of the first keywords, determining a first word frequency of the first keyword appearing in the first text description; determining a first inverse document frequency of occurrence of the first keyword in a plurality of the first text descriptions; The first similarity is determined based on the first term frequency and the first inverse document frequency.

9. The method for processing advertisement implantation information according to claim 8, characterized in that: The step of determining, for each of the first keywords, a first word frequency of the first keyword appearing in the first text description includes: For each of the first keywords, determining the number of first occurrences of the first keyword in the first text description; extracting second keywords from the first text description, and determining a second keyword number of the second keywords; The first word frequency is determined based on the first number of times and the second number of keywords.

10. The method for processing advertisement implantation information according to claim 8, characterized in that: The determining a first inverse document frequency of the first keyword appearing in the plurality of the first text descriptions comprises: determining a first number of the first text descriptions that include the first keyword; Get the total number of first text descriptions; The first inverse document frequency is determined based on the first number and the first total number of text descriptions.

11. The method for processing advertisement implantation information according to claim 10, characterized in that: The determining a first number of the first text descriptions containing the first keyword includes: extracting second keywords from each of the first text descriptions; Taking each of the extracted second keywords as a first index, the first index and the first text description identifier of each of the first text descriptions containing the second keyword are stored in a first index comparison table in correspondence; The first index comparison table is searched using the first keyword as the first index to obtain the first text description identifier stored corresponding to the first index, thereby determining the first number of the first text descriptions containing the first keyword.

12. The method for processing advertisement implantation information according to claim 1, characterized in that: The first representation is a first abstract feature of the plot segment; The determining a first matching degree between the advertisement demand information and the first representation of each of the plot segments includes: Inputting the advertisement demand information into the segment representation prediction model to obtain a second abstract feature of the advertisement demand information; The first matching degree is calculated based on the second abstract feature and the first abstract feature of each of the plot segments.

13. The method for processing advertisement implantation information according to claim 1, characterized in that: The step of determining a target segment from the plurality of plot segments based on the first matching degree includes: Inputting the storyboard segment into the segment representation prediction model to obtain a second representation representing the storyboard segment; Determining a second matching degree between the advertisement demand information and the second representation of each of the storyboard segments; Based on the first matching degree and the second matching degree, the target segment is determined from among the plurality of plot segments and the plurality of storyboard segments.

14. The method for processing advertisement implantation information according to claim 13, characterized in that: The step of determining the target segment from the plurality of plot segments and the plurality of storyboard segments based on the first matching degree and the second matching degree comprises: Weighting the first matching degree according to a first weight value to obtain a first weighted matching degree, and weighting the second matching degree according to a second weight value to obtain a second weighted matching degree; Determining the first weighted matching degree or the second weighted matching degree as the matching degree to be examined; The plot segment or the storyboard segment whose matching degree to be examined satisfies the first condition is determined as the target segment.

15. An advertisement placement information processing device, characterized in that: The device comprises: A storyboard segment acquisition unit, used for aggregating video frames belonging to the same storyboard in a target video into storyboard segments, wherein the target video is a video to be embedded with target advertising information; A plot segment acquisition unit, used for aggregating the storyboard segments belonging to the same plot in the target video into a plot segment; The segment representation prediction unit is used to obtain text information corresponding to the plot segment from the target video; combine the text information with a first prompt to obtain a first combined text, wherein the first prompt is text information for indicating that the segment representation prediction model outputs a first representation that satisfies the plot segment; input the plot segment and the first combined text into the segment representation prediction model to obtain the first representation, wherein the segment representation prediction model is a multimodal large model; An advertisement demand acquisition unit, used for acquiring advertisement demand information; a target segment matching unit, configured to determine a first matching degree between the advertisement demand information and the first representation of each of the plot segments, and determine a target segment from among the plurality of plot segments based on the first matching degree; The target advertisement embedding unit is used to embed the target advertisement information into the target segment.

16. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method for processing advertisement insertion information according to any one of claims 1 to 14 is implemented.

17. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for processing advertisement insertion information according to any one of claims 1 to 14 is implemented.

18. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device executes the method for processing advertisement placement information according to any one of claims 1 to 14.

Images