Promotional content processing method and apparatus, device, medium, and computer program product

Abstract

A promotional content processing method, executed by a computer device, and comprising: acquiring a video, and aggregating video frames in the video belonging to a same shot to form a shot clip (410); aggregating shot clips in the video belonging to a same plot to form a plot clip (420); generating first feature representations representing plot clips (430); acquiring promotional requirement information of promotional content (440); determining first degrees of matching between the promotional requirement information and the first feature representations of the plot clips, and on the basis of the first degrees of matching, determining a selected video clip from among the plurality of plot clips (450); and embedding the promotional content into the selected video clip (460).

Description

Method, device, equipment, medium and computer program product for promoting content processing

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 202411224414.2, filed on September 3, 2024, entitled "Method for processing advertisement implantation information, related device and medium", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the field of data processing, and in particular, to a method, device, equipment, medium and computer program product for processing promoting content. BACKGROUND

[0004] In a video website, it is often necessary to embed promoting content. For example, in a certain video, there is a scene in which the female protagonist applies skin care products. At this time, a floating layer appears in the scene, displaying an advertisement for skin lotion A. In the traditional technology, when embedding promoting content, video features and audio features are often extracted from the video, and a deep learning model is used to identify elements such as target detection, face recognition, behavior recognition, and scene recognition. The identified objects, characters, behaviors, and scenes are labeled to form labels. Then, the promotion requirement information is compared with the labels. If there is a label that matches the promotion requirement information, the promoting content is embedded.

[0005] The disadvantages of this scheme are lack of flexibility and poor accuracy. The lack of flexibility is reflected in the difficulty of covering flexible and variable advertising needs. Once there is an advertising need that cannot be covered by the existing label system (for example, the existing label system has hundreds of food, but the advertising need is for "snacks"), it is difficult to find the embedding position of the promoting content in the video. The poor accuracy is reflected in that sometimes a certain label appears in the video, but the position where the label appears is not suitable for embedding the promoting content, resulting in poor embedding effect of the promoting content. SUMMARY

[0006] The present disclosure provides a method, device, equipment, medium and computer program product for processing promoting content.

[0007] According to an aspect of the present disclosure, a method for processing promoting content is provided, which is executed by a computer device, and includes:

[0008] obtaining a video, and aggregating video frames belonging to the same shot in the video into a shot segment;

[0009] aggregating the shot segments belonging to the same plot in the video into a plot segment;

[0010] generate a first feature representation representing the plot segment;

[0011] obtain promotion demand information of the promotion content;

[0012] determine a first matching degree of the promotion demand information and the first feature representation of each plot segment, and determine a selected video segment from the plot segments based on the first matching degree; and

[0013] embed the promotion content in the selected video segment.

[0014] According to an aspect of the present disclosure, a promotion content processing apparatus is provided, which comprises:

[0015] a shot segment obtaining unit configured to obtain a video and aggregate video frames belonging to a same shot in the video into a shot segment;

[0016] a plot segment obtaining unit configured to aggregate the shot segments belonging to a same plot in the video into a plot segment;

[0017] a segment representation predicting unit configured to generate a first feature representation representing the plot segment;

[0018] a condition obtaining unit configured to obtain promotion demand information of the promotion content;

[0019] a selected video segment matching unit configured to determine a first matching degree of the promotion demand information and the first feature representation of each plot segment, and determine a selected video segment from the plot segments based on the first matching degree; and

[0020] a promotion content embedding unit configured to embed the promotion content in the selected video segment.

[0021] According to an aspect of the present disclosure, a computer device is provided, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the promotion content processing method as described above when executing the computer program.

[0022] According to an aspect of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the promotion content processing method as described above.

[0023] According to an aspect of the present disclosure, a computer program product is provided, which comprises a computer program, the computer program is read and executed by a processor of a computer device, so that the computer device executes the promotion content processing method as described above.

[0024] Additional features and advantages of the present disclosure will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of the present disclosure. The objectives and other advantages of the present disclosure will be realized and attained by the structure particularly pointed out in the description and claims. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings are included to provide a further understanding of the technical solutions of the present disclosure, and constitute a part of the specification, and are used to explain the technical solutions of the present disclosure together with the embodiments of the present disclosure, and do not constitute a limitation to the technical solutions of the present disclosure.

[0026] FIG. 1 is a schematic diagram of generating a tag library by extracting tags from a video;

[0027] FIG. 2 is a system architecture diagram to which a content promotion processing method according to an embodiment of the present disclosure is applied;

[0028] FIGS. 3A to 3D are schematic diagrams of an embodiment of the present disclosure applied to a scenario of selecting a video segment matched to an input advertisement demand in a Ruyi post advertisement;

[0029] FIG. 4 is a flowchart of a content promotion processing method according to an embodiment of the present disclosure;

[0030] FIG. 5 is a first flowchart of generating a first feature representation representing a plot segment in step 430 in FIG. 4;

[0031] FIG. 6 is a schematic diagram of obtaining the first feature representation representing the plot segment in FIG. 5;

[0032] FIG. 7 is a flowchart of obtaining text information corresponding to the plot segment from the video in step 510 in FIG. 5;

[0033] FIG. 8 is a schematic diagram of obtaining the text information corresponding to the plot segment in FIG. 7;

[0034] FIG. 9 is a flowchart of identifying the dialogue information from the plot segment by using a subtitle recognition model and a speech recognition model in step 710 in FIG. 7;

[0035] FIG. 10 is a schematic diagram of identifying the dialogue information by using the subtitle recognition model and the speech recognition model in FIG. 9;

[0036] FIG. 11 is a flowchart of determining the dialogue information from a plurality of candidate dialogue information based on a first probability and a second probability respectively corresponding to each candidate dialogue information in step 930 in FIG. 9;

[0037] FIG. 12 is a flowchart of inputting the plot segment and the text information into a segment representation prediction model to obtain the first feature representation representing the plot segment in step 520 in FIG. 5;

[0038] FIG. 13 is a schematic diagram of the first feature representation of the plot segment obtained by FIG. 12;

[0039] FIG. 14 is a flowchart of the input of the plot segment and the first combined text into the multi-modal large model by step 1220 in FIG. 12 to obtain the first feature representation of the plot segment;

[0040] FIG. 15 is a schematic diagram of the first feature representation of the plot segment obtained by FIG. 14 based on the multi-modal large model;

[0041] FIG. 16 is a second flowchart of the generation of the first feature representation of the plot segment by step 430 in FIG. 4;

[0042] FIG. 17 is a schematic diagram of the first feature representation of the plot segment obtained by FIG. 16 by incorporating the public figure information;

[0043] FIG. 18 is a flowchart of the input of the plot segment and the public figure information into the plot representation prediction model by step 1630 in FIG. 16 to obtain the first feature representation of the plot segment;

[0044] FIG. 19 is a schematic diagram of the first feature representation of the plot segment obtained by FIG. 18 based on the second combined text;

[0045] FIG. 20 is a first flowchart of the determination of the first matching degree between the promotion demand information and the first feature representation of each plot segment by step 450 in FIG. 4;

[0046] FIG. 21 is a schematic diagram of the determination of the first matching degree between the promotion demand information and the first text description by FIG. 20;

[0047] FIG. 22 is a flowchart of the extraction of a plurality of first keywords from the promotion demand information by step 2010 in FIG. 20;

[0048] FIG. 23 is a schematic diagram of the extraction of the first keywords by FIG. 22;

[0049] FIG. 24 is a flowchart of the determination of the first similarity between the first keyword and the first text description for each first keyword by step 2020 in FIG. 20;

[0050] FIG. 25 is a flowchart of the determination of the first term frequency of the first keyword in the first text description for each first keyword by step 2410 in FIG. 24;

[0051] FIG. 26 is a schematic diagram of the determination of the first term frequency of each first keyword by FIG. 25;

[0052] FIG. 27 is a flowchart of the determination of the first inverse document frequency of the first keyword in the plurality of first text descriptions by step 2420 in FIG. 24;

[0053] FIG. 28 is a diagram illustrating the determination of the first inverse document frequency of each first keyword in FIG. 27;

[0054] FIG. 29 is a flowchart illustrating the determination of the first number of first text descriptions containing the first keyword in step 2710 of FIG. 27;

[0055] FIG. 30 is a diagram illustrating the construction of the first index lookup table for determining the first number in FIG. 27;

[0056] FIG. 31 is a second flowchart illustrating the determination of the first matching degree of the promotion requirement information and the first feature representation of each plot segment in step 450 of FIG. 4;

[0057] FIG. 32 is a diagram illustrating the determination of the first matching degree based on the first deep feature and the second deep feature in FIG. 31;

[0058] FIG. 33 is a flowchart illustrating the determination of the first matching degree of the promotion requirement information and the first feature representation of each plot segment in step 450 of FIG. 4, and the determination of the selected video segment from the plot segments based on the first matching degree in FIG. 4;

[0059] FIG. 34 is a diagram illustrating the segment matching based on the first feature representation and the second feature representation in FIG. 33;

[0060] FIG. 35 is a flowchart illustrating the input of the shot segment to the segment representation prediction model to obtain the second feature representation representing the shot segment in step 3310 of FIG. 33;

[0061] FIG. 36 is a diagram illustrating the determination of the second feature representation based on the shot segment and the second prompt in FIG. 35;

[0062] FIG. 37 is a flowchart illustrating the input of the shot segment and the second prompt to the segment representation prediction model together to obtain the second feature representation representing the shot segment in step 3510 of FIG. 35;

[0063] FIG. 38 is a diagram illustrating the obtaining of the second feature representation based on the segment representation prediction model in FIG. 37;

[0064] FIG. 39 is a first flowchart illustrating the determination of the second matching degree of the promotion requirement information and the second feature representation of each shot segment in step 3320 of FIG. 33;

[0065] FIG. 40 is a diagram illustrating the determination of the second matching degree of the promotion requirement information and the second text description in FIG. 39;

[0066] FIG. 41 is a flowchart illustrating the extraction of the first keywords from the promotion requirement information in step 3910 of FIG. 39;

[0067] FIG. 42 is a flowchart of step 3920 in FIG. 39 for determining, for each first keyword, a second similarity of the first keyword to the second text description;

[0068] FIG. 43 is a flowchart of step 4210 in FIG. 42 for determining, for each first keyword, a second term frequency of the first keyword appearing in the second text description;

[0069] FIG. 44 is a diagram of determining the second term frequency of each first keyword in FIG. 43;

[0070] FIG. 45 is a flowchart of step 4220 in FIG. 42 for determining a second inverse document frequency of the first keyword appearing in the plurality of second text descriptions;

[0071] FIG. 46 is a diagram of determining the second inverse document frequency of each first keyword in FIG. 45;

[0072] FIG. 47 is a flowchart of step 4510 in FIG. 45 for determining a second number of the second text descriptions containing the first keyword;

[0073] FIG. 48 is a diagram of determining the second number based on the second index lookup table in FIG. 47;

[0074] FIG. 49 is a second flowchart of step 3320 in FIG. 33 for determining a second matching degree of the promotion demand information to the second feature representation of each sub-shot;

[0075] FIG. 50 is a diagram of determining the second matching degree based on the second deep feature and the third deep feature in FIG. 49;

[0076] FIG. 51 is a first flowchart of step 3330 in FIG. 33 for determining the selected video segment in the plurality of story segments and the plurality of sub-shots based on the first matching degree and the second matching degree;

[0077] FIG. 52 is a second flowchart of step 3330 in FIG. 33 for determining the selected video segment in the plurality of story segments and the plurality of sub-shots based on the first matching degree and the second matching degree;

[0078] FIG. 53 is a diagram of determining the selected video segment in FIG. 52;

[0079] FIG. 54 is a block diagram of a promotion content processing apparatus according to an embodiment of the present disclosure;

[0080] FIG. 55 is a terminal structure diagram of the promotion content processing method shown in FIG. 4 according to an embodiment of the present disclosure;

[0081] FIG. 56 is a server structure diagram of the promotion content processing method shown in FIG. 4 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0083] Before the embodiments of the present disclosure are further described in detail, the terms and phrases involved in the embodiments of the present disclosure are explained, and the terms and phrases involved in the embodiments of the present disclosure are applicable to the following explanations:

[0084] Scenario advertising: Scenario advertising is a type of advertising in which a brand finds an implantation scene that matches the product tone based on the plot or video screen, accurately combines the elements in the scene with the brand / product, and appears in the form of a floating layer without interruption along with the video screen.

[0085] Automatic Speech Recognition (ASR): a technology that converts speech signals into text information.

[0086] Optical Character Recognition (OCR): a technology that converts text images into machine-readable text information.

[0087] Transformer model: a deep learning model structure based on self-attention mechanism, which has good effect in natural language processing tasks. In the self-attention mechanism, the Transformer model calculates the correlation between each token in the input sequence and other tokens, thereby capturing long-distance dependencies within the sequence. This mechanism allows the Transformer to process the entire sequence in parallel, improving computational efficiency.

[0088] Token: Token in Transformer refers to a word or sub-word (such as a single character in Chinese) in text, which is the basic unit used by the model when processing sequences; in a multi-modal model, visual information is also mapped into a token and input into the language model.

[0089] Term Frequency-Inverse Document Frequency (TF-IDF): a statistical method for evaluating the importance of a word in a document set or corpus. The core idea is to measure the importance of a word to a document by calculating the frequency of its occurrence in a particular document (i.e. term frequency, TF) and the sparsity of its occurrence in the entire corpus (i.e. inverse document frequency, IDF).

[0090] In video websites, it is often necessary to embed promotional content. For example, in a video, there is a scene where the female protagonist applies skin care products. At this time, a floating layer appears in the picture, showing the advertisement of skin lotion A. In related technologies, when performing promotional content embedding processing, video features and audio features are often extracted from the video, and a deep learning model is used to identify elements such as target detection, face recognition, behavior recognition, and place recognition. The identified objects, characters, behaviors, and places are labeled to form labels, as shown in FIG. 1. Then, compare the promotional demand information with the labels. If there is a label that matches the promotional demand information, embed the promotional content.

[0091] The disadvantages of this scheme are lack of flexibility and poor accuracy. The lack of flexibility is reflected in the difficulty of covering flexible and variable advertising needs. Once there is an advertising demand that cannot be covered by the existing label system (for example, the existing label system has hundreds of food, but the advertising demand is for "snacks"), it is difficult to find the promotional content embedding position in the video. Poor accuracy is reflected in that sometimes a certain label appears in the video, but the position where the label appears is not necessarily suitable for embedding promotional content, resulting in poor promotional content embedding effect.

[0092] System architecture and scenario of the application of the embodiments of the present disclosure

[0093] FIG. 2 is a system architecture diagram applied to the promotional content processing method according to an embodiment of the present disclosure. It includes a server 210, the Internet 220, an object terminal 230, and the like.

[0094] The server 210 refers to a computer system capable of providing a promotion content processing service, for example, it is capable of matching out a video clip that best matches the promotion demand information from the object terminal 230 and outputting it. Relative to the object terminal 230, the server 210 requires higher stability, security, performance, etc. The server 210 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. The server 210 can also communicate with the Internet 220 in a wired or wireless manner to exchange data.

[0095] The object terminal 230 is a device that shows the object a video based on embedded promotion content, or provides an operation interface for the object to input promotion demand information, select / upload a video to be embedded with promotion content. It includes desktop computers, laptops, PDAs (personal digital assistants), mobile phones, car terminals, home theater terminals, dedicated terminals, etc. in various forms. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network and share a display device to work together, collectively constituting a terminal. The terminal can also communicate with the Internet 220 in a wired or wireless manner to exchange data.

[0096] The embodiments of the present disclosure can be applied in various scenarios, for example, the scenario of Youyitie advertising shown in FIGS. 3A-3D.

[0097] As shown in FIG. 3A, the content management platform interface can show the promotion content placement user a content library, which provides the promotion content placement user with a list of selectable episode names. The promotion content placement user selects the episode that needs to be placed in the scene by clicking or frame selection.

[0098] After the promotion content placement user completes the selection of the episode to be placed, it will jump to the advertisement demand input interface, as shown in FIG. 3B. In this interface, the promotion content placement user can input specific advertisement demand in the advertisement demand input box according to specific placement demand, and click “Search” to start clip matching. For example, the product that needs to be placed, the promotion language, the target scene of placement, etc.

[0099] In the background server of the content management platform, a plurality of episodes requiring advertisement placement are respectively taken as videos. For each video, the background server aggregates video frames belonging to the same shot in the video into a shot segment, aggregates shot segments belonging to the same plot in the video into a plot segment, and generates a first feature representation representing the plot segment. The background server further inputs promotion demand information input by a promotion content placement user based on the promotion content placement, determines a first matching degree between the promotion demand information and the first feature representation of each plot segment, and determines a selected video segment from the plurality of plot segments based on the first matching degree.

[0100] After the background server determines the selected video segment corresponding to each episode requiring advertisement placement, the background server generates an advertisement matching list based on the title of the episode to which the selected video segment belongs, the corresponding time (hour: minute: second), the matched text description (the first feature representation), and the specific thumbnail corresponding to the selected video segment, and displays the advertisement matching list on an interface of the content management platform to the promotion content placement user for confirmation or further screening, as shown in FIG. 3C.

[0101] After the promotion content placement user determines that the selected video segment matched by the advertisement is correct, the background server embeds the promotion content in the selected video segment. In this example, an advertisement floating layer is “pasted” on the selected video segment of the episode in the form of a wish post. As shown in FIG. 3D, when a video viewing user views the corresponding episode on a terminal (such as a mobile phone), the corresponding advertisement floating layer will be seen in the scene related to the advertisement.

[0102] In other examples, during the matching process of the advertisement and the selected video segment, the background server further extracts a shot segment of the video, inputs the shot segment into a segment representation prediction model to obtain a corresponding second feature representation, and matches the promotion demand information with both the first feature representation and the second feature representation to obtain a matched plot segment or shot segment. In this way, advertisement matching can be performed based on multiple different segment granularities, and the accuracy of the matching can be improved.

[0103] In summary, the video frames belonging to the same shot in the video to be embedded with the promotion content are aggregated into a shot segment, and the shot segments belonging to the same plot in the video are aggregated into a plot segment. Such a plot segment can have a complete semantic in common due to belonging to the same plot. A first feature representation representing the plot segment is generated. The first feature representation expresses the complete semantic of the plot segment. Comparing the promotion requirement information with the first feature representation of each plot segment obtains a first matching degree of the promotion requirement information and the first feature representation of each plot segment. The matching degree represents the accurate matching degree of the semantic of the entire promotion requirement information and the semantic of each plot segment. The video segment to be embedded with the promotion content is determined through the matching degree. Due to the deep semantic analysis, the matching degree has higher accuracy than the simple label comparison in the related art. Meanwhile, since the example relies on the deep semantic analysis and matching to identify the embedding position of the promotion content, the example does not depend on a fixed label system, and thus has higher flexibility in processing the promotion content.

[0104] It should be understood that the above only illustrates the application scenarios of some embodiments of the present disclosure. The business scenarios to which the present disclosure can be applied can include, but are not limited to, the specific embodiments described above.

[0105] General description of embodiments of the present disclosure

[0106] It should be emphasized that the embodiments of the present disclosure can be applied to various application scenarios, such as movie and television play playing, video website, video software, live broadcast advertisement insertion, media information processing, and the like. In the related art, there are problems of insufficient flexibility and poor accuracy in embedding promotion content by labeling elements identified from a video and matching the labels with promotion requirement information. Some embodiments of the present disclosure provide a promotion content processing method, device, equipment, medium, and computer program product, which can improve the flexibility and accuracy of promotion content processing.

[0107] The promotion content processing method is a method of obtaining, by a segment representation prediction model, a first feature representation of each plot segment of a video, and accurately matching the first feature representation with advertisement requirement based on the first feature representation, so as to determine a video segment suitable for embedding promotion content.

[0108] The promotion content processing method of the embodiments of the present disclosure can be executed on the server 210, or on the object terminal 230, or partially on the server 210 and partially on the object terminal 230.

[0109] As shown in FIG. 4, according to one embodiment of the present disclosure, the promotion content processing method includes:

[0110] Step 410, obtaining a video, and aggregating video frames belonging to the same split shot in the video into a split shot segment.

[0111] Step 420, aggregating split shot segments belonging to the same plot in the video into a plot segment.

[0112] Step 430, generating a first feature representation of the plot segment.

[0113] Specifically, the plot segment can be input into a segment representation prediction model to obtain the first feature representation of the plot segment.

[0114] Step 440, obtaining promotion demand information of the promotion content.

[0115] Step 450, determining a first matching degree between the promotion demand information and the first feature representation of each plot segment, and determining a selected video segment from the multiple plot segments based on the first matching degree.

[0116] Step 460, embedding the promotion content in the selected video segment.

[0117] In step 410, the video refers to a video in which the promotion content needs to be embedded. Exemplarily, the video can be a video selected by a promotion content delivery user as needed. The video can be a certain episode or a certain number of episodes of a series, or a certain film or a certain number of films of a film and television series, and a film and television series includes multiple episodes of a series.

[0118] The video frames of the same split shot refer to video frames in which the picture scene in the picture is the same or similar, and the shot layout is the same in the video. The split shot segment refers to a segment obtained by aggregating video frames belonging to the same split shot, which is used to represent a split shot. Each split shot segment is relatively coherent video picture content.

[0119] Exemplarily, the video is segmented into multiple split shot segments by a shot segmentation tool (PySceneDetect) based on picture statistical information. PySceneDetect is an open source tool for detecting scene changes in a video and automatically segmenting the video, which mainly identifies shot transitions by analyzing the pixel changes of each frame, and segments the video into multiple segments.

[0120] PySceneDetect performs scene change detection on a video, identifies the transition points of different scenes by analyzing the video frames of the video; then performs frame-by-frame analysis on the video, and counts the information of each video frame, such as counting the average of the pixel values of each video frame; based on the pre-set threshold value, and the information of each video frame counted, scene cutting is performed. PySceneDetect finally outputs multiple video segments with clear scene change points, i.e. split shots. It should be noted that PySceneDetect is a specific model example provided in this example, and in addition to this, other shot segmentation models can also be used, which are not specifically limited in this disclosure.

[0121] In step 420, the split shots of the same plot refer to split shots that occur in the same scene and can form a continuous story.

[0122] The plot segment refers to a video segment obtained by aggregating multiple split shots, used to represent a continuous plot story.

[0123] Exemplarily, the plot segmentation model in this embodiment can adopt a model of a recurrent neural network (RNN). By inputting a training set of shot features with segmentation point labels, the plot segmentation model is trained so that the trained plot segmentation model can automatically identify the segmentation points in the continuous multiple shot features input thereto. The multiple split shots obtained in step 410 are input into the trained plot segmentation model, the plot segmentation model automatically identifies the segmentation points in the multiple split shots, and aggregates the split shots between each adjacent two segmentation points into a plot segment, thereby aggregating the multiple split shots into at least one plot segment.

[0124] In this example, the plot segmentation model adopts a model of a recurrent neural network (RNN). For example, by inputting a training set of shot features with segmentation point labels, the plot segmentation model can be trained so that the trained plot segmentation model can automatically identify the segmentation points in the continuous multiple shot features input thereto, and aggregate the shot features between each adjacent two segmentation points into a plot segment.

[0125] In this example, the multiple split shots obtained in step 410 are input into the trained plot segmentation model, and the plot segmentation model automatically identifies the segmentation points in the multiple split shots. Finally, the plot segmentation model aggregates the split shots between each adjacent two segmentation points into a plot segment, thereby aggregating the multiple split shots into at least one plot segment. It should be noted that the use of the plot segmentation model to obtain the plot segment is only a specific model example provided in this example, and in addition to this, other models with neural network structures can also be used, which are not specifically limited in this disclosure.

[0126] In step 430, the segment representation prediction model refers to a model capable of analyzing the plot segment, obtaining the semantic information corresponding to the plot segment, and outputting in various forms. For example, the segment representation prediction model can be a multi-modal large model. The semantic information corresponding to the plot segment refers to information capable of representing the specific content of the plot segment.

[0127] The first feature representation refers to the information output by the segment representation prediction model after analyzing the input plot segment, which is used to represent the summary description of the content corresponding to the plot segment, i.e., the semantic information corresponding to the plot segment described above. Exemplarily, the first feature representation can be a textual description composed of natural language, or an abstract expression composed of a feature vector form.

[0128] In step 440, the promotion demand information refers to information used to represent the expectations / demands of the promotion content placement user for the position of the embedded promotion content in the video.

[0129] Exemplarily, the promotion demand information can include specific promotion terms, promotion products, promotion celebrity endorsers, and promotion product applicable scenarios (such as family gatherings, outdoor sports, etc.).

[0130] Exemplarily, the promotion demand information can be input into the system by the promotion content placement user through the content management platform provided in FIGS. 3A-3C, so that the system can obtain the promotion demand information when executing the promotion content processing method. The promotion demand information can also be extracted from the promotion demand information table pre-stored in the system, which is obtained by statistically analyzing the demands of the promotion content placement user.

[0131] In step 450, the first matching degree refers to the semantic similarity between the promotion demand information and the first feature representation of the plot segment. The higher the similarity, the higher the first matching degree.

[0132] After determining the first matching degree of each plot segment and the promotion demand information, the plot segment corresponding to the first matching degree meeting the requirement is selected as the selected video segment. Exemplarily, the plot segment with a first matching degree greater than a preset matching degree threshold can be selected as the selected video segment. Alternatively, the plot segments are sorted in descending order of the first matching degree, and the top, for example, the top 10% of the plot segments are selected as the selected video segment. Alternatively, the plot segment with the largest first matching degree is selected as the selected video segment.

[0133] In step 460, the promotion content refers to the promotion content corresponding to the promotion demand information, which is expected to be embedded in the selected video segment of the video. Exemplarily, the promotion content can include product name, promotion language, promotion picture, and promotion content display form (such as a floating layer, a video top scroll, a video bottom scroll, etc.). The promotion content can be commercial advertising information or public welfare promotion information such as a missing person notice and travel promotion information.

[0134] Exemplarily, the promotion content can be embedded in the form of a floating layer. Specifically, the product name, promotion language, and promotion picture corresponding to the promotion content are edited to obtain a completed floating layer image. In terms of technical implementation, first, the time range (start time and end time) of the selected video segment in the video needs to be determined. The time range can be obtained through the timestamp information of the video. Then, during video playback, a video processing library (such as FFmpeg) is used to monitor the playback time of the video. When the playback time reaches the start time of the selected video segment, the floating layer image is superimposed on the video frame. The specific way of superimposition can be to mix the pixel values of the floating layer image and the pixel values of the video frame according to a certain transparency. For example, the alpha channel is used to control the transparency of the floating layer image, and the pixel values of the floating layer image are multiplied by the transparency coefficient and then added to the pixel values of the video frame to obtain the superimposed pixel values. When the playback time reaches the end time of the selected video segment, the superposition of the floating layer image is stopped. Thus, when the video is played to these selected video segments, the corresponding floating layer image appears to display the promotion content, as shown in FIG. 3D.

[0135] The embodiments of steps 410 to 460 aggregate the video frames belonging to the same split shot in the video into which the promotion content is to be embedded into a split shot segment, and then aggregate the split shot segments belonging to the same plot in the video into a plot segment. Such a plot segment can have a complete semantic because it belongs to the same plot. A first feature representation representing the plot segment is generated. This first feature representation expresses the complete semantic of the plot segment. Comparing the promotion demand information with the first feature representation of each plot segment obtains a first matching degree of the promotion demand information and the first feature representation of each plot segment. This matching degree represents the degree of accurate matching of the semantic of the entire promotion demand information and the semantic of each plot segment. The video segment in which the promotion content is embedded is determined through the matching degree, which has higher accuracy than the simple label comparison of related technologies because of the deep semantic analysis and matching. Meanwhile, because the deep semantic analysis and matching are used to identify the embedding position of the promotion content in the embodiments of the present disclosure, the embodiments do not depend on a fixed label system, thereby alleviating the poor flexibility caused by the fixed label system and having higher flexibility of promotion content embedding processing.

[0136] The above is a general description of steps 410-460, which have been described in more detail above, so only the specific implementation process of steps 430 and 450 will be described in detail below.

[0137] Detailed description of step 430

[0138] In step 430, a first feature representation of the plot segment is generated.

[0139] In an embodiment, referring to FIG. 5, step 430 includes:

[0140] Step 510, obtaining text information corresponding to the plot segment from the video.

[0141] Step 520, inputting the plot segment and the text information into a segment representation prediction model to obtain the first feature representation of the plot segment.

[0142] The steps 510 and 520 will be described in detail below.

[0143] In step 510, the text information corresponding to the plot segment refers to information expressed in text form, including dialogue, characters, etc. in the plot segment.

[0144] In some examples, the text information can include only dialogue information, or only character information, or both dialogue and character information based on requirements.

[0145] In other examples, the character information can be the information of the character corresponding to the role in the plot, or the information of the actor playing the role.

[0146] In step 520, the first feature representation refers to the plot summary information representing the plot segment, which is output by the segment representation prediction model by respectively performing semantic extraction on the plot segment and the text information and combining the semantics of the two.

[0147] The process of obtaining the first feature representation of the plot segment by the segment representation prediction model will be explained in detail below in conjunction with FIG. 6.

[0148] As shown in FIG. 6, each video selected to be promoted is preprocessed. The preprocessing includes segmenting each video to obtain a plurality of plot segments, such as segment 1 to segment n in the figure, and extracting text from each video to obtain text information containing dialogue information, and segmenting the text information to obtain text 1 to text n corresponding to segment 1 to segment n. In other examples, text extraction processing can also be performed on each plot segment to obtain text information corresponding to each plot segment.

[0149] Segment 1 to segment n and corresponding text 1 to text n are input into a segment representation prediction model in pairs. The segment representation prediction model outputs first feature representations corresponding to each segment, such as plot 1 to plot n corresponding to segment 1 to segment n in the figure. In this example, the first feature representation of a plot segment can be a description text of a plot summary of the plot segment, or a feature vector corresponding to the plot summary of the plot segment.

[0150] In related technologies, text extraction and video feature extraction are performed on a video respectively to obtain text content and video feature content, and then the text content and the video feature content are labeled respectively to obtain each label. In steps 510 and 520, the plot segment and the corresponding text information are input into the segment representation prediction model to obtain the first feature representation, so that the semantic of the plot segment corresponding to the first feature representation is more complete, and the accuracy of matching the first feature representation with the promotion requirement information can be improved.

[0151] In an embodiment, referring to FIG. 7, step 510 includes:

[0152] Step 710, using a subtitle recognition model and a speech recognition model to recognize dialogue information from the plot segment.

[0153] Step 720, using a face recognition model to recognize character information corresponding to the dialogue information from the plot segment.

[0154] Step 730, integrating the dialogue information and the character information corresponding to the dialogue information into text information.

[0155] The steps 710 to 730 are described in detail as follows.

[0156] In step 710, the exemplary subtitle recognition model employs optical character recognition (OCR) technology. First, the subtitle recognition model converts each video frame of the plot segment into a grayscale image to reduce data volume and computational complexity. Then, the subtitle recognition model converts the grayscale image into a black-and-white image to highlight the text information. Next, the subtitle recognition model locates the text region using an edge detection algorithm and performs character segmentation on the located text region to separate continuous text into individual characters. Finally, the subtitle recognition model identifies each character using a character classifier and combines the identified characters into the dialogue information. In other examples, other image processing-based subtitle recognition models can be selected, which are not limited herein.

[0157] In step 710, the exemplary subtitle recognition model employs optical character recognition (OCR) technology. First, the subtitle recognition model converts each video frame of the plot segment into a grayscale image to reduce data volume and computational complexity. Then, the subtitle recognition model converts the grayscale image into a black-and-white image to highlight the text information. Next, the subtitle recognition model locates the text region using an edge detection algorithm and performs character segmentation on the located text region to separate continuous text into individual characters. Finally, the subtitle recognition model identifies each character using a character classifier and combines the identified characters into the dialogue information. In other examples, other image processing-based subtitle recognition models can be selected, which are not limited herein.

[0158] The dialogue information refers to the dialogue of each character in the plot segment. Dialogue is an important element for driving the plot development and explaining the plot to the audience. Therefore, extracting the dialogue information in the plot segment as one of the text information is conducive to semantic analysis of the plot segment.

[0159] In step 720, the face recognition model can recognize the characters appearing in the plot segment and obtain the name, appearance time, and position of the characters. The specific process is as follows: first, the face recognition model detects the faces in the video frames of the plot segment using a face detection algorithm such as Haar cascade classifier or deep learning-based face detection algorithm (such as MTCNN). Then, the face recognition model extracts features of the detected faces and converts the face images into feature vectors using a pre-trained face recognition model (such as FaceNet, ArcFace, etc.). Next, the face recognition model compares the extracted feature vectors with a pre-constructed character feature library containing feature vectors and corresponding character names of known characters. By calculating the similarity (such as cosine similarity) between the feature vectors, the face recognition model finds the most matching character name. Meanwhile, the face recognition model records the position information (such as coordinates) of the face in the video frames and the video frame time when the character first appears and last appears, thereby obtaining the appearance time and position of the character. The character information includes the name, appearance time, and appearance position of the character corresponding to the dialogue information.

[0160] In step 730, the text information integrates the dialogue information and the character information, and at this time, the text information contains the specific dialogue of the plot segment and the character corresponding to the dialogue. The integration can be performed in the following manner: for each piece of dialogue information, the corresponding character information (character name) is searched, and then the character name and the dialogue information are combined in the format of "character name: dialogue content". For example, if the dialogue of the character "Zhang San" is "hello", the integrated information is "Zhang San: hello". All the information thus combined is summarized together to form the final text information.

[0161] The process of obtaining text information from a plot segment will be explained below in conjunction with FIG. 8. As shown in FIG. 8, the video is segmented to obtain a plurality of plot segments, such as segment 1 to segment n. Segment 1 to segment n are respectively input into the speech recognition model, the subtitle recognition model, and the face recognition model.

[0162] The speech recognition model recognizes the dialogue information in segment 1 to segment n from the speech level. The subtitle recognition model recognizes the dialogue information in segment 1 to segment n from the image (subtitle) level. The recognition results of the speech recognition model and the subtitle recognition model are combined to obtain dialogue 1 to dialogue n corresponding to segment 1 to segment n.

[0163] The face recognition model respectively recognizes the faces in segment 1 to segment n to obtain character information of each plot segment, such as character 1 to character n.

[0164] Dialogue 1 to dialogue n and character 1 to character n are integrated one by one, for example, dialogue 1 is integrated with character 1, dialogue 2 is integrated with character 2, and so on. Finally, the text information corresponding to each plot segment is obtained, such as text 1 to text n.

[0165] If the text information is directly extracted from the plot segment, the calculation amount is large, and the model requirement is high, and the extraction result is usually difficult to meet the requirements. In steps 710 to 730, the speech recognition model and the subtitle recognition model are used to extract the dialogue information from the speech dimension and the image dimension, and the face recognition model is used to extract the character information. By using different models to extract different elements (dialogue and character) in the plot segment, the final text information is obtained by integration. Compared with directly extracting all information from the plot segment, the present embodiment uses different models to extract the required information and then integrates the text information, which can more accurately obtain various required information, has low model requirement, and has small calculation load.

[0166] In an embodiment, referring to FIG. 9, step 710 includes:

[0167] Step 910, using a subtitle recognition model, performing subtitle recognition in the plot segment to obtain a first probability corresponding to each candidate dialogue information.

[0168] Step 920, using a speech recognition model, performing speech recognition in the plot segment to obtain a second probability corresponding to each candidate dialogue information.

[0169] Step 930, based on the first probability and the second probability corresponding to each candidate dialogue information, determining the dialogue information from the plurality of candidate dialogue information.

[0170] The steps 910 to 930 will be described in detail below.

[0171] In step 910, in the process of using the subtitle recognition model to recognize the subtitles in the plot segment and output the dialogue information, the subtitle recognition model generates the probability information corresponding to the dialogue information, i.e. the first probability, while generating the dialogue information. The candidate dialogue information is the dialogue information corresponding to the plot segment output by the subtitle recognition model. Since the dialogue information at this time is not the final result, it is candidate dialogue information. There is a first probability corresponding to each candidate dialogue information.

[0172] In step 920, in the process of using the speech recognition model to recognize the speech information in the plot segment and output the dialogue information, the speech recognition model generates the probability information corresponding to the dialogue information, i.e. the second probability, while generating the dialogue information. The candidate dialogue information is the dialogue information corresponding to the plot segment output by the speech recognition model. Since the dialogue information at this time is not the final result, it is candidate dialogue information. There is a second probability corresponding to each candidate dialogue information.

[0173] In step 930, when recognizing the same plot segment, for each sentence of dialogue, since the subtitle recognition model and the speech recognition model are used for recognition respectively, there will be two output results as candidate dialogue information, and the candidate dialogue information corresponds to the first probability and the second probability respectively.

[0174] Exemplarily, for each pair of candidate dialogue information, the first probability and the second probability corresponding thereto are compared. If the first probability is greater than the second probability, the candidate dialogue information corresponding to the first probability is selected as the final dialogue information; if the second probability is greater than the first probability, the candidate dialogue information corresponding to the second probability is selected as the final dialogue information.

[0175] The process of using the subtitle recognition model and the speech recognition model to recognize the dialogue information will be explained and described below in conjunction with FIG. 10.

[0176] The subtitle recognition model identifies the plot segment and outputs candidate lines 1_1, candidate lines 2_1, …, candidate lines n_1, each of which corresponds to a first probability. The speech recognition model identifies the plot segment and outputs candidate lines 1_2, candidate lines 2_2, …, candidate lines n_2, each of which corresponds to a second probability. Among them, candidate line 1_1 and candidate line 1_2 are a pair of candidate line information 1, and the first probability and the second probability; similarly, candidate line 2_1 and candidate line 2_2 are a pair of candidate line information 2, and the first probability and the second probability, and so on.

[0177] For each pair of candidate line information, based on the larger probability of the first probability and the second probability, the candidate line corresponding to the larger probability is taken as the line information. For example, in candidate line information 1, the first probability is greater than the second probability, so candidate line 1_1 is selected as the line information, that is, line information 1_1 is obtained; in candidate line information 2, the second probability is greater than the first probability, so candidate line 2_2 is selected as the line information, that is, line information 2_2 is obtained. In this way, line information 1_1, line information 2_2, …, line information n_1 are finally obtained.

[0178] In related technologies, a single model is usually used to extract the subtitles of a video. Due to the existence of problems such as noisy environment, pronunciation problems, or inaccurate subtitles, the single speech recognition model or the subtitle recognition model may have inaccurate problems. In steps 910 to 930, the subtitle recognition model and the speech recognition model are used simultaneously to identify the plot segment to obtain multiple candidate line information, and the candidate line information with a larger probability is selected as the final line information according to the first probability and the second probability. The line information obtained in this way is determined by the candidate line information with a larger probability (confidence) in the two models, which can improve the accuracy of line information identification.

[0179] In an embodiment, referring to FIG. 11, step 930 includes:

[0180] Step 1110, obtaining a first weight of the subtitle recognition model and a second weight of the speech recognition model.

[0181] Step 1120, determining a weighted sum of the first probability and the second probability based on the first weight and the second weight.

[0182] Step 1130, determining the line information from the multiple candidate line information based on the weighted sum.

[0183] The steps 1110 to 1130 are described in detail as follows.

[0184] In step 1110, the first weight is an index for characterizing the reliability degree of the recognition result of the subtitle recognition model. The second weight is an index for characterizing the reliability degree of the recognition result of the speech recognition model. Exemplarily, the first weight and the second weight can be pre-set based on the actual situation of the subtitles and the character speeches in the video. For example, in a certain video, the subtitles of the video are clear and complete, but the character speeches have a large number of dialects or different national languages, at this time, the speech recognition model has a high recognition complexity and is prone to deviation. At this time, the first weight is set to be greater than the second weight, and the first weight is set to be larger, and the second weight is set to be smaller.

[0185] Exemplarily, the subtitle recognition model and the speech recognition model are trained with the same type of video as a sample, and based on the accuracy of the output results of the subtitle recognition model and the speech recognition model during training, the first weight and the second weight of the subtitle recognition model and the speech recognition model for the recognition of this type are set. Wherein, the video type refers to whether the video has dialect, whether it has multi-national language, whether it has multi-language subtitles, whether it has missing / subtitle unclear, etc.

[0186] In step 1120, the weighted sum is the sum of the product of the first probability and the first weight and the product of the second probability and the second weight. The calculation formula of the weighted sum S is S = w1 x P1 + w2 x P2, wherein w1 represents the first weight of the subtitle recognition model, P1 represents the first probability corresponding to the candidate script information, w2 represents the second weight of the speech recognition model, and P2 represents the second probability corresponding to the candidate script information.

[0187] In step 1130, exemplarily, in the case that the first weight of the subtitle recognition model is greater than the second weight of the speech recognition model, the weighted sum is compared with a pre-set threshold value. If the weighted sum is greater than the threshold value, the candidate script information output by the subtitle recognition model is taken as the script information, that is, the candidate script information corresponding to the first probability is selected as the script information. If the weighted sum is less than the threshold value, the candidate script information output by the speech recognition model is taken as the script information, that is, the candidate script information corresponding to the second probability is selected as the script information.

[0188] In the case that the second weight of the speech recognition model is greater than the first weight of the subtitle recognition model, the weighted sum is compared with a pre-set threshold value. If the weighted sum is greater than the threshold value, the candidate script information output by the speech recognition model is taken as the script information, that is, the candidate script information corresponding to the second probability is selected as the script information. If the weighted sum is less than the threshold value, the candidate script information output by the subtitle recognition model is taken as the script information, that is, the candidate script information corresponding to the first probability is selected as the script information.

[0189] The difficulty of caption recognition or speech recognition for different videos is often different. If only the corresponding probabilities of the candidate caption information in the respective model are selected, the video type factor cannot be considered. In steps 1110-1130, the first weight of the caption recognition model and the second weight of the speech recognition model are obtained. When determining the caption information from the plurality of candidate caption information, the weighted sum of the first probability and the second probability is used as the basis for determining the caption information. For the same video or plot segment, the greater the weight, the more accurate and reliable the recognition result of the corresponding model. Therefore, the determination of the caption information based on the weighted sum of the first probability and the second probability not only considers the probability of the model itself, but also considers the contribution of the model to the video (the first weight and the second weight), thereby improving the accuracy and reliability of the caption information.

[0190] In an embodiment, the segment representation prediction model is a multi-modal large model. Referring to FIG. 12, step 520 includes:

[0191] Step 1210: combine the text information into the first prompt to obtain a first combined text.

[0192] Step 1220: input the plot segment and the first combined text into the multi-modal large model to obtain a first feature representation of the plot segment.

[0193] The steps 1210 and 1220 are described in detail below.

[0194] In step 1210, the first prompt refers to the text information input by the user to instruct the segment representation prediction model to output the first feature representation of the plot segment that meets the user's demand. In some examples, the first prompt can be a pre-set templated prompt. In other examples, the first prompt can also be a prompt input by the user on the spot based on the plot segment.

[0195] For example, the first prompt is "Please combine the picture information and the caption information to summarize the plot summary of this segment." At this time, the segment representation prediction model will output the summary description of the corresponding plot segment as the first feature representation under the guidance of the first prompt.

[0196] The first combined text refers to text information corresponding to the drama segment combined with the first prompt in a natural language manner. It is used to guide the multi-modal large model to analyze and process the drama segment, and output the first feature representation of the drama segment required by the user. For example, when the text information includes dialogue information and character information, the specific dialogue and characters included in the text information are combined with the first prompt "summarize the plot summary of this segment". The first combined text is "The characters in the drama segment include … (character information), and the dialogue of the characters is … (dialogue information). Please summarize the plot summary of this segment based on the picture information and dialogue information."

[0197] In step 1220, the multi-modal large model can integrate and process input data from different modalities (such as text, images, audio, etc.), and output corresponding content based on user needs.

[0198] After inputting the drama segment and the first combined text into the multi-modal large model, the multi-modal large model will perform feature extraction and feature conversion on the drama segment, mapping the drama segment into features that the multi-modal large model can process. The multi-modal large model will also perform feature extraction on the first combined text to obtain corresponding features. At this time, the features corresponding to the drama segment and the features corresponding to the first combined text belong to the same modality.

[0199] The multi-modal large model then analyzes and processes the drama segment and the first combined text based on the semantics corresponding to the first prompt in the first combined text, and outputs the content required by the user (i.e., the first feature representation of the drama segment).

[0200] Next, referring to FIG. 13, the process of obtaining the first feature representation by inputting the drama segment and the first combined text into the multi-modal large model will be explained. As shown in FIG. 13, the drama segment is shown in the six-grid image in the input picture of FIG. 13.

[0201] Correspondingly, the text information of the drama segment includes dialogue information: [Please stay | I have a proposal | Today is a good day | Our two families are together | Let's take some photos for memory | That's right | It's an art student meeting | It's worth remembering | Let's take photos of the two kids first | They are busy | When we can't see them in the future | Let's look at the photos | Keep a memory | Good | Come on | Come on | No, you two are so awkward | Be closer | Be closer | It seems that we still need to teach by example | Sister Lu | Let's give them a demonstration | You | like this | Yes | this way].

[0202] The first prompt is "Please summarize the plot summary of this segment based on picture information and dialogue information". The first prompt and the text information are combined to obtain the combined text (the first combined text) as shown in FIG. 13.

[0203] The plot segment is input into the multi-modal large model together with the combined text. Based on the first prompt, the multi-modal large model determines that it needs to output specific content that should be a summary description of the plot segment. The multi-modal large model extracts and analyzes the semantic features of the image data and the text data of the plot segment and the dialogue information in the combined text, and generates target content based on the semantic features. Finally, the plot segment summary is output as: "This segment may be a gathering of two families at a restaurant, taking photos for memories, and Lucy and others trying to make the two children closer, but their interaction is awkward."

[0204] In related technologies, different models are used to extract features from the plot segment and convert them into corresponding labels. For example, after extracting the plot segment of FIG. 13, specific labels such as round table, chair, gathering, Lucy (person), etc. can be obtained. However, this cannot obtain the complete semantics of the plot segment, which is not conducive to subsequent matching with promotion demand information. In the above steps 1210 and 1220, the multi-modal large model can directly process the image data of the plot segment and the text data of the first combined text. Moreover, the first combined text combining the first prompt and the text information is used to teach the multi-modal large model, so that the multi-modal large model can accurately output the first feature representation of the plot segment required by the user. Compared with the single label extraction of the plot segment in traditional technologies, the present embodiment uses the first prompt to guide the multi-modal large model to perform high-level semantic extraction on the plot segment in the dimension of plot segmentation, and obtains the first feature representation containing the overall semantic information of the plot segment. Since the first feature representation contains the overall semantic information of the plot segment, the matching degree accuracy can be improved when matching with promotion demand information in the subsequent, and a selected video segment that is more suitable for the promotion demand information can be obtained.

[0205] In an embodiment, referring to FIG. 14, step 1220 includes:

[0206] Step 1410, a plurality of first representative frames are extracted from the plot segment, and a first representative frame vector is generated based on the first representative frames.

[0207] Step 1420, the plurality of first representative frame vectors are converted into first features suitable for input into the multi-modal large model.

[0208] Step 1430, based on the first combined text, a second feature suitable for input into the multi-modal large model is generated.

[0209] Step 1440, the first feature and the second feature are jointly input into the multi-modal large model to obtain a first feature representation representing the plot segment.

[0210] The steps 1410 to 1440 are described in detail below.

[0211] In step 1410, the first representative frames refer to the video frames extracted from the drama segment. Illustratively, N video frames are uniformly extracted from the drama segment as N first representative frames, where N is a natural number. In another example, the drama segment can be preliminarily divided by hierarchical clustering, and then the histogram features of the preliminarily extracted key frames are compared to remove redundant frames in combination with the k-means clustering algorithm, to finally determine the sequence of first representative frames of the drama segment.

[0212] Illustratively, N video frames are uniformly extracted from the drama segment as N first representative frames, where N is a natural number. In another example, the specific operation of preliminarily dividing the drama segment by hierarchical clustering is as follows: first, the similarity between each video frame in the drama segment is calculated (for example, the similarity is calculated based on color histogram, feature points, etc.), and a distance matrix is constructed according to the similarity. Then, using a hierarchical clustering algorithm (such as agglomerative hierarchical clustering), starting from each video frame as a separate cluster, gradually merging clusters with high similarity until a preset number of clusters is reached or other stopping conditions are met. For each cluster obtained by preliminary division, a representative key frame is selected.

[0213] Then, the histogram features of the preliminarily extracted key frames are compared to remove redundant frames in combination with the k-means clustering algorithm. The specific steps are as follows: convert the preliminarily extracted key frames into color histograms, and use these histograms as data points. Set the value of k (the number of clusters), and randomly initialize k cluster centers. Calculate the distance (such as Euclidean distance) from each data point to each cluster center, and assign the data point to the cluster where the nearest cluster center is located. Update the cluster center of each cluster to be the mean of all data points in the cluster. Repeat the above assignment and update steps until the cluster centers no longer change significantly or the maximum number of iterations is reached. Finally, select a most representative key frame from each cluster as a first representative frame, thereby determining the sequence of first representative frames of the drama segment.

[0214] The process of extracting second representative frames from the shot segment is the same as the process of extracting first representative frames described above. Similarly, first preliminarily divide by hierarchical clustering, and then remove redundant frames in combination with the k-means clustering algorithm to finally determine the sequence of second representative frames of the shot segment.

[0215] The first representative frame vector is a vector obtained by feature extraction of the first representative frame. Illustratively, a video image feature model such as Contrastive Language-Image Pre-training (CLIP) can be used to convert multiple first representative frames into multiple d-dimensional feature vectors, i.e., multiple first representative frame vectors.

[0216] In step 1420, the first feature can be obtained by converting the first representative frame vector through the cross-modal matching module. The first feature is data that can be understood by the multi-modal large model.

[0217] Exemplarily, the cross-modal matching module adopts a network based on a transformer structure. The network calculates the degree of association between each element and other elements in the first representative frame vector through a self-attention mechanism, captures long-distance dependencies within the vector, and maps the first representative frame vector to the first feature that can be understood by the multi-modal large model. Alternatively, the cross-modal matching module adopts a multilayer perceptron (MLP) to perform nonlinear transformation on the first representative frame vector through multiple fully connected layers, and converts it into the first feature.

[0218] In step 1430, the second feature refers to feature extraction on the first combined text to obtain feature data that can be understood by the multi-modal large model. The first combined text can be processed by word segmentation to convert each word into a corresponding word vector, and then a pre-trained language model can be used to encode the word vectors to obtain a feature representation of the first combined text, i.e., the second feature suitable for input to the multi-modal large model.

[0219] In step 1440, the multi-modal large model can use LLaVa, InternVL, etc. LLaVA (Large Language and Vision Assistant) is a new type of end-to-end trained large multi-modal model, which aims to combine a visual encoder and a large language model to achieve comprehensive understanding and generation of images and text. InternVL is a multi-modal large language model (MLLM) that aims to expand a visual base model and align it with a large language model to efficiently handle multiple visual-language tasks.

[0220] The process of obtaining a first feature representation by inputting a multi-modal large model with a plot segment and a first combined text will be explained below in conjunction with FIG. 15.

[0221] As shown in FIG. 15, an example of a pair of plot segments and corresponding first combined texts is shown. First, a plurality of video frames are uniformly extracted from the plot segments as first representative frames. The plurality of first representative frames are input into a video image feature model for feature extraction to obtain a plurality of first representative frame vectors. The plurality of first representative frame vectors are input into a cross-modal matching module for modal conversion to complete semantic mapping of the plurality of first representative frame vectors to obtain more compact first features.

[0222] While processing the first representative frame, feature extraction is performed based on the first combined text to obtain second features suitable for input into the multi-modal large model. It should be noted that in this example, the multi-modal large model is capable of understanding the semantics of text modal data, and therefore maps the image features corresponding to the plot segment to feature data of the text modal.

[0223] The first features and the second features are jointly input into the multi-modal large model. The multi-modal large model is capable of combining the semantics corresponding to the plot segment and the semantics corresponding to the first combined text to perform high-level semantic abstraction on the plot of the plot segment, and obtain a first feature representation of the plot segment.

[0224] The related art is to separately train multiple models on sub-tasks, and extract different labels from videos based on different models to match promotion demand information, which has the problems of weak generalization ability and insufficient plot understanding of videos. In the steps 1410 to 1440, the multi-modal large model is used to combine the input plot segment and the first combined text to output the first feature representation of the plot segment. Since the multi-modal large model has strong semantic understanding ability, it can better understand the plot context, emotion, and context information in the plot segment under the prompt of the first combined text, and can have a deeper understanding of the semantics of the plot segment. Thus, the first feature representation output by the multi-modal large model can completely represent the deep semantics of the corresponding plot segment, and matching the first feature representation with the promotion demand information is conducive to finding selected video segments that better meet the promotion demand information, and improving the effect of promotion content embedding.

[0225] In an embodiment, referring to FIG. 16, step 430 includes:

[0226] Step 1610, identifying a public figure in the plot segment from the video.

[0227] Step 1620, obtaining public figure information of the public figure.

[0228] Step 1630, jointly inputting the plot segment and the public figure information into the segment representation prediction model to obtain a first feature representation representing the plot segment.

[0229] The steps 1610 to 1630 are described in detail below.

[0230] In step 1610, the public figure refers to an actor playing a role in the plot segment. Illustratively, a face recognition model can be used to identify the public figure in the plot segment.

[0231] In step 1620, the public figure information refers to the name, appearance time, position, and the like of the actor.

[0232] In step 1630, the first feature representation in this embodiment includes semantic information of the plot summary corresponding to the plot segment, and also includes semantic information corresponding to the public figure.

[0233] Next, referring to FIG. 17, the process of obtaining the first feature representation combined with the public figure information is explained. As shown in FIG. 17, each video selected for promotion content placement is preprocessed. The preprocessing includes segmenting each video to obtain a plurality of plot segments, such as segment 1 to segment n in the figure, and identifying public figures in each video to obtain public figure information, such as public figure 1 to public figure n in the figure. In other examples, public figure identification can also be performed on each plot segment to obtain public figure information corresponding to each plot segment.

[0234] The segment 1 to segment n and the corresponding public figure 1 to public figure n are input into the segment representation prediction model in pairs. The segment representation prediction model outputs the first feature representation corresponding to each segment, such as plot 1 to plot n corresponding to segment 1 to segment n in the figure. In this example, the first feature representation of the plot segment can be a description text of the plot summary of the plot segment, or a feature vector corresponding to the plot summary of the plot segment. In addition, the plot 1 to plot n in this example also carries public figure information. Therefore, in the subsequent process of matching the plot segment with the promotion requirement information, the public figure can also be matched as a matching factor.

[0235] If the first feature representation of the plot segment is directly predicted by the segment representation prediction model, the segment representation prediction model may not be able to accurately identify the public figure appearing in the plot segment, so that the first feature representation may not include information related to the public figure. In steps 1610 to 1630 described above, the public figure in the plot segment is identified from the video, and the information corresponding to the public figure is obtained. The public figure information and the plot segment are input into the segment representation prediction model to obtain the first feature representation of the plot segment, so that the first feature representation can accurately carry information related to the public figure appearing in the plot segment. When the promotion requirement information involving a specified public figure (such as a spokesperson advertisement) is involved, the public figure can also be used as a matching factor in the matching process. Therefore, the selected video segment corresponding to the promotion requirement information of the specified public figure can be accurately identified, and the embedding effect of the promotion content can be improved.

[0236] In an embodiment, the segment representation prediction model is a multi-modal large model; referring to FIG. 18, step 1630 includes:

[0237] Step 1810, combine the public figure information into the second prompt to obtain the second combined text.

[0238] Step 1820, input the plot segment and the second combined text into the multi-modal large model together to obtain the first feature representation representing the plot segment.

[0239] The steps 1810 and 1820 are described in detail below.

[0240] In step 1810, the second prompt is the text information input by the user to indicate that the plot segment represented by the first feature representation output by the segment representation prediction model meets the user's needs. In this embodiment, the second prompt at least includes information indicating that the first feature representation output by the segment representation prediction model needs to include public figure information. In some examples, the second prompt can be a pre-set templated prompt. In other examples, the second prompt can also be a prompt input by the user on the spot based on the plot segment.

[0241] For example, the second prompt is "Please summarize the plot of this segment." At this time, the segment representation prediction model will output an abstract description containing public figure information as the second feature representation under the guidance of the second prompt.

[0242] The second combined text is the text information obtained by combining the public figure information with the second prompt in a natural language manner. It enables the multi-modal large model to include public figure information when outputting the first feature representation of the plot segment, so as to consider the public figure factor in subsequent matching. For example, the public figure information includes the specific actor's name, and is combined with the second prompt "Please summarize the plot of this segment". The second combined text is "The public figure information in the plot segment includes actress A, actor A, middle-aged actress B, and middle-aged actor B. Please summarize the plot of this segment in combination with the picture information and the public figure information."

[0243] In step 1820, the multi-modal large model can integrate and process input data from different modalities (such as text, image, audio, etc.), and output corresponding content based on user needs.

[0244] After inputting the plot segment and the second combined text into the multi-modal large model, the multi-modal large model will perform feature extraction and feature conversion on the plot segment, mapping the plot segment into features that the multi-modal large model can process. The multi-modal large model will also perform feature extraction on the second combined text to obtain corresponding features. At this time, the features corresponding to the plot segment and the features corresponding to the second combined text belong to the same modality.

[0245] The multi-modal large model further analyzes and processes the plot segment and the second combined text based on the semantics of the second prompt in the second combined text, and outputs the content required by the user (i.e., the first feature representation containing public figure information).

[0246] The process of obtaining the first feature representation combined with the public figure information is explained below in connection with FIG. 19. As shown in FIG. 19, the plot segment is as shown in the six-grid image in the input picture of FIG. 19.

[0247] Correspondingly, the public figure information of the plot segment includes actress A, actor A, middle-aged actress B, and middle-aged actor B.

[0248] The second prompt is “Please summarize the plot of this segment in combination with the picture information and the public figure information”. In combination with the second prompt and the public figure information, the combined text (second combined text) as shown in FIG. 19 is obtained: “This is a plot segment in a film or television series. The public figures in the segment include actress A, actor A, middle-aged actress B, and middle-aged actor B. Please summarize the plot of this segment in combination with the picture information and the public figure information”.

[0249] The plot segment and the second combined text are input into the multi-modal large model. Based on the second prompt, the multi-modal large model determines that the specific content it needs to output should be the summary description corresponding to the plot segment, and needs to include the public figure information. The multi-modal large model obtains semantic features corresponding to image data and text data by feature extraction and analysis of the plot segment and the public figure information in the second combined text, and generates target content based on the semantic features. Finally, the plot segment summary is output as “This segment may be a gathering of two families at a restaurant, taking photos for memories, and middle-aged actress B and middle-aged actor B and others try to make actress A and actor B closer, but their interaction is awkward”.

[0250] If the public figure of the video is directly matched as a label alone, that is, as long as the figure appears in the video, it is considered to be matched successfully, there may be a situation in actual application that the scene in which the figure exists in a certain video segment is not suitable for embedding promotional content. In steps 1810 and 1820, the public figure information is combined with the second prompt to generate the second combined text. The second combined text and the plot segment are jointly input into the segment representation prediction model, so that the public figure information is included in the output first feature representation. In this way, when matching a segment with a specified public figure promotional demand information (such as a celebrity endorsement), the plot semantic and the public figure in the first feature representation can be matched together, avoiding the situation that the public figure is matched but the plot of the selected video segment is not suitable for embedding promotional content, improving the acceptance of users to embedded promotional content, and avoiding discomfort.

[0251] Detailed description of step 450

[0252] In step 450, a first matching degree between the promotion demand information and the first feature representation of each plot segment is determined, and based on the first matching degree, a selected video segment is determined from the plurality of plot segments.

[0253] The first feature representation can be in different forms. In an embodiment, the first feature representation is a first text description of the plot segment. Referring to FIG. 20, step 450 includes:

[0254] Step 2010, extracting a plurality of first keywords from the promotion demand information.

[0255] Step 2020, for each first keyword, determining a first similarity between the first keyword and the first text description.

[0256] Step 2030, based on the first similarity between each first keyword and the first text description, determining a first matching degree between the promotion demand information and the first text description.

[0257] When the first feature representation is in other forms, subsequent processing methods will be provided (for example, the case where the first feature representation is a first deep feature is described in subsequent embodiments).

[0258] The steps 2010 to 2030 are described in detail below.

[0259] In step 2010, the first keyword refers to a keyword representing the key information of the promotion demand information after the redundant words in the promotion demand information are removed by preprocessing.

[0260] In step 2020, the first similarity refers to the TF-IDF value of the first keyword with respect to the first text description. The first similarity is used to represent the importance of the corresponding first keyword to the first text description.

[0261] In step 2030, the first matching degree is calculated based on the TF-IDF value of each first keyword in the promotion demand information, and is used to represent the matching degree between the promotion demand information and the first text description.

[0262] The process of determining the first matching degree is explained in combination with FIG. 21. As shown in FIG. 21, the first keyword 1 to the first keyword n are extracted from the promotion demand information. The first similarity between each first keyword and the first text description is calculated, and the first similarity 1 to the first similarity n are obtained. In this example, the first similarity is the TF-IDF value of the first keyword with respect to the first text description.

[0263] The first keyword 1 to the first keyword n of the promotion demand information and the corresponding first similarity 1 to the first similarity n are converted into a vector form. Each first similarity constitutes a dimension, and the first similarity 1 to the first similarity n are combined into a document vector of the promotion demand information.

[0264] The document vector of the promotion demand information is calculated and the document vector of the first text description , and the cosine similarity is taken as the first matching degree of the promotion demand information and the first text description. The calculation formula of the cosine similarity cos θ is wherein denotes the dot product of vectors and , respectively. and denote the modulus of vectors and , respectively. The first matching degree takes a value range of [-1, 1], and the value closer to 1 indicates that the promotion demand information is more matched with the first text description.

[0265] The related technology is to label the video, match the promotion demand information based on each label, and has the problems of insufficient flexibility and poor matching accuracy. In the steps 2010 to 2030, the first text description capable of representing the semantic of the complete plot segment is matched with the promotion demand information in the document level semantic similarity. When matching, more semantic details can be considered, so that the accuracy of the matching can be improved. And since the matching is based on semantics, not fixed labels, the flexibility of the matching is higher.

[0266] In an embodiment, referring to FIG. 22, the step 2010 includes:

[0267] Step 2210, tokenizing the promotion demand information.

[0268] Step 2220, identifying and removing stop words from the tokenized words.

[0269] Step 2230, removing non-entity words from the words after removing the stop words to obtain a plurality of first keywords.

[0270] The steps 2210 to 2230 are described in detail below.

[0271] In step 2210, word segmentation refers to dividing each word in the promotion demand information to obtain a plurality of words. A rule-based word segmentation method can be used, for example, a forward maximum matching method, which matches the text of the promotion demand information with the words in the dictionary from left to right, and matches the longest word as much as possible; a statistical-based word segmentation method can also be used, such as Hidden Markov Model (HMM) or Conditional Random Field (CRF), which trains a large amount of text data to learn the probability distribution and transition probability of the word, thereby realizing word segmentation; a deep learning-based word segmentation method can also be used, such as a model based on recurrent neural network (RNN) and its variants (LSTM, GRU) or Transformer architecture.

[0272] In step 2220, stop words refer to common conjunctions, prepositions, articles, etc., such as “of”, “ed”, “is”, “in”, etc. Stop words are important for the promotion demand information and represent specific meanings, so they need to be removed to improve the accuracy of subsequent semantic matching.

[0273] For example, stop words can be identified by a pre-set stop word list. The stop word list can be derived from a general natural language processing stop word library, which usually contains conjunctions, prepositions, articles, etc. in common languages that have no actual semantic contribution. The general stop word list can also be extended and customized according to specific business scenarios and the characteristics of the promotion demand information. When building the stop word list, a large amount of text data related to the promotion content can be collected, and the words with high frequency but no key role in semantic understanding can be counted and added to the stop word list. At the same time, reference can be made to professional stop word lists in related fields to further improve the stop word list.

[0274] In step 2230, non-entity words refer to words without actual meaning. To determine whether a word is a non-entity word, a part-of-speech tagging method can be used to tag the segmented words with parts of speech, such as nouns, verbs, adjectives, etc. Generally, words with parts of speech such as mood words, auxiliary words, and interjection words are non-entity words. Semantic knowledge bases such as WordNet can also be used to query the semantic information of the word. If a word has no clear semantic direction in the knowledge base or is irrelevant to the core semantics of the promotion content, it can be determined as a non-entity word. Business rules can also be used to predefine some characteristics or patterns of non-entity words, such as specific symbols or meaningless strings, and words that meet these characteristics or patterns can be determined as non-entity words. In this step, the plurality of first keywords refer to the remaining words after removing the stop words and non-entity words from the segmented words in the promotion demand information.

[0275] The process of extracting the first keyword from the promotion demand information is explained below in combination with FIG. 23. As shown in FIG. 23, it is assumed that the promotion demand information corresponds to "The product is a TV box, and needs to be placed in a family watching TV segment." First, the promotion demand information is segmented, and the segmented words are "product", "is", "TV box", ",", "need", "place", "in", "a family", "watch", "TV", "of", "segment", "in", ".".

[0276] The stop words in the segmented words are identified and removed, and the words after removing the stop words are "product", "TV box", ",", "need", "place", "a family", "watch", "TV", "segment", ".".

[0277] In the words after removing the stop words, the punctuation and non-entity words without actual meaning are removed, and the first keywords are "TV box", "place", "a family", "watch", "TV".

[0278] It should be noted that the above example is only for the convenience of understanding the process of extracting the first keywords from the promotion demand information, and the assumed example does not limit the actual implementation.

[0279] If the promotion demand information is directly matched with the first text description, the useless words contained therein can easily interfere with the matching result. In steps 2210 to 2230, the promotion demand information is preprocessed to remove useless stop words and non-entity words, and the first keywords are obtained. Based on the first keywords and the first text description, the noise and redundant information can be effectively reduced, and the accuracy of the matching can be improved.

[0280] In an embodiment, referring to FIG. 24, step 2020 includes:

[0281] Step 2410, for each first keyword, determining a first word frequency of the first keyword appearing in the first text description.

[0282] Step 2420, determining a first inverse document frequency of the first keyword appearing in the plurality of first text descriptions.

[0283] Step 2430, determining a first similarity based on the first word frequency and the first inverse document frequency.

[0284] The steps 2410 to 2430 are described in detail below.

[0285] In step 2410, the first term frequency refers to the number of times the first keyword appears in the first text description. The first term frequency is used to measure the frequency of the first keyword appearing in the first text description. The higher the first term frequency, the more important the first keyword is in the first text description.

[0286] In step 2420, the first inverse document frequency is used to represent the universality of the first keyword in the plurality of first text descriptions. The higher the first inverse document frequency, the fewer the number of first text descriptions in which the corresponding first keyword appears, and the more important the first keyword is.

[0287] In step 2430, the first similarity is the TF-IDF value of the corresponding first keyword. The first similarity is equal to the product of the first term frequency and the first inverse document frequency.

[0288] For example, assume that a certain first keyword appears 3 times in a certain first text description, and the first text description has a total of 30 words; at this time, there are 20 first text descriptions in total, of which 4 first text descriptions appear the first keyword.

[0289] Correspondingly, the first term frequency of the first keyword with respect to the first text description is equal to the number of times the first keyword appears divided by the total number of words in the first text description, i.e., the first term frequency = 3 / 30 = 0.1.

[0290] The first inverse document frequency of the first keyword is equal to the total number of first text descriptions divided by the number of first text descriptions in which the first keyword appears, i.e., the first inverse document frequency = 20 / 4 = 5.

[0291] Finally, the first similarity of the first keyword is equal to the product of the first term frequency and the first inverse document frequency, i.e., the first similarity = 0.1*5 = 0.5.

[0292] It should be noted that the above example is for the convenience of understanding the process of calculating the first similarity by assuming numerical values, and does not represent a limitation on actual implementation.

[0293] Compared to the general matching of keywords according to promotion demand information with tags extracted from videos, in steps 2410 to 2430 above, the first similarity (TF-IDF value) of each first keyword of the promotion demand information is calculated as the basis for matching the promotion demand information with the first text description. Using the TF-IDF value as the first similarity, the relevance of the first text description and the first keyword can be accurately identified, and those words that frequently appear in certain documents but are not important to the overall corpus can be distinguished.

[0294] In an embodiment, referring to FIG. 25, step 2410 includes:

[0295] Step 2510, determining, for each first keyword, a first number of times the first keyword appears in the first text description.

[0296] Step 2520, extracting second keywords from the first text description and determining a second keyword number of the second keywords.

[0297] Step 2530, determining a first term frequency based on the first number and the second keyword number.

[0298] The steps 2510 to 2530 are described in detail as follows.

[0299] In step 2510, the first number refers to the number of times the corresponding first keyword appears in a first text description.

[0300] In step 2520, the second keyword refers to a keyword extracted from the first text description after data preprocessing. The data preprocessing includes tokenization, stop word recognition, stop word removal, non-entity word removal, stem extraction, etc. The second keyword number refers to the total number of second keywords extracted from the first text description.

[0301] Specifically, first, the first text description is preprocessed, including tokenization, stop word recognition, stop word removal, non-entity word removal, stem extraction, etc. Then, the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm can be used to extract the second keyword. The term frequency (TF) of each preprocessed word in the first text description and the inverse document frequency (IDF) in the entire text set are calculated, and the TF and IDF are multiplied to obtain the TF-IDF value. The word with a higher TF-IDF value is selected as the second keyword. The TextRank algorithm can also be used, treating the words in the text as nodes in a graph, constructing a graph according to the co-occurrence relationship between the words, and calculating the importance score of the nodes through iteration, and selecting the word with a higher score as the second keyword. The second keyword number of the second keyword is determined, that is, the total number of extracted second keywords is counted.

[0302] In step 2530, the first term frequency is equal to the first number divided by the second keyword number.

[0303] Since the first keyword is also a keyword extracted from the promotion demand information after data preprocessing, the word in the first text description that is the same as the first keyword should also be extracted as a second keyword.

[0304] The process of determining the first word frequency is explained below in combination with FIG. 26. As shown in FIG. 26, it is assumed that the first keyword 1 to the first keyword n are extracted from the promotion demand information, and 30 second keywords (the second keyword 1 to the second keyword 30) are extracted from the first text description.

[0305] Taking the first keyword 1 as an example, the number of times that the first keyword 1 appears in the second keyword 1 to the second keyword 30 is determined. It is assumed that the first keyword 1 appears 3 times. Correspondingly, the first word frequency 1 of the first keyword 1 is 3 / 30 = 0.1.

[0306] Similarly, the first word frequency 2 to n of the remaining first keyword 2 to the first keyword n relative to the first text description is obtained.

[0307] If the first word frequency of the first keyword is directly determined based on the first text description, the first word frequency may be small due to the existence of stop words and non-useful words in the first text description. In the steps 2510 to 2530, the first word frequency is determined based on the extracted second keywords and the first keyword. The unnecessary noise words in the first text description are removed, and the numerical accuracy of the first word frequency is improved.

[0308] In an embodiment, referring to FIG. 27, the step 2420 includes:

[0309] The step 2710 determines the first number of the first text description containing the first keyword.

[0310] The step 2720 obtains the total number of the first text description.

[0311] The step 2730 determines the first inverse document frequency based on the first number and the total number of the first text description.

[0312] The steps 2710 to 2730 are described in detail below.

[0313] In the step 2710, the first number refers to the number of the first text description containing the first keyword in the plurality of first text descriptions.

[0314] In the step 2720, the total number of the first text description refers to the number of the first feature representation (the first text description) of each plot segment output by the plot representation prediction model in the current promotion content processing process.

[0315] In the step 2730, the first inverse document frequency is equal to the total number of the first text description divided by the first number.

[0316] Next, referring to FIG. 28, the process of determining the first inverse document frequency is explained. As shown in FIG. 28, in a certain promotion content processing process, at least one video to be embedded with promotion content is preprocessed to obtain a plurality of plot segments (segment 1 to segment n) and a plurality of text information (text 1 to text n). Then, segment 1 to segment n and text 1 to text n are respectively input into the segment representation prediction model in a one-to-one correspondence to obtain the first text description corresponding to each plot segment (plot 1 to plot 20).

[0317] Suppose n is 20, that is, the total number of first text descriptions is 20 (plot 1 to plot 20).

[0318] The promotion demand information is keyword extracted to obtain first keyword 1 to first keyword m. Taking first keyword 1 as an example. Suppose that first keyword 1 is currently contained in plot 1, plot 3, plot 5, and plot 17, that is, the first number is 4. Then the first inverse document frequency of first keyword 1 = 20 / 4 = 5.

[0319] Similarly, the first inverse document frequencies 2 to m of the remaining first keywords 2 to m are obtained.

[0320] If only the word frequency of the first keyword in each first text description is used as the basis for matching the promotion demand information and the first text description, since the first keyword can be a commonly used word in the first text description, it will appear in multiple first text descriptions. At this time, the matching result will not be accurate enough. In steps 2710 to 2730, by calculating the first inverse document frequency of the first keyword, the universality of the first keyword in the plurality of first text descriptions can be accurately measured. Then, the first inverse document frequency of the first keyword is combined with the first word frequency to obtain the first similarity of the first keyword and the first text description. Using the first similarity as the basis for matching the promotion demand information and the first text description can improve the accuracy of the matching.

[0321] In an embodiment, referring to FIG. 29, step 2710 includes:

[0322] Step 2910, extracting second keywords from each first text description.

[0323] Step 2920, taking each extracted second keyword as a first index, storing the first index and the first text description identifier of each first text description containing the second keyword into a first index correspondence table in correspondence.

[0324] Step 2930, taking the first keyword as a first index, searching the first index correspondence table to obtain the first text description identifier stored in correspondence with the first index, thereby determining the first number of first text descriptions containing the first keyword.

[0325] The steps 2910, 2920, and 2930 are described in detail as follows.

[0326] In the step 2920, the first index is an index value used to represent the first text description identifier of the corresponding at least one first text description, and the specific value is the second keyword extracted from each first text description. By constructing the first index correspondence table, the first index can be used to quickly query the first text description containing a specific keyword.

[0327] In the embodiment, the specific value of the first index is the corresponding second keyword. The first index correspondence table is a data storage structure composed of a plurality of first indexes and corresponding first text description identifiers. It is stored in the form of an inverted index, which can improve the efficiency of querying the first text description containing a specific keyword, thereby determining the first number of first text descriptions containing the first keyword.

[0328] The first text description identifier is an identifier used to indicate the first text description. Exemplarily, the first text description identifier can be represented by a specific numerical value, or can be formed by one or more combinations of letters, numbers, and symbols.

[0329] In the step 2930, using the first keyword as the first index means setting the specific value of the first index to the first keyword. Thus, the same index is found in the first index correspondence table, and the corresponding stored first text description identifiers of the same index in the first index correspondence table are obtained. The number of these found first text descriptions is the first number.

[0330] Next, referring to FIG. 30, the process of determining the first number of first text descriptions containing the first keyword is explained.

[0331] As shown in FIG. 30, it is assumed that there are n first text descriptions, and m second keywords (second keyword 1 to second keyword m) are extracted from the first text description 1 to the first text description n. Each second keyword is used as a first index, and the first index 1 to the first index m are obtained correspondingly.

[0332] It is assumed that the first text description 1, the first text description 3, and the first text description 4 all contain the second keyword 1. Then, the mapping relationship between the first index 1 and the first text description identifier 1, the first text description identifier 3, and the first text description identifier 4 is constructed and stored in the first index correspondence table. Similarly, the at least one first text description identifier corresponding to each of the remaining first indexes 2 to the first index m is obtained, and finally the completed first index correspondence table is obtained.

[0333] The first keyword 1 is taken as the first index 1_1, and the first index correspondence table is looked up. Assuming that the first index 1_1 is the same as the first index 2, the first text description identifier 6 and the first text description identifier 8 corresponding to the first index 2 are obtained based on the mapping relationship. Thus, it is determined that the first text description 6 and the first text description 8 include the first keyword 1, and the first number of the first text descriptions containing the first keyword is 2.

[0334] If each first keyword is compared with each first text description to determine the number of first text descriptions containing the first keyword, the query efficiency will be low. In steps 2910 to 2930, the first index correspondence table of the inverted index structure is constructed. Each first keyword only needs to be compared with each first index of the first index correspondence table, and the first text description identifier of all the first text descriptions containing the first keyword can be quickly queried, so that the first number is quickly determined. Based on this, the matching efficiency of the promotion demand information and the first text description can be improved.

[0335] In an embodiment, the first feature representation is a first deep feature of a plot segment. Referring to FIG. 31, step 450 includes:

[0336] Step 3110, inputting the promotion demand information into the segment representation prediction model to obtain a second deep feature of the promotion demand information.

[0337] Step 3120, calculating a first matching degree based on the second deep feature and the first deep feature of each plot segment.

[0338] The steps 3110 and 3120 are described in detail below.

[0339] In step 3110, the second deep feature refers to a feature vector obtained after the promotion demand information is extracted by the segment representation prediction model.

[0340] In step 3120, the first deep feature refers to a first feature representation in the form of a feature vector obtained by the segment representation prediction model based on a plot segment and corresponding text information.

[0341] There is a first matching degree between the promotion demand information and each plot segment. The first matching degree is a numerical value calculated based on the first deep feature of a plot segment and the second deep feature of the promotion demand information, and is used to represent the matching degree between the promotion demand information and the plot segment.

[0342] For example, the first matching degree can be obtained by calculating the cosine similarity of the first deep feature vector and the second deep feature vector The calculation formula of the cosine similarity cosα is wherein denotes the dot product of vectors and denotes the norm of vector and denotes the norm of vector and In other examples, the first matching degree can also be obtained by calculating the Euclidean distance between the first deep feature vector and the second deep feature vector The formula for calculating the Euclidean distance d is wherein E i and F i denote the i-th element of vectors and respectively, and n denotes the dimension of the vector.

[0343] The process of calculating the first matching degree based on the first deep feature and the second deep feature will be explained below in conjunction with FIG. 32. As shown in FIG. 32, each video selected for which the content needs to be promoted is preprocessed. The preprocessing includes segmenting each video to obtain multiple plot segments, such as segment 1 to segment n in the figure, and performing text extraction on each video to obtain text information containing dialogue information, and segmenting the text information to obtain text 1 to text n corresponding to segment 1 to segment n. In other examples, text extraction can also be performed on each plot segment to obtain text information corresponding to each plot segment.

[0344] Segment 1 to segment n and corresponding text 1 to text n are input into the segment representation prediction model in pairs. The segment representation prediction model outputs the first feature representation corresponding to each segment, such as first deep feature 1 to first deep feature n corresponding to segment 1 to segment n in the figure.

[0345] The promotion requirement information is input into the segment representation prediction model to obtain the corresponding second deep feature. The second deep feature is compared with the first deep feature 1 to the first deep feature n to obtain the first matching degree 1 to the first matching degree n. The similarity calculation can be the calculation of the cosine similarity between the first deep feature and the second deep feature, or the calculation of the Euclidean distance between the first deep feature and the second deep feature.

[0346] Finally, based on the first matching degree, at least one plot segment matching the promotion requirement information is selected as the selected video segment.

[0347] In other examples, each plot segment can also be input into the segment representation prediction model. The segment representation prediction model can perform representative frame selection, representative frame feature vector extraction, feature modality conversion, etc. on each plot segment to obtain the first feature of each plot segment (for details, refer to the corresponding implementation details of steps 1410 and 1420 provided in the above embodiments).

[0348] The promotion demand information is then input into the segment representation prediction model to obtain the corresponding second deep feature. The second deep feature is subjected to similarity calculation with the first feature of each plot segment to obtain the first matching degree between the promotion demand information and each plot segment.

[0349] In steps 3110 and 3120, the feature vector corresponding to the plot segment and the feature vector corresponding to the promotion demand information are subjected to first matching degree calculation. Compared with the text similarity calculation between the first text description corresponding to the plot segment and the promotion demand information, the first matching degree calculation based on cross-modal features can cover some details of visual information ignored by the text description and recall more possible selected video segments.

[0350] In an embodiment, referring to FIG. 33, step 450 includes:

[0351] Step 3310: inputting the split shot segment into the segment representation prediction model to obtain the second feature representation of the split shot segment.

[0352] Step 3320: determining the second matching degree between the promotion demand information and the second feature representation of each split shot segment.

[0353] Step 3330: determining the selected video segment from the plurality of plot segments and the plurality of split shot segments based on the first matching degree and the second matching degree.

[0354] Steps 3310 to 3330 are described in detail below.

[0355] In step 3310, the second feature representation refers to the information output by the segment representation prediction model after analyzing and predicting the input split shot segment, which is used to represent the detailed description of the content corresponding to the split shot segment, i.e., the semantic information corresponding to the split shot segment. Exemplarily, the second feature representation can be a textual description composed of natural language, or an abstract expression composed of a feature vector.

[0356] In step 3320, the second matching degree refers to the semantic similarity between the promotion demand information and the second feature representation of the split shot segment. The higher the similarity, the higher the second matching degree.

[0357] In step 3330, the corresponding first matching degree meets the requirements of the plot segment or the corresponding second matching degree meets the requirements of the shot segment as the selected video segment. The selected video segment can only include the plot segment that meets the requirements, only include the shot segment that meets the requirements, or include both the shot segment and the plot segment that meet the requirements.

[0358] Here, "meets the requirements" can have multiple cases. For example, the plot segment with the first matching degree greater than the preset first matching degree threshold value can be selected as the selected video segment, and the shot segment with the second matching degree greater than the preset second matching degree threshold value can be selected as the selected video segment.

[0359] The first matching degree threshold value and the second matching degree threshold value can be the same or different, and are set according to the actual implementation. In addition, the plot segments can be sorted in descending order of the first matching degree, and a certain percentage (such as the top x%) of the plot segments with high sorting can be selected as the selected video segment. The shot segments can be sorted in descending order of the second matching degree, and a certain percentage (such as the top y%) of the shot segments with high sorting can be selected as the selected video segment.

[0360] For example, the plot segments are sorted in descending order of the first matching degree, and the top 5% of the plot segments are selected as the first candidate selected video segment. The shot segments are sorted in descending order of the second matching degree, and the top 5% of the shot segments are selected as the second candidate selected video segment. The first candidate selected video segment and the second candidate selected video segment are integrated to obtain the final determined target candidate segment. The percentage can be set according to actual needs, and the percentages of the shot segments and the plot segments can be the same or different.

[0361] For example, the plot segments and the shot segments are sorted in descending order of the corresponding first matching degree and the second matching degree to obtain a candidate segment sequence. The top 10% of the segments in the candidate segment sequence are selected as the selected video segment.

[0362] For example, the plot segment with the largest first matching degree and the shot segment with the largest second matching degree are selected as the selected video segment.

[0363] The process of determining the selected video segment from multiple plot segments and multiple shot segments will be explained below in conjunction with FIG. 34.

[0364] As shown in FIG. 34, each video that needs to be promoted for content placement is preprocessed. The preprocessing includes:

[0365] Each video is segmented to obtain multiple plot segments, such as segments 1 to n in the figure.

[0366] The video is shot by shot to obtain a plurality of shot segments, such as shot 1 to shot m in the figure.

[0367] Text information containing dialogue information is obtained by performing text extraction processing on each video. The text information is segmented to obtain text 1 to text n corresponding to segment 1 to segment n. In other examples, text information corresponding to each plot segment can also be obtained by performing text extraction processing on each plot segment.

[0368] Segment 1 to segment n and corresponding text 1 to text n are input into the segment representation prediction model in pairs. The segment representation prediction model outputs first feature representations corresponding to each plot segment, such as plot summary 1 to plot summary n corresponding to segment 1 to segment n in the figure.

[0369] Shot 1 to shot m are input into the segment representation prediction model. The segment representation prediction model outputs second feature representations corresponding to each shot segment, such as shot details 1 to shot details m corresponding to shot 1 to shot m in the figure.

[0370] In other examples, text information corresponding to each shot segment can also be obtained by performing text extraction processing on each shot segment. Shot 1 to shot m and corresponding text information are input into the segment representation prediction model in pairs to obtain shot details 1 to shot details m (not shown in the figure).

[0371] The selected video segments are obtained by performing segment matching on the promotion demand information and the plot summary 1 to plot summary n and the shot 1 to shot m.

[0372] In related technologies, text extraction and video feature extraction are performed on a video respectively to obtain text content and video feature content. Then, the text content and the video feature content are labeled respectively to obtain each label. Compared with matching of video segments that can embed promotion content based on labels, in the above steps 3310 to 3330, the video is divided into plot segments and shot segments in the plot dimension and the shot dimension. Then, the first feature representations of the plot segments and the second feature representations of the shot segments are obtained through the segment representation prediction model. Finally, the selected video segments are obtained by performing segment matching on the promotion demand information and the semantics corresponding to the first feature representations and the semantics corresponding to the second feature representations. From different dimensions, the matching based on semantics can more accurately match the selected video segments corresponding to the promotion demand information.

[0373] In an embodiment, the segment representation prediction model is a multi-modal large model. Referring to FIG. 35, step 3310 includes:

[0374] Step 3510: inputting the split shot and the second prompt together into the shot representation prediction model to obtain the second feature representation representing the split shot.

[0375] Step 3510 is described in detail below.

[0376] In step 3510, the second prompt refers to text information input by the user to instruct the shot representation prediction model to output the second feature representation representing the split shot that meets the user's demand. In some examples, the second prompt can be a pre-set templated prompt. In other examples, the second prompt can also be a prompt input by the user on the spot based on the split shot.

[0377] For example, the second prompt is "This is a shot in a film or TV series. Please describe the picture in detail, including the clothing, action, expression and position of the characters in the picture, and describe the foreground, background and other important elements of the picture." At this time, the shot representation prediction model will output a detailed description of the corresponding split shot as the second feature representation under the guidance of the second prompt.

[0378] The process of obtaining the second feature representation by inputting the multi-modal large model according to the split shot and the second prompt is explained and described below in combination with FIG. 36. As shown in FIG. 36, the split shot is shown as an image in the input picture column in FIG. 36.

[0379] Correspondingly, the second prompt is "This is a shot in a film or TV series. Please describe the picture in detail, including the clothing, action, expression and position of the characters in the picture, and describe the foreground, background and other important elements of the picture." as shown in the prompt column in FIG. 36.

[0380] The split shot segment and the prompt are input into the multi-modal large model. Based on the second prompt, the multi-modal large model determines that it needs to output specific content that should be a detailed description corresponding to the split shot segment, and needs details such as clothing, action, expression, position, foreground, background, etc. The multi-modal large model extracts and analyzes the features of the split shot segment and the second prompt to obtain semantic features corresponding to the image data and the text data, and generates target content based on the semantic features. Finally, the detailed description of the split shot segment is output as: "The picture shows a mobile phone screen, and the screen displays a photo being taken. The photo has two characters: the left character is a male, wearing a dark suit, with a white shirt inside, smiling, with his right hand naturally hanging down and his left hand on the shoulder of the female next to him, showing intimacy and friendliness. The right character is a female, wearing a dark dress, with her left hand on the waist of the male next to her, showing intimacy and friendliness. The background is an indoor environment, which may be a restaurant or a family gathering place. There are tables and chairs in the background, and some tableware and food on the table, showing warmth and bustle. The walls in the background have some decorations, adding beauty to the environment. In the foreground, the photographer's hand can be seen, and the fingers are operating the mobile phone to take a photo."

[0381] In the related art, different models are used to extract features from the plot segment and convert them into corresponding labels. For example, after extracting the split shot segment in FIG. 36, specific labels such as hand, mobile phone, and camera can be obtained. However, this cannot obtain the complete semantics of the plot segment, which is not conducive to subsequent matching with promotion demand information. In the above step 3510, the multi-modal large model can directly process the image data of the split shot segment and the text data of the second prompt together. Moreover, the multi-modal large model is guided by the second prompt, so that it can accurately output the second feature representation of the split shot segment required by the user. Compared with the single label extraction of the split shot segment in the related art, the present embodiment uses the second prompt to guide the multi-modal large model to perform high-level semantic extraction on the split shot segment in the split shot dimension, obtaining a second feature representation containing detailed semantic information of the split shot segment. Therefore, when the second feature representation is matched with the promotion demand information in the subsequent step, the matching degree and accuracy can be improved, and the selected video segment that is more suitable for the promotion demand information can be obtained.

[0382] In an embodiment, referring to FIG. 37, step 3510 includes:

[0383] Step 3710, extracting a plurality of second representative frames from the split shot segment, and generating a second representative frame vector based on the second representative frames.

[0384] Step 3720, converting the plurality of second representative frame vectors into third features suitable for input into the multi-modal large model.

[0385] Step 3730, generating a fourth feature suitable for input into the multi-modal large model based on the second prompt.

[0386] Step 3740, inputting the third feature and the fourth feature into the multi-modal large model together to obtain a second feature representation representing the split shot.

[0387] The steps 3710 to 3730 are described in detail below.

[0388] In step 3710, the second representative frame refers to a video frame extracted from the split shot. Illustratively, N frames of video frames are uniformly extracted from the split shot as N second representative frames, where N is a natural number. In another example, the split shot can also be preliminarily divided by hierarchical clustering, and then the histogram features of the preliminarily extracted key frames are compared to remove redundant frames, and finally the sequence of second representative frames of the split shot is determined.

[0389] The second representative frame vector is a vector obtained by feature extraction of the second representative frame. Illustratively, a plurality of second representative frame vectors can be converted into a plurality of d-dimensional feature vectors by a video image feature model, such as a contrastive language-image pre-training (CLIP), from a plurality of second representative frames.

[0390] In step 3720, the third feature can be obtained by converting the second representative frame vector through the cross-modal matching module. The third feature is a deep feature data corresponding to the split shot that can be understood by the multi-modal large model.

[0391] Illustratively, if the cross-modal matching module adopts a network based on transformer structure, it will perform self-attention calculation on the second representative frame vector, learn the association between vector elements, and convert it into a third feature that can be understood by the multi-modal large model; if a multilayer perceptron (MLP) is used, the second representative frame vector is linearly and nonlinearly transformed through multiple fully connected layers to obtain the third feature.

[0392] In step 3730, the fourth feature refers to the feature data that can be understood by the multi-modal large model obtained by feature extraction of the second prompt. The second prompt can be first analyzed for word, and stop words are removed, and then the remaining words are converted into word vectors. Then, the pre-trained text encoder is used to process these word vectors, and the second prompt is encoded into a fourth feature that can be understood by the multi-modal large model.

[0393] The process of obtaining the second feature representation of the split shot by the multi-modal large model from the split shot and the second prompt is explained below in connection with FIG. 38.

[0394] As shown in FIG. 38, taking a pair of split shots and a corresponding second prompt as an example. First, a plurality of video frames are uniformly extracted from the split shots as second representative frames. The plurality of second representative frames are input into the video image feature model for feature extraction to obtain a plurality of second representative frame vectors. The plurality of second representative frame vectors are input into the cross-modal matching module for modal conversion, completing semantic mapping of the plurality of second representative frame vectors to obtain more compact third features.

[0395] While processing the second representative frames, feature extraction is performed based on the second prompt to obtain fourth features suitable for input into the multi-modal large model.

[0396] The third features and the fourth features are jointly input into the multi-modal large model. The multi-modal large model can combine the semantics corresponding to the split shots and the semantics corresponding to the second prompt to perform high-level semantic abstraction on the detailed content of the split shots to obtain a second feature representation of the split shots.

[0397] In related technologies, a plurality of models are trained separately on sub-tasks, and different labels are extracted from videos based on different models to match promotion demand information, which has weak generalization ability and insufficient understanding of video plots. In the steps 3710 to 3740, the multi-modal large model is used to combine the input split shots and the second prompt to output the second feature representation of the split shots. Since the multi-modal large model has strong semantic understanding ability, it can better understand the details such as characters, clothing, actions, expressions, positions, foregrounds, backgrounds, etc. in the split shots under the guidance of the second prompt, and can understand the semantics of the split shots more deeply. Thus, the second feature representation output by the multi-modal large model can completely represent the deep semantics of the corresponding split shots, and matching the second feature representation with the promotion demand information is conducive to finding selected video segments that better meet the promotion demand information, and improving the effect of embedding promotion content.

[0398] In an embodiment, the second feature representation is a second text description of the split shots; referring to FIG. 39, the step 3320 includes:

[0399] Step 3910, extracting a plurality of first keywords from the promotion demand information.

[0400] Step 3920, for each first keyword, determining a second similarity between the first keyword and the second text description.

[0401] Step 3930, based on the second similarity between each first keyword and the second text description, determining a second matching degree between the promotion demand information and the second text description.

[0402] Steps 3920 and 3930 are described in detail below.

[0403] In step 3920, the second similarity refers to the TF-IDF value of the first keyword relative to the second text description. The second similarity is used to characterize the importance of the corresponding first keyword to the second text description.

[0404] In step 3930, the second matching degree is calculated based on the TF-IDF value of each first keyword in the promotion demand information, and is used to characterize the degree of matching between the promotion demand information and the second text description.

[0405] The process of determining the second matching degree is explained below with reference to Figure 40. As shown in Figure 40, the first keyword 1 to the first keyword n ​​are extracted from the promotion demand information. The second similarity between each first keyword and the second text description is calculated, resulting in second similarity 1 to second similarity n. In this example, the second similarity is the TF-IDF value of the first keyword relative to the second text description.

[0406] The first keywords 1 to n and their corresponding second similarities 1 to n of the promotional demand information are converted into vector form. Each second similarity constitutes a dimension, and the second similarities 1 to n are combined to form the document vector of the promotional demand information.

[0407] Calculate the document vector of promotion demand information The document vector of the second text description The cosine similarity is used as the second matching score between the promotion demand information and the second text description. The calculation formula is: in Representing vectors and dot product, and Representing vectors respectively and The modulus. The second matching degree ranges from [-1, 1], and the closer the value is to 1, the better the promotion demand information matches the second text description.

[0408] In the related art, by tagging the video, matching based on each label and promotion demand information, there is a problem of insufficient flexibility and poor matching accuracy. In the above steps 3910 to 3930, the semantic level similarity is calculated between the document vector of the second text description capable of representing the detailed semantics of the shot segment and the document vector of the promotion demand information. When matching, more semantic details can be considered, thereby improving the accuracy of matching. And since it is based on semantic matching, not fixed labels, the flexibility of matching is higher.

[0409] In an embodiment, referring to FIG. 41, step 3910 includes:

[0410] Step 4110, tokenizing the promotion demand information.

[0411] Step 4120, identifying and removing stop words from the tokenized words.

[0412] Step 4130, removing non-entity words from the words after removing the stop words to obtain a plurality of first keywords.

[0413] It should be noted that the feature explanation, implementation details and beneficial effects of steps 4110 to 4130 in this embodiment all correspond to steps 2210 to 2230 in the above-mentioned embodiments, and will not be repeated here.

[0414] In an embodiment, referring to FIG. 42, step 3920 includes:

[0415] Step 4210, determining, for each first keyword, a second word frequency of the first keyword appearing in the second text description.

[0416] Step 4220, determining a second inverse document frequency of the first keyword appearing in the plurality of second text descriptions.

[0417] Step 4230, determining a second similarity based on the second word frequency and the second inverse document frequency.

[0418] The steps 4210 to 4230 are described in detail below.

[0419] In step 4210, the second word frequency refers to the number of times the first keyword appears in the second text description. The second word frequency is used to measure the frequency of the first keyword appearing in the second text description. The higher the second word frequency, the more important the first keyword is in the second text description.

[0420] In step 4220, the second inverse document frequency is used to represent the commonality of the first keyword in the plurality of second text descriptions. The higher the second inverse document frequency, the fewer the number of the corresponding first keyword appearing in the plurality of second text descriptions, and correspondingly, the higher the importance of the second keyword.

[0421] In step 4230, the second similarity is the TF-IDF value of the corresponding first keyword with respect to the second text description. The second similarity is equal to the product of the second term frequency and the second inverse document frequency.

[0422] Exemplarily, it is assumed that a certain first keyword appears 6 times in a certain second text description, and the second text description has a total of 25 words; at this time, there are 15 second text descriptions in total, of which 3 second text descriptions appear the first keyword.

[0423] Correspondingly, the second term frequency of the first keyword with respect to the second text description is equal to the number of times the first keyword appears divided by the total number of words of the second text description, i.e., the second term frequency = 6 / 25 = 0.24.

[0424] The second inverse document frequency of the first keyword is equal to the total number of second text descriptions divided by the number of second text descriptions in which the first keyword appears, i.e., the second inverse document frequency = 15 / 3 = 5.

[0425] Finally, the second similarity of the first keyword is equal to the product of the second term frequency and the second inverse document frequency, i.e., the second similarity = 0.24*5 = 1.2.

[0426] It should be noted that the above example is for the convenience of understanding the process of calculating the second similarity by assuming numerical values, and does not represent a limitation on actual implementation.

[0427] Compared with the general matching of keywords according to promotion demand information and tags extracted from the video, in steps 4210 to 4230, the second similarity of each first keyword of the promotion demand information is calculated as the basis for matching the promotion demand information and the second text description. Using the TF-IDF value as the second similarity can accurately identify the relevance of the second text description and the first keyword, and can distinguish those words that frequently appear in some documents but are not important to the overall corpus.

[0428] In an embodiment, referring to FIG. 43, step 4210 includes:

[0429] Step 4310, for each first keyword, determining a second number of times the first keyword appears in the second text description.

[0430] Step 4320, extracting a third keyword from the second text description and determining a third keyword number of the third keyword.

[0431] Step 4330, determining the second word frequency based on the second number and the third keyword number.

[0432] The steps 4310 to 4330 are described in detail below.

[0433] In step 4310, the second number refers to the number of times the corresponding first keyword appears in a second text description.

[0434] In step 4320, the third keyword refers to a keyword extracted from the second text description after data preprocessing. The data preprocessing includes tokenization, stop word identification, stop word removal, non-entity word removal, stem extraction, etc. The third keyword number refers to the total number of third keywords extracted from the second text description.

[0435] Specifically, first, the second text description is preprocessed, including tokenization, stop word identification, stop word removal, non-entity word removal, stem extraction, etc. Then, a method similar to extracting the second keyword can be used, such as using the TF-IDF algorithm, calculating the word frequency of each preprocessed word in the second text description and the inverse document frequency in the entire text collection, multiplying the two to get the TF-IDF value, and selecting the word with a higher TF-IDF value as the third keyword. Or use the TextRank algorithm, construct a graph from the words in the text, calculate the importance score of the nodes according to the co-occurrence relationship between the words, and select the word with a higher score as the third keyword. Finally, count the total number of extracted third keywords, which is the third keyword number.

[0436] In step 4330, the second word frequency is equal to the second number divided by the third keyword number.

[0437] Since the first keyword is also a keyword extracted from the promotion demand information after data preprocessing, the same word as the first keyword in the second text description should also be extracted as a third keyword.

[0438] Next, the process of determining the second word frequency is explained in conjunction with Figure 44. As shown in Figure 44, suppose that the first keyword 1 to the first keyword n are extracted from the promotion demand information, and 25 third keywords (third keyword 1 to third keyword 25) are extracted from the second text description.

[0439] Taking the first keyword 1 as an example, the number of words in the third keywords 1 to the third keywords 25 that are the same as the first keyword 1 is determined as the second number. Assuming that the third keyword 3, the third keyword 4, and the third keyword 9 are the same as the first keyword 1, the second number is 3; correspondingly, the second word frequency 1 of the first keyword 1 is 3 / 25 = 0.12.

[0440] Similarly, the second word frequencies 2 to n of the remaining first keywords 2 to n with respect to the second text description are obtained.

[0441] In the steps 4310 to 4330, the second word frequency is determined based on the extracted third keywords and the first keywords. Compared with directly determining the second word frequency of the first keyword based on the second text description, the embodiment removes unnecessary noise words in the second text description, and improves the numerical accuracy of the second word frequency.

[0442] In an embodiment, referring to FIG. 45, the step 4220 includes:

[0443] Step 4510, determining the second number of the second text descriptions containing the first keyword.

[0444] Step 4520, obtaining the total number of the second text descriptions.

[0445] Step 4530, determining the second inverse document frequency based on the second number and the total number of the second text descriptions.

[0446] The steps 4510 to 4530 are described in detail below.

[0447] In the step 4510, the second number refers to the number of the second text descriptions containing the first keyword in the plurality of second text descriptions.

[0448] In the step 4520, the total number of the second text descriptions refers to the number of the second feature representations (second text descriptions) of the shot segments output by the shot representation prediction model in the current promotion content processing process.

[0449] In the step 4530, the second inverse document frequency is equal to the total number of the second text descriptions divided by the second number.

[0450] Next, referring to FIG. 46, the process of determining the second inverse document frequency is explained. As shown in FIG. 46, in a certain promotion content processing process, at least one video to be embedded with promotion content is processed by shot, to obtain a plurality of shot segments (shot 1 to shot m). Then, the shot 1 to the shot m are input into the shot representation prediction model respectively, to obtain the second text descriptions (shot details 1 to shot details m) corresponding to the respective shot segments.

[0451] Suppose m is 15, i.e. the total number of second text descriptions is 15 (shot detail 1 to shot detail m, m = 15).

[0452] The keyword extraction is performed on the promotion demand information to obtain first keyword 1 to first keyword n. Taking first keyword 1 as an example. Suppose that first keyword 1 is contained in shot description 1, shot description 5, shot description 7, shot description 11 and shot description 12, i.e. the second number is 5. Then the second inverse document frequency of first keyword 1 = 15 / 5 = 3.

[0453] Similarly, the second inverse document frequencies 2 to n of the remaining first keywords 2 to n are obtained.

[0454] In the above steps 4510 to 4530, the second inverse document frequency of the first keyword is calculated, which can accurately measure the universality of the first keyword in the plurality of second text descriptions. Then the second similarity between the first keyword and the second text description is obtained by combining the second inverse document frequency of the first keyword and the second word frequency. Compared with the matching basis between the promotion demand information and the second text description based only on the word frequency of the first keyword in each second text description, the second similarity combined with the second word frequency and the second inverse document frequency is used as the matching basis between the promotion demand information and the second text description in the embodiment, which can improve the accuracy of the matching.

[0455] In an embodiment, referring to FIG. 47, step 4510 includes:

[0456] Step 4710, extracting third keywords from each second text description.

[0457] Step 4720, taking each extracted third keyword as a second index, and storing the second index and the second text description identifier of each second text description containing the third keyword corresponding to the second index into a second index correspondence table.

[0458] Step 4730, taking the first keyword as a second index, searching the second index correspondence table to obtain the second text description identifier stored corresponding to the second index, so as to determine the second number of second text descriptions containing the first keyword.

[0459] The steps 4720 and 4730 are described in detail below.

[0460] In step 4720, the second index is an index value used to represent the second text description identifier of the corresponding at least one second text description, and its specific value is the third keyword extracted from each second text description. With the aid of the second index and the second index correspondence table, the second number of second text descriptions containing the first keyword can be quickly determined.

[0461] In the embodiment, the specific value of the second index is the corresponding third keyword. The second index correspondence table is a data storage structure composed of a plurality of second indexes and corresponding second text description identifiers. It also adopts an inverted index structure, which can improve the matching efficiency of the promotion demand information and the second text description, and quickly query the second text description containing a specific keyword.

[0462] The second text description identifier is an identifier for indicating the second text description. Exemplarily, the second text description identifier can be represented by a specific numerical value, or can be formed by one or more combinations of letters, numbers and symbols.

[0463] In step 4730, the first keyword is used as the second index, which means that the specific value of the second index is set to the first keyword. Thus, the same index is found in the second index correspondence table, and the corresponding stored second text description identifiers of the same index in the second index correspondence table are obtained. The number of these found second text descriptions is the second number.

[0464] Next, referring to FIG. 48, the process of determining the second number by constructing the index is explained.

[0465] As shown in FIG. 48, it is assumed that there are n second text descriptions, from which m third keywords (third keyword 1 to third keyword m) are extracted. Each third keyword is used as a second index, and the corresponding second index 1 to second index m is obtained.

[0466] It is assumed that the third keyword 1 is contained in the second text description 1 and the second text description 4, and the mapping relationship between the second index 1 and the second text description identifier 1 and the second text description identifier 4 is constructed and stored in the second index correspondence table. Similarly, the remaining second index 2 to second index m each corresponds to at least one second text description identifier, and the completed second index correspondence table is finally obtained.

[0467] The first keyword 1 is used as the second index 1_2, and the second index correspondence table is looked up. It is assumed that the second index 1_2 is the same as the second index 3, and the second text description identifier 16 to the second text description identifier 21 corresponding to the second index 3 is obtained based on the mapping relationship. Thus, it is determined that the first keyword 1 is contained in the second text description 16 to the second text description 21, and correspondingly, the second number of the second text description containing the first keyword is 6.

[0468] In the steps 4710 to 4730, a second index table of an inverted index structure is constructed. Each first keyword only needs to be compared with the second indexes of the second index table to quickly query the second text description identifiers of all second text descriptions containing the first keyword, so as to quickly determine the second number. Compared with comparing each first keyword with the second text descriptions one by one to determine the number of second text descriptions containing the first keyword, the manner of determining the second number in the embodiment can improve the matching efficiency of the promotion demand information and the second text descriptions.

[0469] In an embodiment, the second feature representation is a third deep feature of the sub-shot segment; referring to FIG. 49, the step 3320 comprises:

[0470] The step 4910 comprises: inputting the promotion demand information into the segment representation prediction model to obtain a second deep feature of the promotion demand information.

[0471] The step 4920 comprises: calculating a second matching degree based on the second deep feature and the third deep feature of each sub-shot segment.

[0472] The steps 4910 and 4920 are described in detail below.

[0473] In the step 4910, the second deep feature refers to a feature vector obtained after the promotion demand information is extracted by the segment representation prediction model.

[0474] In the step 4920, the third deep feature refers to a feature vector form of the second feature representation output by the segment representation prediction model.

[0475] Each sub-shot segment corresponds to a second matching degree with the promotion demand information. The second matching degree is a numerical value calculated based on the third deep feature of a sub-shot segment and the second deep feature of the promotion demand information, and is used to represent the matching degree between the promotion demand information and the sub-shot segment.

[0476] The process of calculating the second matching degree based on the second deep feature and the third deep feature is described below with reference to FIG. 50. As shown in FIG. 50, each video to which the content to be promoted needs to be delivered is preprocessed. The preprocessing comprises: performing sub-shot processing on each video to obtain a plurality of sub-shot segments, such as sub-shot 1 to sub-shot m in the figure.

[0477] Each sub-shot segment is input into the segment representation prediction model. The segment representation prediction model performs operations such as representative frame selection, representative frame feature vector extraction, and feature mode conversion on each sub-shot segment to obtain the third feature of each sub-shot segment as the third deep feature (for details, refer to the corresponding implementation details of the steps 3710 and 3720 provided in the above embodiment).

[0478] The promotion demand information is input into the segment representation prediction model to obtain the corresponding second deep feature. The similarity of the second deep feature with the third deep feature 1 to the third deep feature m is calculated to obtain the second matching degree 1 to the second matching degree m. The similarity calculation can be performed by calculating the third deep feature vector. Second depth feature vector The cosine similarity between them can also be calculated by calculating the third deep feature vector. Second depth feature vector The Euclidean distance between them. The formula for calculating the cosine similarity cosβ is: in Representing vectors and dot product, and Representing vectors respectively and The modulus. The formula for calculating the Euclidean distance e is: Among them G i and H i Representing vectors respectively and The i-th element, m represents the dimension of the vector.

[0479] Finally, based on the second matching degree, at least one storyboard segment that matches the promotional needs information is selected as the selected video segment.

[0480] In other examples, each scene segment also corresponds to a second cue. In this example, after preprocessing to obtain scenes 1 to m, scenes 1 to m are paired with their corresponding second cue 1 to second cue m and input into the segment representation prediction model. The segment representation prediction model outputs the second feature representation corresponding to each scene segment. Since the data form of this second feature representation is a feature vector that can be understood by the segment representation prediction model, and in the context of this embodiment, in order to facilitate subsequent operations such as matching degree calculation based on feature vectors, each second feature representation is used as a third deep feature.

[0481] The promotion demand information is then input into the segment representation prediction model to obtain the corresponding second deep features. The similarity between the second deep features and the second feature representations of each segment is calculated to obtain the second matching degree between the promotion demand information and each segment.

[0482] In the step 4910 and the step 4920, the second matching degree is calculated by matching the feature vector corresponding to the sub-shot segment with the feature vector corresponding to the promotion demand information. Compared with calculating the similarity between texts by matching the second text description corresponding to the sub-shot segment with the promotion demand information, the second matching degree is directly calculated based on the cross-modal feature in the embodiment, which can cover some details of visual information ignored by the text description and recall more possible selected video segments.

[0483] In an embodiment, referring to FIG. 51, the step 3330 includes:

[0484] In the step 5110, the first matching degree is weighted according to the first weight value to obtain the first weighted matching degree, and the second matching degree is weighted according to the second weight value to obtain the second weighted matching degree.

[0485] In the step 5120, the first weighted matching degree or the second weighted matching degree is determined as the to-be-inspected matching degree.

[0486] In the step 5130, the plot segment or the sub-shot segment that satisfies the first condition is determined as the selected video segment.

[0487] The steps 5110 to 5130 are described in detail as follows.

[0488] In the step 5110, the first weight value is set based on the importance of the plot segment compared with the sub-shot segment, and is used to represent the importance of the plot segment. The greater the first weight value, the greater the proportion of the plot segment in the finally determined selected video segment. The first weighted matching degree is obtained by weighting the first matching degree according to the first weight value. The first weight value is set based on the importance of the plot segment compared with the sub-shot segment, and the first weighted matching degree is used to determine the to-be-inspected matching degree in the subsequent step, and then determine the selected video segment. The first weighted matching degree can be a matching degree obtained by multiplying the first matching degree by the first weight value.

[0489] The to-be-inspected matching degree can be the first weighted matching degree or the second weighted matching degree, or a set element obtained by integrating the first weighted matching degree and the second weighted matching degree into a set. According to whether the to-be-inspected matching degree satisfies the first condition, the selected video segment can be determined.

[0490] The second weight is set based on the importance of the split shot segment compared to the episode segment, and is used to represent the importance of the split shot segment. The greater the second weight, the greater the proportion of the split shot segment in the final selected video segment. The second weighted matching degree is a matching degree obtained by weighting the second matching degree according to the second weight. The second weight is set based on the importance of the split shot segment compared to the episode segment, and the second weighted matching degree is used to determine the to-be-investigated matching degree, and then determine the selected video segment. The second weighted matching degree can be a matching degree obtained by multiplying the second matching degree by the second weight.

[0491] In step 5120, both the first weighted matching degree and the second weighted matching degree obtained in step 5110 can be used as the to-be-investigated matching degree. The first weighted matching degree and the second weighted matching degree are integrated to obtain a set of to-be-investigated matching degrees. Specifically, the first weighted matching degree and the second weighted matching degree are integrated into a set, and the elements in the set are determined as the to-be-investigated matching degrees.

[0492] In step 5130, for example, the first condition can be that the corresponding to-be-investigated matching degree is greater than a preset matching degree threshold. At this time, the episode segment or the split shot segment corresponding to the to-be-investigated matching degree greater than the matching degree threshold is determined as the selected video segment.

[0493] The first condition can have multiple cases, for example: if the first condition is that the corresponding to-be-investigated matching degree is greater than a preset matching degree threshold T, then the episode segment or the split shot segment corresponding to the to-be-investigated matching degree greater than T is determined as the selected video segment; if the first condition is the top x% in the to-be-investigated matching degree sequence, then the to-be-investigated matching degrees in the to-be-investigated matching degree set are sorted from large to small to obtain a to-be-investigated matching degree sequence, and the top x% to-be-investigated matching degrees in the sequence are selected as the selected video segment; if the first condition is the maximum matching degree in the to-be-investigated matching degree, then the episode segment or the split shot segment corresponding to the maximum matching degree in the multiple to-be-investigated matching degrees is determined as the selected video segment.

[0494] For example, the to-be-investigated matching degrees in the to-be-investigated matching degree set are sorted from large to small to obtain a to-be-investigated matching degree sequence, and the first condition is the top 10% in the to-be-investigated matching degree sequence. At this time, the episode segment or the split shot segment corresponding to the to-be-investigated matching degrees in the top 10% of the to-be-investigated matching degree sequence is determined as the selected video segment.

[0495] For example, the first condition can also be the maximum matching degree in the to-be-investigated matching degree. At this time, the episode segment or the split shot segment corresponding to the maximum matching degree in the multiple to-be-investigated matching degrees is determined as the selected video segment.

[0496] For different promotion contents, the embedding preferences for the plot segments or the shot segments can be different. In the steps 5110 to 5130, the emphasis between the plot segments and the shot segments is also taken as one of the conditions for determining the selected video segments by setting the first weight and the second weight for the first matching degree and the second matching degree respectively. Compared with directly selecting the selected video segments based on the first matching degree and the second matching degree, the embodiment determines the final selected video segments based on the first weighted matching degree and the second weighted matching degree, which can improve the accuracy of the selected video segment identification, and the final matching selected video segments can be more in line with the promotion demand information of the promotion content or the preferences for the plot segments or the shot segments in the actual situation.

[0497] In an embodiment, referring to FIG. 52, the step 3330 includes:

[0498] The step 5210 determines the plot segments with the first matching degree satisfying the second condition as target plot segments.

[0499] The step 5220 determines the shot segments with the second matching degree satisfying the third condition in the target plot segments as the selected video segments.

[0500] The steps 5210 and 5220 are described in detail below.

[0501] In the step 5210, the second condition can be that the first matching degree is greater than a preset threshold T1; or the first matching degrees can be sorted from large to small, and the top x% of the first matching degrees, where x is a positive integer and can be set according to actual needs; or the maximum matching degree in the first matching degrees.

[0502] The target plot segments refer to the plot segments with the corresponding first matching degree satisfying the second condition. In subsequent processing, the shot segments in the target plot segments will be further screened to determine the final selected video segments.

[0503] In the step 5220, the third condition is similar to the second condition. The third condition can be that the second matching degree is greater than a preset threshold T2; or the second matching degrees can be sorted from large to small, and the top y% of the second matching degrees, where y is a positive integer and can be set according to actual needs; or the maximum matching degree in the second matching degrees.

[0504] Next, referring to FIG. 53, the process of determining the shot segments as the selected video segments from the plot segments is explained and described.

[0505] As shown in FIG. 53, each video that needs to be promoted is preprocessed. The preprocessing includes:

[0506] Each video is shot script processed to obtain a plurality of shot script segments, such as shot script 1 to shot script m in the figure.

[0507] Based on the m shot script segments, the shot script segments belonging to the same plot are aggregated to obtain a plurality of plot segments, such as segment 1 to segment n in the figure.

[0508] Each video is text extraction processed to obtain text information containing dialogue information, and the text information is segmented to obtain text 1 to text n corresponding to segment 1 to segment n. In other examples, text extraction processing can also be performed on each plot segment to obtain text information corresponding to each plot segment.

[0509] Segment 1 to segment n and corresponding text 1 to text n are input into the segment representation prediction model in pairs. The segment representation prediction model outputs first feature representations corresponding to each plot segment, such as plot summary 1 to plot summary n corresponding to segment 1 to segment n in the figure.

[0510] Shot script 1 to shot script m are input into the segment representation prediction model. The segment representation prediction model outputs second feature representations corresponding to each shot script segment, such as shot script details 1 to shot script details m corresponding to shot script 1 to shot script m in the figure. In other examples, text extraction processing can also be performed on each shot script segment to obtain text information corresponding to each shot script segment. And shot script 1 to shot script m and corresponding text information are input into the segment representation prediction model in pairs to obtain shot script details 1 to shot script details m (not shown in the figure).

[0511] The promotion demand information is matched with the plot summary 1 to the plot summary n to obtain first matching degrees 1 to n corresponding to the segment 1 to the segment n respectively.

[0512] The promotion demand information is matched with the shot script description 1 to the shot script description m to obtain second matching degrees 1 to m corresponding to the shot script 1 to the shot script m respectively.

[0513] Suppose the second condition and the third condition in this example are that the matching degree is greater than a preset threshold. Based on the first matching degree 1 to the first matching degree n, the target plot segment is determined from the n plot segments. Since the plot segment is aggregated based on the shot script, the shot script segment in the target plot segment is the shot script segment in the shot script 1 to the shot script m. At this time, the second matching degree of the shot script segment in the target plot segment has been determined. Based on the third condition, the shot script segment corresponding to the second matching degree that meets the third condition is selected from the shot script segment in the target plot segment as the selected video segment.

[0514] In other examples, each video can be segmented first to obtain multiple plot segments, such as segment 1 to segment n in the figure. Segment 1 to segment n and corresponding text 1 to text n are input into the segment representation prediction model in pairs. The segment representation prediction model outputs first feature representations corresponding to each plot segment, such as plot abstract 1 to plot abstract n corresponding to segment 1 to segment n in the figure.

[0515] The plot abstract 1 to plot abstract n are matched with the promotion demand information respectively to obtain corresponding first matching degrees 1 to first matching degrees n. Based on the second condition and the first matching degrees, a plot segment that meets the second condition is selected as a target plot segment.

[0516] After obtaining the target plot segment, the target plot segment is processed by shot segmentation to obtain multiple shot segments. Each shot segment is input into the segment representation prediction model to obtain a shot detail corresponding to each shot segment. Each shot detail is matched with the promotion demand information to obtain a second matching degree. Based on the third condition and the second matching degrees, a shot segment that meets the third condition is selected as a selected video segment.

[0517] In the above steps 5210 and 5220, the promotion demand information is matched with the plot segment first, and then the target plot segment that meets the condition is matched at the shot level. Through the multi-level matching from large to small, compared with directly matching based on the plot segment, the embodiment can more accurately determine the selected video segment and improve the fine degree of embedding the promotion content.

[0518] Device and equipment of the embodiment of the disclosure

[0519] It can be understood that, although each step in each of the above flowcharts is displayed in sequence according to the arrow representation, these steps are not necessarily executed in sequence according to the arrow representation. Unless otherwise stated in the embodiment, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowcharts can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0520] FIG. 54 is a structural schematic diagram of a promotion content processing apparatus 5400 provided by the embodiment of the disclosure. The promotion content processing apparatus 5400 includes:

[0521] The shot segment acquisition unit 5410 is configured to acquire a video and aggregate video frames belonging to the same shot in the video into a shot segment.

[0522] The episode segment acquisition unit 5420 is configured to aggregate the shot segments belonging to the same episode in the video into an episode segment.

[0523] The segment representation prediction unit 5430 is configured to generate a first feature representation representing the episode segment.

[0524] The condition acquisition unit 5440 is configured to acquire promotion demand information of the promotion content.

[0525] The selected video segment matching unit 5450 is configured to determine a first matching degree of the promotion demand information and the first feature representation of each episode segment, and determine a selected video segment from the plurality of episode segments based on the first matching degree.

[0526] The promotion content embedding unit 5460 is configured to embed the promotion content in the selected video segment.

[0527] Optionally, the segment representation prediction unit 5430 is specifically configured to:

[0528] Acquire text information corresponding to the episode segment from the video.

[0529] Input the episode segment and the text information into a segment representation prediction model to obtain the first feature representation representing the episode segment.

[0530] Optionally, the segment representation prediction unit 5430 is specifically configured to:

[0531] Identify the dialogue information from the episode segment by using a subtitle recognition model and a speech recognition model.

[0532] Identify the character information corresponding to the dialogue information from the episode segment by using a face recognition model.

[0533] Integrate the dialogue information and the character information corresponding to the dialogue information into the text information.

[0534] Optionally, the segment representation prediction unit 5430 is specifically configured to:

[0535] Perform subtitle recognition in the episode segment by using a subtitle recognition model to obtain a first probability corresponding to each of a plurality of candidate dialogue information.

[0536] Perform speech recognition in the episode segment by using a speech recognition model to obtain a second probability corresponding to each of the plurality of candidate dialogue information.

[0537] Determine the dialogue information from the plurality of candidate dialogue information based on the first probability and the second probability corresponding to each of the plurality of candidate dialogue information.

[0538] Optionally, the segment representation prediction unit 5430 is specifically configured to:

[0539] obtain a first weight of the subtitle recognition model and a second weight of the speech recognition model.

[0540] determine a weighted sum of the first probability and the second probability based on the first weight and the second weight.

[0541] determine the script information from the plurality of candidate script information based on the weighted sum.

[0542] Optionally, the segment representation prediction model is a multi-modal large model.

[0543] The segment representation prediction unit 5430 is specifically configured to:

[0544] combine the text information into the first prompt to obtain a first combined text.

[0545] input the plot segment and the first combined text into the multi-modal large model to obtain a first feature representation representing the plot segment.

[0546] Optionally, the segment representation prediction unit 5430 is specifically configured to:

[0547] extract a plurality of first representative frames from the plot segment, and generate a first representative frame vector based on the first representative frames.

[0548] convert the plurality of first representative frame vectors into first features suitable for input into the multi-modal large model.

[0549] generate second features suitable for input into the multi-modal large model based on the first combined text.

[0550] input the first features and the second features into the multi-modal large model to obtain the first feature representation representing the plot segment.

[0551] Optionally, the first feature representation is a first textual description of the plot segment.

[0552] The selected video segment matching unit 5450 is specifically configured to:

[0553] extract a plurality of first keywords from the promotion demand information.

[0554] determine, for each first keyword, a first similarity between the first keyword and the first textual description.

[0555] determine, based on the first similarity between each first keyword and the first textual description, a first matching degree between the promotion demand information and the first textual description.

[0556] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0557] perform word segmentation on the promotion demand information.

[0558] stop words are identified in the split words, and the stop words are removed.

[0559] In the words after the stop words are removed, non-entity words are removed, to obtain a plurality of first keywords.

[0560] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0561] For each first keyword, a first term frequency of the first keyword appearing in the first text description is determined.

[0562] A first inverse document frequency of the first keyword appearing in the plurality of first text descriptions is determined.

[0563] Based on the first term frequency and the first inverse document frequency, a first similarity is determined.

[0564] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0565] For each first keyword, a first number of times that the first keyword appears in the first text description is determined.

[0566] Second keywords are extracted from the first text description, and a second keyword number of the second keywords is determined.

[0567] Based on the first number of times and the second keyword number, the first term frequency is determined.

[0568] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0569] A first number of the first text descriptions containing the first keyword is determined.

[0570] A total number of the first text descriptions is obtained.

[0571] Based on the first number and the total number of the first text descriptions, the first inverse document frequency is determined.

[0572] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0573] Second keywords are extracted from each of the first text descriptions.

[0574] With each of the extracted second keywords as a first index, the first index and a first text description identifier of each of the first text descriptions containing the second keyword are stored in a first index correspondence table.

[0575] With the first keyword as the first index, the first index correspondence table is searched to obtain the first text description identifier stored in correspondence with the first index, so as to determine the first number of the first text descriptions containing the first keyword.

[0576] Optionally, the first feature representation is a first deep feature of the plot segment.

[0577] The selected video segment matching unit 5450 is specifically configured to:

[0578] The promotion demand information is input into the segment representation prediction model to obtain a second deep feature of the promotion demand information.

[0579] Based on the second deep feature and the first deep feature of each plot segment, a first matching degree is calculated.

[0580] Optionally, the segment representation prediction unit 5430 is specifically configured to:

[0581] The shot segment is input into the segment representation prediction model to obtain a second feature representation of the shot segment.

[0582] The selected video segment matching unit 5450 is specifically configured to:

[0583] A second matching degree of the promotion demand information and the second feature representation of each shot segment is determined.

[0584] Based on the first matching degree and the second matching degree, a selected video segment is determined from the plurality of plot segments and the plurality of shot segments.

[0585] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0586] The first matching degree is weighted according to a first weight to obtain a first weighted matching degree, and the second matching degree is weighted according to a second weight to obtain a second weighted matching degree.

[0587] The first weighted matching degree or the second weighted matching degree is determined as a to-be-considered matching degree.

[0588] The plot segment or the shot segment whose to-be-considered matching degree satisfies a first condition is determined as the selected video segment.

[0589] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0590] The plot segment whose first matching degree satisfies a second condition is determined as a target plot segment.

[0591] The shot segment in the target plot segment whose second matching degree satisfies a third condition is determined as the selected video segment.

[0592] Optionally, the segment representation prediction model is a multi-modal large model.

[0593] The segment representation prediction unit 5430 is specifically configured to:

[0594] The shot representation prediction model is specifically configured to:

[0595] Optionally, the shot representation prediction model is specifically configured to:

[0596] The second representative frame vector is generated based on the second representative frames.

[0597] The plurality of second representative frame vectors are converted into third features suitable for inputting into the multi-modal large model.

[0598] The fourth features suitable for inputting into the multi-modal large model are generated based on the second prompt.

[0599] The third features and the fourth features are jointly input into the multi-modal large model to obtain the second feature representation representing the shot.

[0600] Optionally, the second feature representation is a second textual description of the shot.

[0601] The selected video segment matching unit 5450 is specifically configured to:

[0602] The plurality of first keywords are extracted from the promotion demand information.

[0603] For each first keyword, a second similarity between the first keyword and the second textual description is determined.

[0604] Based on the second similarity between each first keyword and the second textual description, a second matching degree between the promotion demand information and the second textual description is determined.

[0605] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0606] The promotion demand information is segmented.

[0607] Among the segmented words, stop words are identified and removed.

[0608] Among the words after removing the stop words, non-entity words are removed to obtain the plurality of first keywords.

[0609] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0610] For each first keyword, a second term frequency of the first keyword appearing in the second textual description is determined.

[0611] A second inverse document frequency of the first keyword appearing in the plurality of second textual descriptions is determined.

[0612] Based on the second term frequency and the second inverse document frequency, the second similarity is determined.

[0613] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0614] For each first keyword, determine a second number of occurrences of the first keyword in the second text description.

[0615] Extract a third keyword from the second text description, and determine a third keyword number of the third keyword.

[0616] Determine a second term frequency based on the second number and the third keyword number.

[0617] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0618] Determine a second number of second text descriptions containing the first keyword.

[0619] Obtain a total number of second text descriptions.

[0620] Determine a second inverse document frequency based on the second number and the total number of second text descriptions.

[0621] Optionally, the selected video segment matching unit 5450 is specifically configured to:

[0622] Extract a third keyword from each second text description.

[0623] Store, as a second index, each extracted third keyword and a second text description identifier of each second text description containing the third keyword into a second index correspondence table.

[0624] Take the first keyword as a second index, search the second index correspondence table to obtain a second text description identifier stored in correspondence with the second index, thereby determining a second number of second text descriptions containing the first keyword.

[0625] Optionally, the second feature is a third deep feature of the shot segment.

[0626] The selected video segment matching unit 5450 is specifically configured to:

[0627] Input the promotion demand information into the segment representation prediction model to obtain a second deep feature of the promotion demand information.

[0628] Calculate a second matching degree based on the second deep feature and the third deep feature of each shot segment.

[0629] Optionally, the segment representation prediction unit 5430 is specifically configured to:

[0630] Identify a public figure in the plot segment from the video.

[0631] Obtaining public figure information of a public figure.

[0632] Inputting the plot segment and the public figure information into a segment representation prediction model to obtain a first feature representation of the plot segment.

[0633] Optionally, the segment representation prediction model is a multi-modal large model.

[0634] The segment representation prediction unit 5430 is specifically configured to:

[0635] Combining the public figure information into the second prompt to obtain a second combined text.

[0636] Inputting the plot segment and the second combined text into the multi-modal large model to obtain the first feature representation of the plot segment.

[0637] Referring to FIG. 55, FIG. 55 is a structural block diagram of part of a terminal implementing the content promotion processing method of the embodiments of the present disclosure, which includes a radio frequency (RF) circuit 5510, a memory 5515, an input unit 5530, a display unit 5540, a sensor 5550, an audio circuit 5560, a wireless fidelity (WiFi) module 5570, a processor 5580, and a power supply 5590, and the like. Those skilled in the art can understand that the structure of the terminal shown in FIG. 55 does not constitute a limitation on the mobile phone or computer, and can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0638] The RF circuit 5510 can be used for receiving and sending signals in the process of receiving information or calling, especially receiving the downlink information of the base station and processing by the processor 5580; in addition, the uplink data is sent to the base station.

[0639] The memory 5515 can be used to store software programs and modules, and the processor 5580 executes various functions and data processing of the content terminal by running the software programs and modules stored in the memory 5515.

[0640] The input unit 5530 can be used to receive input digital or character information, and generate key signal input related to the setting and function control of the content terminal. Specifically, the input unit 5530 can include a touch panel 5531 and other input devices 5532.

[0641] The display unit 5540 can be used to display the input information or the provided information and various menus of the content terminal. The display unit 5540 can include a display panel 5541.

[0642] The audio circuit 5560, the speaker 5561, and the microphone 5562 can provide an audio interface.

[0643] In this embodiment, the processor 5580 included in the terminal can perform the promotion content processing method of the foregoing embodiments.

[0644] The terminal of the embodiments of the present disclosure includes but is not limited to a mobile phone, a computer, a smart voice interactive device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and the like. The embodiments of the present disclosure can be applied to various scenarios, including but not limited to a movie and television play playing, a video website, a video software, live advertisement insertion, media information processing, and the like.

[0645] FIG. 56 is a structural block diagram of a part of a server implementing the promotion content processing method of the embodiments of the present disclosure. The server can greatly differ due to different configurations or performances, and can include one or more central processing units (CPUs) 5622 (for example, one or more processors) and a memory 5632, one or more storage media 5630 (for example, one or more mass storage devices) storing application programs 5642 or data 5644. Among them, the memory 5632 and the storage media 5630 can be temporary storage or persistent storage. The programs stored in the storage media 5630 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Further, the central processing unit 5622 can be configured to communicate with the storage media 5630 and execute the series of instruction operations in the storage media 5630 on the server.

[0646] The server can also include one or more power supplies 5626, one or more wired or wireless network interfaces 5650, one or more input and output interfaces 5658, and / or one or more operating systems 5641, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0647] The central processing unit 5622 in the server can be configured to execute the promotion content processing method of the embodiments of the present disclosure.

[0648] The embodiments of the present disclosure also provide a computer readable storage medium for storing program codes, the program codes being used to execute the promotion content processing method of the foregoing embodiments.

[0649] The embodiments of the present disclosure also provide a computer program product including a computer program. A processor of a computer device reads the computer program and executes it, so that the computer device performs the promotion content processing method as described above.

[0650] To sum up, the application provides a promotion content processing method, device, equipment, computer readable storage medium and computer program product. By acquiring a video, the video frames belonging to the same split shot in the video are aggregated into a split shot segment. The video pictures in the split shot segment are relatively coherent, the video frames of the same split shot have the same or similar picture scene and shot layout, and aggregating the split shot segment helps subsequent analysis of the video in a more reasonable unit. Then, the split shot segments belonging to the same plot are aggregated into a plot segment, so that the plot segment has a common complete semantics, laying a foundation for subsequent accurate analysis of the video content. A first feature representation representing the plot segment is generated, which expresses the complete semantics of the plot segment. The promotion demand information is compared with the first feature representation of each plot segment to obtain a first matching degree, and a selected video segment is determined based on the first matching degree and embedded with promotion content. Since it has deep semantic analysis, compared with the traditional technology of only extracting features from the video for simple label comparison, it can more accurately understand the semantics of the video content and the promotion demand information, thereby improving the accuracy of the promotion content embedding. At the same time, it does not depend on a fixed label system, avoiding the problem of difficulty in finding an embedding position due to the inability of the label system to cover flexible and variable advertising needs, alleviating the poor flexibility, and improving the flexibility of the promotion content embedding processing.

[0651] Further, when generating the first feature representation representing the plot segment, text information corresponding to the plot segment is acquired from the video, and the plot segment and the text information are jointly input into a segment representation prediction model to obtain the first feature representation. The text information contains information such as dialogues and characters in the plot segment, and the segment representation prediction model combines the semantics of the plot segment and the text information to predict and output. In this way, the first feature representation not only contains the information of the video picture, but also integrates the semantics carried by the text. The semantics of the corresponding plot segment is more complete, which can more comprehensively reflect the content of the plot segment, thereby improving the accuracy of matching the first feature representation with the promotion demand information.

[0652] Further, when obtaining the text information corresponding to the plot segment, the dialogue information is recognized from the plot segment by using a subtitle recognition model and a speech recognition model, and the character information corresponding to the dialogue information is recognized by using a face recognition model, and the two are integrated into the text information. The subtitle recognition model and the speech recognition model extract dialogue information from different dimensions (image and speech), which can more comprehensively and accurately obtain the dialogue content. The face recognition model can accurately identify the character corresponding to the dialogue. By using different models to extract different elements in the plot segment and then integrating them to obtain the final text information, compared with directly extracting all information from the plot segment, the limitations of a single model in processing complex information are avoided, the required information can be more accurately obtained, the model requirements are low, the computational load is reduced, and the processing efficiency is improved.

[0653] Further, when the dialogue information is recognized from the plot segment by using the subtitle recognition model and the speech recognition model, the first probability corresponding to each of the plurality of candidate dialogue information is obtained by using the subtitle recognition model for subtitle recognition, the second probability corresponding to each of the plurality of candidate dialogue information is obtained by using the speech recognition model for speech recognition, and the dialogue information is determined from the plurality of candidate dialogue information based on the first probability and the second probability. The traditional technology usually uses a single model to extract video subtitles, which is easily affected by factors such as noisy environment, pronunciation problems, or inaccurate subtitles, resulting in inaccurate recognition. However, the present method simultaneously uses two models for recognition, and selects the more reliable candidate dialogue information as the final dialogue information according to the probability size, so that the obtained dialogue information is determined by the candidate dialogue information with higher probability (confidence) in the two models, which can effectively improve the accuracy of dialogue information recognition.

[0654] Further, when the dialogue information is determined based on the first probability and the second probability, the first weight of the subtitle recognition model and the second weight of the speech recognition model are obtained, the weighted sum of the first probability and the second probability is determined based on the first weight and the second weight, and the dialogue information is determined from the plurality of candidate dialogue information based on the weighted sum. Different videos have differences in subtitle clarity, character speaking language, etc., which will affect the recognition difficulty and accuracy of the subtitle recognition model and the speech recognition model. By setting the first weight and the second weight, the influence of video type on subtitle recognition and speech recognition is considered, in addition to considering the probability of the model itself, the contribution of the model to video recognition is also considered. This makes the determination of the dialogue information more scientific and reasonable, and improves the accuracy and reliability of the dialogue information.

[0655] Further, when the segment representation prediction model is a multi-modal large model, the text information is combined into the first prompt to obtain a first combined text, and the plot segment and the first combined text are jointly input into the multi-modal large model to obtain the first feature representation. The multi-modal large model can integrate and process input data from different modalities (such as text, image, audio, etc.). The first prompt guides the multi-modal large model to output the first feature representation of the plot segment required by the user, and the first combined text naturally combines the text information with the first prompt, providing more explicit semantic guidance for the multi-modal large model. Using the multi-modal large model can directly process the image data of the plot segment and the text data of the first combined text together, and use the first prompt to guide the multi-modal large model to perform high-level semantic abstraction on the plot segment to obtain the first feature representation containing the overall semantic information of the plot segment. Compared with the traditional technology of extracting features from the plot segment by different models and converting them into corresponding labels, the first feature representation obtained by the method can better reflect the complete semantics of the plot segment, and can improve the matching accuracy when matching with the promotion demand information in the subsequent process.

[0656] Further, when the plot segment and the first combined text are jointly input into the multi-modal large model to obtain the first feature representation, a plurality of first representative frames are extracted from the plot segment and a first representative frame vector is generated, the first representative frame vector is converted into a first feature suitable for input into the multi-modal large model, a second feature suitable for input into the multi-modal large model is generated based on the first combined text, and the first feature and the second feature are jointly input into the multi-modal large model to obtain the first feature representation. Extracting the first representative frame from the plot segment can effectively reduce the data volume while retaining the key information of the plot segment. Converting the first representative frame vector into the first feature and converting the first combined text into the second feature make the data format input into the multi-modal large model meet its processing requirements. The multi-modal large model has strong semantic understanding ability, and under the guidance of the first combined text, it can understand the semantics of the plot segment at a deeper level, and the output first feature representation can completely represent the deep semantics of the corresponding plot segment. Compared with the traditional technology of training multiple models on sub-tasks and extracting different labels from videos based on different models for matching, the method can better understand the plot context, emotion, and context information in the plot segment, which is conducive to finding selected video segments that better meet the promotion demand information and improving the effect of embedding promotion content.

[0657] Further, when the first feature is represented as a first text description of the plot segment, a plurality of first keywords are extracted from the promotion demand information, a first similarity of each first keyword with the first text description is determined, and a first matching degree of the promotion demand information with the first text description is determined based on the first similarities. The first text description can represent the semantics of the complete plot segment. By extracting the first keywords and calculating the first similarities of the first keywords with the first text description, the matching degree of the promotion demand information with the plot segment can be measured from the semantic level. Compared with the traditional technology of matching based on the keywords of the promotion demand information and the labels extracted from the video, the present method can consider more semantic details and improve the matching accuracy. Moreover, since the matching is based on semantics rather than fixed labels, it can adapt to flexible and variable advertising demands, and therefore has higher flexibility.

[0658] Further, when the first feature is represented as a first text description of the plot segment, a plurality of first keywords are extracted from the promotion demand information, a first similarity of each first keyword with the first text description is determined, and a first matching degree of the promotion demand information with the first text description is determined based on the first similarities. The first text description can represent the semantics of the complete plot segment. By extracting the first keywords and calculating the first similarities of the first keywords with the first text description, the matching degree of the promotion demand information with the plot segment can be measured from the semantic level. Compared with the traditional technology of matching based on the keywords of the promotion demand information and the labels extracted from the video, the present method can consider more semantic details and improve the matching accuracy. Moreover, since the matching is based on semantics rather than fixed labels, it can adapt to flexible and variable advertising demands, and therefore has higher flexibility.

[0659] Further, when the first feature is represented as a first text description of the plot segment, a plurality of first keywords are extracted from the promotion demand information, a first similarity of each first keyword with the first text description is determined, and a first matching degree of the promotion demand information with the first text description is determined based on the first similarities. The first text description can represent the semantics of the complete plot segment. By extracting the first keywords and calculating the first similarities of the first keywords with the first text description, the matching degree of the promotion demand information with the plot segment can be measured from the semantic level. Compared with the traditional technology of matching based on the keywords of the promotion demand information and the labels extracted from the video, the present method can consider more semantic details and improve the matching accuracy. Moreover, since the matching is based on semantics rather than fixed labels, it can adapt to flexible and variable advertising demands, and therefore has higher flexibility.

[0660] Further, when determining the first term frequency of the first keyword appearing in the first text description, the first number of times of the first keyword appearing in the first text description is determined for each first keyword, the second keyword is extracted from the first text description and the second keyword number of the second keyword is determined, and the first term frequency is determined based on the first number of times and the second keyword number. There may be some noise words such as stop words, non-useful words, etc. in the first text description, and if the first term frequency is directly determined based on the first text description, the first term frequency will be too small and cannot accurately reflect the importance of the first keyword. By extracting the keywords from the first text description, unnecessary noise words in the first text description are removed, and the first term frequency is determined based on the extracted multiple second keywords and the first keyword, which can improve the numerical accuracy of the first term frequency, thereby more accurately measuring the importance of the first keyword in the first text description.

[0661] Further, when determining the first inverse document frequency of the first keyword appearing in the multiple first text descriptions, the first number of first text descriptions containing the first keyword is determined, the total number of first text descriptions is obtained, and the first inverse document frequency is determined based on the first number and the total number of first text descriptions. The first inverse document frequency is used to represent the universality of the first keyword in the multiple first text descriptions. If only the term frequency of the first keyword in each first text description is used as the basis for matching the promotion demand information and the first text description, since the first keyword may be a commonly used word in the first text description, it will appear in multiple first text descriptions, and the matching result will not be accurate. By calculating the first inverse document frequency of the first keyword, the universality of the first keyword in the multiple first text descriptions can be accurately measured, and then the first similarity between the first keyword and the first text description is obtained by combining the first inverse document frequency of the first keyword with the first term frequency. Using the first similarity as the basis for matching the promotion demand information and the first text description can improve the accuracy of the matching.

[0662] Further, in determining the first number of the first text descriptions containing the first keyword, second keywords are extracted from the respective first text descriptions, each extracted second keyword is taken as a first index, and the first index and the first text description identifier of the respective first text description containing the second keyword are stored in a first index correspondence table with the first keyword as the first index. The first index correspondence table is looked up with the first keyword as the first index to obtain the first text description identifier stored in correspondence with the first index, thereby determining the first number. If each first keyword is compared with each first text description to determine the number of first text descriptions containing the first keyword, the query efficiency will be low. By constructing a first index correspondence table in an inverted index structure, each first keyword only needs to be compared with each first index of the first index correspondence table to quickly query the first text description identifier of all first text descriptions containing the first keyword, thereby quickly determining the first number. Based on this, the matching efficiency of the promotion demand information and the first text description can be improved.

[0663] Further, when the first feature represents the first deep feature of the plot segment, the promotion demand information is input into the segment representation prediction model to obtain a second deep feature of the promotion demand information, and the first matching degree is calculated based on the second deep feature and the first deep feature of each plot segment. The first deep feature and the second deep feature are feature vectors obtained by the segment representation prediction model extracting features from the plot segment and the promotion demand information, which can more comprehensively and deeply reflect the features of the plot segment and the promotion demand information. By calculating the first matching degree between the feature vector corresponding to the plot segment and the feature vector corresponding to the promotion demand information, compared with calculating the similarity between texts by the first text description corresponding to the plot segment and the promotion demand information, directly calculating the first matching degree based on cross-modal features can cover some detailed visual information ignored by the text description, recall more possible selected video segments, and improve the opportunity and effect of embedding promotion content.

[0664] Further, in determining the selected video segment based on the first matching degree, the split shot segment is input into the segment representation prediction model to obtain a second feature representation representing the split shot segment, the second matching degree of the promotion demand information and the second feature representation of each split shot segment is determined, and the selected video segment is determined among the plurality of plot segments and the plurality of split shot segments based on the first matching degree and the second matching degree. Traditional technologies usually analyze and match videos from a single dimension, while the present method matches based on semantics from two different dimensions of plot segments and split shot segments. The plot segment can reflect the overall plot semantics of the video, and the split shot segment can provide more detailed picture and scene information. Determining the selected video segment by combining the first matching degree and the second matching degree can more accurately match the selected video segment corresponding to the promotion demand information, improving the comprehensiveness and accuracy of the matching.

[0665] Further, when determining the selected video segment based on the first matching degree and the second matching degree, the first matching degree is weighted according to a first weight value to obtain a first weighted matching degree, the second matching degree is weighted according to a second weight value to obtain a second weighted matching degree, the first weighted matching degree or the second weighted matching degree is determined as a to-be-considered matching degree, and a plot segment or a shot segment that meets a first condition of the to-be-considered matching degree is determined as the selected video segment. For different promotional contents, the embedding preferences for plot segments or shot segments can be different. By setting the first weight value and the second weight value for the first matching degree and the second matching degree respectively, the emphasis degree between the plot segment and the shot segment is also taken as one of the conditions for determining the selected video segment. Compared with directly selecting the selected video segment based on the first matching degree and the second matching degree, the method can adjust the importance of the plot segment and the shot segment in the matching process according to actual needs, improve the accuracy of the selected video segment identification, and finally match the selected video segment that can better meet the promotional demand information of the promotional content or the preference for the plot segment or the shot segment in the actual situation.

[0666] Further, when determining the selected video segment based on the first matching degree and the second matching degree, the first matching degree is weighted according to a first weight value to obtain a first weighted matching degree, the second matching degree is weighted according to a second weight value to obtain a second weighted matching degree, the first weighted matching degree or the second weighted matching degree is determined as a to-be-considered matching degree, and a plot segment or a shot segment that meets a first condition of the to-be-considered matching degree is determined as the selected video segment. For different promotional contents, the embedding preferences for plot segments or shot segments can be different. By setting the first weight value and the second weight value for the first matching degree and the second matching degree respectively, the emphasis degree between the plot segment and the shot segment is also taken as one of the conditions for determining the selected video segment. Compared with directly selecting the selected video segment based on the first matching degree and the second matching degree, the method can adjust the importance of the plot segment and the shot segment in the matching process according to actual needs, improve the accuracy of the selected video segment identification, and finally match the selected video segment that can better meet the promotional demand information of the promotional content or the preference for the plot segment or the shot segment in the actual situation.

[0667] Further, when generating the first feature representation representing the plot segment, a public figure in the plot segment can also be identified from the video, public figure information of the public figure is obtained, the plot segment and the public figure information are jointly input into the segment representation prediction model, and the first feature representation representing the plot segment is obtained. If the plot segment is directly predicted by the segment representation prediction model for the first feature representation, the segment representation prediction model may not be able to accurately identify the public figure appearing in the plot segment, so that the first feature representation may not include the related information of the public figure. However, the method first identifies the public figure in the plot segment from the video, obtains the corresponding information of the public figure, and then inputs the public figure information and the plot segment into the segment representation prediction model to obtain the first feature representation of the plot segment, so that the first feature representation can accurately carry the related information of the public figure appearing in the plot segment. When the promotion demand information (such as endorsement advertising) of the specified public figure is involved, the public figure can also be used as a matching factor to participate in the matching process, so that the selected video segment corresponding to the promotion demand information of the specified public figure can be accurately identified, and the embedding effect of the promotion content is improved.

[0668] Further, when the segment representation prediction model is a multi-modal large model, the public figure information is combined into the second prompt to obtain a second combined text, and the plot segment and the second combined text are jointly input into the multi-modal large model to obtain the first feature representation representing the plot segment. If the public figure of the video is directly matched as a label alone, that is, as long as the character appears in the video, it is considered that the matching is successful, and in actual application, there may be a case that the scene in which the character exists in a certain video segment is not suitable for embedding the promotion content. The method combines the public figure information and the second prompt to generate the second combined text, and inputs the second combined text and the plot segment into the segment representation prediction model, so that the first feature representation includes the public figure information. In this way, when the segment matching of the promotion demand information (such as endorsement advertising) of the specified public figure is performed, the plot semantics in the first feature representation and the public figure can be matched together, so that the situation that the public figure is matched but the plot of the selected video segment corresponding to the public figure is not suitable for embedding the promotion content is avoided, the acceptance of the embedded promotion content by the user is improved, and discomfort is avoided.

[0669] Further, when the split shot is input into the shot representation prediction model to obtain the second feature representation representing the split shot, if the shot representation prediction model is a multi-modal large model, the split shot and the second prompt are jointly input into the multi-modal large model to obtain the second feature representation. Traditional technologies usually extract features of the plot shot through different models and convert them into corresponding labels, which is difficult to obtain the complete semantics of the split shot. However, the multi-modal large model can integrate and process input data from different modalities, and the second prompt guides the multi-modal large model to output the detailed description of the split shot required by the user as the second feature representation. Using the multi-modal large model can directly process the image data of the split shot and the text data of the second prompt together, and the second prompt guides the multi-modal large model to perform high-level semantic extraction on the split shot to obtain the second feature representation containing detailed semantic information of the split shot. Therefore, when the second feature representation is subsequently matched with the promotion demand information, the matching degree accuracy can be improved, and the selected video segment that is more suitable for the promotion demand information can be obtained.

[0670] Further, when the split shot and the second prompt are jointly input into the multi-modal large model to obtain the second feature representation, a plurality of second representative frames are extracted from the split shot, and a second representative frame vector is generated based on the second representative frames. The plurality of second representative frame vectors are converted into third features suitable for input into the multi-modal large model, and fourth features suitable for input into the multi-modal large model are generated based on the second prompt. The third features and the fourth features are jointly input into the multi-modal large model to obtain the second feature representation representing the split shot. Extracting the second representative frames from the split shot can reduce the amount of data and improve processing efficiency while retaining key information of the split shot. Converting the second representative frame vector into the third feature and converting the second prompt into the fourth feature makes the data format input into the multi-modal large model meet its processing requirements. The multi-modal large model has strong semantic understanding ability, and under the guidance of the second prompt, it can understand the semantics of the split shot at a deeper level, and the output second feature representation can completely represent the deep semantics of the corresponding split shot. Compared with traditional technologies that train multiple models on sub-tasks and extract different labels from videos based on different models for matching, the present method can better understand the details of the split shot, such as characters, clothing, actions, expressions, positions, foregrounds, backgrounds, etc., which is conducive to finding selected video segments that better meet the promotion demand information and improving the effect of embedding promotion content.

[0671] Further, when the second feature is represented as a second text description of the shot segment, a plurality of first keywords are extracted from the promotion demand information, a second similarity of each first keyword with the second text description is determined, and a second matching degree of the promotion demand information with the second text description is determined based on the second similarities. The second text description can represent the semantics of the shot segment in detail. By extracting the first keywords and calculating the second similarities of the first keywords with the second text description, the matching degree of the promotion demand information with the shot segment is measured from the semantic level. Compared with the traditional technology of matching the promotion demand information with the labels extracted from the video based on the labels, the present method can consider more detailed semantic details of the shot segment, and improve the matching accuracy. Moreover, the matching is based on semantics rather than fixed labels, which can better adapt to flexible and variable advertising demands and improve the flexibility of matching.

[0672] Further, when the first keywords are extracted from the promotion demand information to determine the second matching degree with the second text description, the promotion demand information is segmented, stop words are identified and removed from the segmented words, and non-entity words are removed from the words after removing the stop words to obtain the first keywords. The stop words and non-entity words in the promotion demand information can interfere with the matching process and reduce the accuracy of the matching. By performing text preprocessing on the promotion demand information to remove useless stop words and non-entity words, and matching based on the first keywords and the second text description, noise and redundant information can be effectively reduced, the matching process can be more focused on key information, and the accuracy of the matching can be improved.

[0673] Further, when the second similarity of the first keyword with the second text description is determined, the second word frequency of the first keyword in the second text description is determined, the second inverse document frequency of the first keyword in the plurality of second text descriptions is determined, and the second similarity is determined based on the second word frequency and the second inverse document frequency. The term frequency-inverse document frequency (TF-IDF) method considers the frequency of a word in a specific document (second text description) and the sparsity of the word in the entire document set (plurality of second text descriptions). The second word frequency reflects the importance of the first keyword in the second text description, and the second inverse document frequency reflects the universality of the keyword in the plurality of second text descriptions. Using the TF-IDF value as the second similarity can accurately identify the relevance of the second text description to the first keyword, and can distinguish those words that frequently appear in some documents but are not important to the overall corpus. Compared with the general matching of the keywords of the promotion demand information with the labels extracted from the video, the present method can more accurately measure the matching degree of the promotion demand information with the second text description.

[0674] Further, in determining the second word frequency of the first keyword appearing in the second text description, the second number of times of the first keyword appearing in the second text description is determined, third keywords are extracted from the second text description and a third keyword number of the third keywords is determined, and the second word frequency is determined based on the second number of times and the third keyword number. There can be some noise words such as stop words, non-useful words, etc. in the second text description, and if the second word frequency of the first keyword is directly determined based on the second text description, the second word frequency will be too small and cannot accurately reflect the importance of the first keyword. By extracting the keywords from the second text description, unnecessary noise words in the second text description are removed, and the second word frequency is determined based on the extracted multiple third keywords and the first keyword, which can improve the numerical accuracy of the second word frequency and more accurately measure the importance of the first keyword in the second text description.

[0675] Further, in determining the second inverse document frequency of the first keyword appearing in the multiple second text descriptions, the second number of second text descriptions containing the first keyword is determined, the total number of second text descriptions is obtained, and the second inverse document frequency is determined based on the second number and the total number of second text descriptions. If only the word frequency of the first keyword in each second text description is used as the basis for matching the promotion demand information and the second text description, since the first keyword can be a commonly used word in the second text description, it will appear in multiple second text descriptions, and the matching result will not be accurate. By calculating the second inverse document frequency of the first keyword, the universality of the first keyword in the multiple second text descriptions can be accurately measured, and the second similarity between the first keyword and the second text description is obtained by combining the second inverse document frequency of the first keyword with the second word frequency. Using the second similarity as the basis for matching the promotion demand information and the second text description can improve the accuracy of the matching.

[0676] Further, in determining the second number of the second text descriptions containing the first keyword, third keywords are extracted from the respective second text descriptions, each extracted third keyword is taken as a second index, and the second index and the second text description identifier of the respective second text description containing the third keyword are stored in a second index correspondence table with the first keyword as the second index. The second index correspondence table is looked up with the first keyword as the second index to obtain the second text description identifier stored in correspondence with the second index, thereby determining the second number. If each first keyword is compared with the second text descriptions one by one to determine the number of the second text descriptions containing the first keyword, the query efficiency will be low. By constructing a second index correspondence table in an inverted index structure, each first keyword only needs to be compared with the respective second index of the second index correspondence table, and the second text description identifier of the second text description containing the first keyword can be quickly queried, thereby quickly determining the second number. Compared with the traditional one-by-one comparison method, the method can significantly improve the matching efficiency of the promotion demand information and the second text descriptions.

[0677] Further, when the second feature is represented as a third depth feature of the sub-shot, the promotion demand information is input into the shot representation prediction model to obtain a second depth feature of the promotion demand information, and the second matching degree is calculated based on the second depth feature and the third depth feature of each sub-shot. The third depth feature and the second depth feature are feature vectors obtained by the shot representation prediction model extracting features from the sub-shot and the promotion demand information, and can more comprehensively and deeply reflect the features of the sub-shot and the promotion demand information. By calculating the second matching degree between the feature vector corresponding to the sub-shot and the feature vector corresponding to the promotion demand information, compared with calculating the similarity between the second text description corresponding to the sub-shot and the promotion demand information, the second matching degree is directly calculated based on the cross-modal features, which can cover some detailed visual information ignored by the text description and recall more possible selected video shots. This increases the opportunity for embedding promotion content and improves the matching effect of the promotion content and the video shot, so that the promotion content can be more accurately embedded into the appropriate sub-shot.

[0678] The terms "first", "second", "third", "fourth", and the like in the description of the disclosure and the above drawings, if any, are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "contain" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device containing a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0679] It should be understood that in the present disclosure, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between the associated content, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated content. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0680] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is two or more, greater than, less than, more than, etc. are not included in the number, above, below, etc. are understood to include the number.

[0681] In several embodiments provided by the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0682] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0683] In addition, each functional unit in various embodiments of the present disclosure can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0684] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0685] It should also be understood that various embodiments provided by the present disclosure can be combined in any way to achieve different technical effects.

[0686] The above is a specific description of the embodiments of the present disclosure, but the present disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present disclosure, and these equivalent modifications or replacements are included in the scope defined by the claims of the present disclosure.

Claims

1. A method for processing promotion content, executed by a computer device, comprising: obtaining a video, and aggregating video frames belonging to a same shot in the video into a shot segment; aggregating the shot segments belonging to a same plot in the video into a plot segment; generating a first feature representation representing the plot segment; obtaining promotion demand information of a promotion content; determining a first matching degree of the promotion demand information and the first feature representation of each plot segment, and determining a selected video segment from the plot segments based on the first matching degree; and embedding the promotion content in the selected video segment.

2. The method of claim 1, wherein the generating a first feature representation representing the plot segment comprises: obtaining text information corresponding to the plot segment from the video; and inputting the plot segment and the text information into a segment representation prediction model to obtain the first feature representation representing the plot segment.

3. The method of claim 2, wherein the obtaining text information corresponding to the plot segment from the video comprises: identifying dialogue information from the plot segment using a subtitle recognition model and a speech recognition model; identifying character information corresponding to the dialogue information from the plot segment using a face recognition model; and integrating the dialogue information and the character information corresponding to the dialogue information into the text information.

4. The method of claim 3, wherein the identifying dialogue information from the plot segment using the subtitle recognition model and the speech recognition model comprises: performing subtitle recognition in the plot segment using the subtitle recognition model to obtain a first probability corresponding to each candidate dialogue information; performing speech recognition in the plot segment using the speech recognition model to obtain a second probability corresponding to each candidate dialogue information; and determining the dialogue information from the candidate dialogue information based on the first probability and the second probability corresponding to each candidate dialogue information.

5. The method of claim 4, wherein the determining the dialogue information from the candidate dialogue information based on the first probability and the second probability corresponding to each candidate dialogue information comprises: obtaining a first weight of the subtitle recognition model and a second weight of the speech recognition model; determining a weighted sum of the first probability and the second probability based on the first weight and the second weight; and determining the dialogue information from the candidate dialogue information based on the weighted sum.

6. The method of any one of claims 2 to 5, wherein the segment representation prediction model is a multi-modal large model; and the inputting the plot segment and the text information into the segment representation prediction model to obtain the first feature representation representing the plot segment comprises: combining the text information into a first prompt to obtain a first combined text. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ inputting the plot segment and the first combination text into the multi-modal large model together to obtain the first feature representation representing the plot segment.

7. The promoted content processing method of claim 6, wherein the inputting the plot segment and the first combination text into the multi-modal large model together to obtain the first feature representation representing the plot segment comprises: extracting a plurality of first representative frames from the plot segment, and generating first representative frame vectors based on the first representative frames; converting the plurality of first representative frame vectors into first features suitable for inputting into the multi-modal large model; generating second features suitable for inputting into the multi-modal large model based on the first combination text; inputting the first features and the second features into the multi-modal large model together to obtain the first feature representation representing the plot segment.

8. The promoted content processing method of any one of claims 2 to 7, wherein the first feature representation is a first text description of the plot segment. The determining the first matching degree of the promotion demand information and each plot segment comprises: extracting a plurality of first keywords from the promotion demand information; determining, for each first keyword, a first similarity of the first keyword and the first text description; determining the first matching degree of the promotion demand information and the first text description based on the first similarity of each first keyword and the first text description.

9. The promoted content processing method of claim 8, wherein the extracting a plurality of first keywords from the promotion demand information comprises: segmenting the promotion demand information; identifying and removing stop words from the segmented words; removing non-entity words from the words after removing the stop words to obtain the plurality of first keywords.

10. The promoted content processing method of claim 8 or 9, wherein the determining, for each first keyword, a first similarity of the first keyword and the first text description comprises: determining, for each first keyword, a first term frequency of the first keyword appearing in the first text description; determining a first inverse document frequency of the first keyword appearing in a plurality of first text descriptions; determining the first similarity based on the first term frequency and the first inverse document frequency.

11. The promoted content processing method of claim 10, wherein the determining, for each first keyword, a first term frequency of the first keyword appearing in the first text description comprises: determining, for each first keyword, a first number of times of the first keyword appearing in the first text description; extracting a second keyword from the first text description, and determining a second keyword number of the second keyword; determining the first term frequency based on the first number of times and the second keyword number.

12. The promoted content processing method of claim 10 or 11, wherein the determining a first inverse document frequency of the first keyword appearing in a plurality of first text descriptions comprises: determining a first number of the first text descriptions containing the first keyword; obtaining a total number of the first text descriptions; determining the first inverse document frequency based on the first number and the total number of the first text descriptions.

13. The method of claim 12, wherein the determining the first number of the first text descriptions containing the first keyword comprises: extracting a second keyword from each of the first text descriptions; storing, as a first index, each of the extracted second keywords and a first text description identifier of each of the first text descriptions containing the second keyword into a first index correspondence table; finding, as the first index, the first index correspondence table to obtain the first text description identifier stored in correspondence with the first index, thereby determining the first number of the first text descriptions containing the first keyword.

14. The method of any one of claims 2 to 13, wherein the first feature representation is a first deep feature representation of the story segment; wherein the determining the first matching degree between the promotion demand information and the first feature representation of each of the story segments comprises: inputting the promotion demand information into the segment representation prediction model to obtain a second deep feature representation of the promotion demand information; calculating the first matching degree based on the second deep feature representation and the first deep feature representation of each of the story segments.

15. The method of any one of claims 2 to 14, wherein the determining the selected video segment from the plurality of story segments based on the first matching degree comprises: inputting the shot segment into the segment representation prediction model to obtain a second feature representation of the shot segment; determining a second matching degree between the promotion demand information and the second feature representation of each of the shot segments; determining the selected video segment from the plurality of story segments and the plurality of shot segments based on the first matching degree and the second matching degree.

16. The method of claim 15, wherein the determining the selected video segment from the plurality of story segments and the plurality of shot segments based on the first matching degree and the second matching degree comprises: weighting the first matching degree by a first weight to obtain a first weighted matching degree, and weighting the second matching degree by a second weight to obtain a second weighted matching degree; determining the first weighted matching degree or the second weighted matching degree as a to-be-inspected matching degree; determining, as the selected video segment, the story segment or the shot segment satisfying a first condition with respect to the to-be-inspected matching degree.

17. An apparatus for processing promotion content, the apparatus comprising: a shot segment obtaining unit configured to obtain a video and aggregate video frames belonging to a same shot in the video into a shot segment; a story segment obtaining unit configured to aggregate the shot segments belonging to a same story in the video into a story segment; a segment representation prediction unit configured to generate a first feature representation of the story segment. a condition obtaining unit, configured to obtain promotion demand information of a promotion content; a selected video segment matching unit, configured to determine a first matching degree of the promotion demand information and the first feature representation of each of the plot segments, and determine a selected video segment from the plot segments based on the first matching degree; and a promotion content embedding unit, configured to embed the promotion content in the selected video segment.

18. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the promotion content processing method according to any one of claims 1 to 16 when executing the computer program.

19. A computer readable storage medium, wherein the storage medium stores a computer program, and the computer program implements the promotion content processing method according to any one of claims 1 to 16 when executed by a processor.

20. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device executes the promotion content processing method according to any one of claims 1 to 16.

Images