An AI-based advertisement material multi-modal intelligent retrieval method and system

Abstract

The application relates to the field of information technology, in particular to an AI-based advertisement material multi-modal intelligent retrieval method and system. The method comprises the following steps: receiving multi-modal advertisement materials, and extracting content descriptions of the advertisement materials; converting the multi-modal advertisement materials and the content descriptions into unified multi-modal joint representation vectors by using a multi-modal fusion model, and constructing a material index library; receiving demand information of a new advertisement provided by a user, wherein the demand information comprises at least one of a content description, a target audience description and a delivery scene description; extracting embedding vectors of the demand information, performing retrieval in the material index library according to the similarity between the embedding vectors and the multi-modal joint representation vectors, and obtaining advertisement materials with a similarity higher than a preset threshold as candidate results; and generating a predicted effect tendency of the candidate results, and outputting the candidate results as retrieval results after screening according to the predicted effect tendency.

Description

Technical Field

[0001] This application relates to the field of information technology, specifically to an AI-based multimodal intelligent retrieval method and system for advertising creatives. Background Technology

[0002] With the rapid development of digital marketing, advertising companies and brands are not only accumulating a large amount of historical advertising materials, but also frequently producing a large amount of novel and original content to cope with rapidly changing market trends and user preferences. These newly created materials often involve significant creative resources and production costs. Without a systematic and intelligent management mechanism, they are prone to falling into the trap of "one-time use" or even "being forgotten before being effectively utilized," resulting in serious resource waste. Historical materials, as important digital assets for enterprises, contain rich creative elements, style paradigms, and campaign experience, possessing extremely high reuse potential. Whether existing or new materials, their forms are generally multimodal (such as video, images, audio, and text), with unstructured content and complex semantics. Traditional management methods based on folder classification or keyword tags are no longer sufficient to meet the needs of refined retrieval, cross-modal association, and semantic-level matching. Therefore, it is necessary to research retrieval technologies suitable for rich multimodal materials. Summary of the Invention

[0003] This specification describes an AI-based multimodal intelligent retrieval method and system for advertising creatives through several embodiments.

[0004] Firstly, the embodiments of this specification provide an AI-based multimodal intelligent retrieval method for advertising creatives, including the following steps:

[0005] Receive multimodal advertising materials and extract the content description of the advertising materials;

[0006] The multimodal advertising materials and content descriptions are converted into a unified multimodal joint representation vector using a multimodal fusion model, and a material index library is constructed.

[0007] Receive new advertising request information provided by users, the request information including at least one of content description, target audience description and placement scenario description;

[0008] The embedding vector of the demand information is extracted, and based on the similarity between the embedding vector and the multimodal joint representation vector, a search is performed in the material index library to obtain advertising materials with a similarity higher than a preset threshold as candidate results.

[0009] The predicted effect tendency of the candidate results is generated, and the candidate results are filtered according to the predicted effect tendency and output as the search results.

[0010] Secondly, embodiments of this specification provide an AI-based multimodal intelligent retrieval system for advertising creatives, including:

[0011] The first receiving module receives multimodal advertising materials and extracts the content description of the advertising materials;

[0012] The conversion module uses a multimodal fusion model to convert the multimodal advertising materials and content descriptions into a unified multimodal joint representation vector, and builds a material index library;

[0013] The second receiving module receives new advertising demand information provided by the user, the demand information including at least one of content description, target audience description and placement scenario description;

[0014] The extraction module extracts the embedding vector of the demand information, and searches the material index library based on the similarity between the embedding vector and the multimodal joint representation vector to obtain advertising materials with a similarity higher than a preset threshold as candidate results.

[0015] The output module generates the predicted effect tendency of the candidate results, filters the candidate results based on the predicted effect tendency, and outputs them as search results.

[0016] Thirdly, embodiments of this specification provide an electronic device, including a processor and a memory;

[0017] The processor is connected to the memory;

[0018] The memory is used to store executable program code;

[0019] The processor runs a program corresponding to the executable program code stored in the memory to perform the method described in any of the above aspects.

[0020] Fourthly, embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above aspects.

[0021] Fifthly, embodiments of this specification provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above aspects.

[0022] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0023] In several embodiments of this specification, the AI-based multimodal intelligent retrieval method and system for advertising creatives, through deep fusion of multiple modal information such as video, images, audio, and text, constructs a unified multimodal joint representation vector, achieving a comprehensive characterization of the semantic content of advertising creatives and improving the accuracy and recall rate of creative retrieval. It exhibits superior retrieval efficiency when processing large volumes of cross-modal advertising content. By transforming new advertising needs input by users into embedded vectors within the same semantic space as the creative representation, and combining this with approximate nearest neighbor indexing technology to achieve efficient and low-latency similarity matching, it helps improve creative reuse rates, shorten advertising production cycles, and effectively helps designers filter relevant advertising creatives from massive amounts of advertising creatives, thus improving design efficiency. A predictive effect tendency evaluation mechanism for advertising creatives is introduced to predict the emotional responses that candidate creatives may evoke in specific audiences, thereby further filtering out advertising creatives with higher relevance based on semantic matching.

[0024] Other features and advantages of various embodiments of this specification will be further revealed in the following detailed description and accompanying drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram illustrating the multimodal intelligent retrieval of advertising materials provided in this manual.

[0027] Figure 2 This is a flowchart illustrating the multimodal intelligent retrieval method for advertising materials provided in this manual.

[0028] Figure 3 This is a schematic diagram of the method for extracting the content description of the advertising materials provided in this specification.

[0029] Figure 4 This is a schematic diagram illustrating the process of constructing the material index library provided in this manual.

[0030] Figure 5 This is a schematic diagram of the method for obtaining multimodal joint representation vectors provided in this specification.

[0031] Figure 6 This is a schematic diagram of the multimodal intelligent retrieval system for advertising materials provided in this manual.

[0032] Figure 7 This is a schematic diagram of the electronic device provided in this manual. Detailed Implementation

[0033] The technical solutions of the embodiments of this specification will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of this specification and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of this specification.

[0034] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0035] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to facilitate the description of the embodiments and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this specification.

[0036] All data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0037] Before introducing the technical solutions described in this manual, the application scenarios and related technologies of the technical solutions will be introduced.

[0038] In today's highly competitive digital marketing environment, the pace of ad content updates is accelerating. Advertisers and creative teams not only rely on the efficient reuse of historical materials to improve production efficiency, but also continuously invest significant resources in creating entirely new, original ad creatives to meet diverse needs such as differentiated communication, personalized recommendations, and platform algorithm preferences. These new creatives are often acquired through various methods, including professional shooting, AI generation (such as text-to-image, video compositing, and voice cloning), user-generated content (UGC) collection, and cross-platform adaptation, encompassing multimodal content such as short videos, animated graphics, interactive audio, and 3D models. However, whether it's high-cost customized professional content or lightweight creatives quickly generated using AIGC tools, without a unified and intelligent management system, they are prone to becoming "produced and disseminated immediately, used and discarded."

[0039] This resulted in materials being stored piecemeal across different systems, making them unable to be effectively labeled, retrieved, or reused. This led to a significant waste of the creative and production costs invested upfront. Especially with the widespread adoption of AIGC (Artificial Intelligence Generated Content), the output of materials has surged. Without structured data storage, semantic understanding, and performance tracking mechanisms, the massive influx of new materials will only exacerbate management chaos and reduce overall operational efficiency.

[0040] Therefore, this specification provides an AI-based multimodal intelligent retrieval method and system for advertising creatives. Please refer to the appendix. Figure 1 The system utilizes artificial intelligence technology to achieve efficient management and accurate retrieval of multimodal advertising creatives 10. Specifically, it receives multimodal advertising creatives 10, including videos, images, audio, and text, and extracts content descriptions 11 from them. Then, it uses a multimodal fusion model to convert these creatives into a unified multimodal joint representation vector 21, and uses this to construct a creative index library 22. When it receives new advertising demand information 31 provided by a user, including at least one of text descriptions, target audience descriptions, and placement scenario descriptions, the system extracts the embedding vector of the demand information 31 and performs a retrieval based on the similarity between the embedding vector and each multimodal joint representation vector 21 in the creative index library 22. It initially filters out advertising creatives 10 with similarity higher than a preset threshold as candidate results. Based on this, it further generates predicted effect tendencies for each candidate creative, such as feelings of joy, anxiety, surprise, and low price perception, and optimizes and filters the candidate results according to these predicted effect tendencies. Finally, it outputs the retrieval result 32 that best matches the new advertising objective, thereby achieving the technical effect of intelligently recommending high-potential advertising content from massive historical and new creatives.

[0041] This manual provides an AI-based multimodal intelligent retrieval method for advertising creatives. Please refer to the appendix. Figure 2 This includes the following steps:

[0042] Step S1) Receive multimodal advertising material 10 and extract the content description 11 of the advertising material 10.

[0043] Advertising creatives (10) encompass various formats including video, images, audio, and text. They are core digital assets accumulated or newly generated by advertising companies during daily operations, creative production, and AIGC (AI-generated content). To achieve a unified understanding and efficient utilization of these heterogeneous creatives, deep semantic analysis of their content is required to form a structured, computable content description (11). Please refer to the appendix. Figure 3 The method for extracting the content description 11 of the advertising material 10 includes:

[0044] Step S11) Extract keyframes from the video footage and use a visual recognition model to generate image semantic labels and scene descriptions.

[0045] For example, a 30-second beverage advertisement video may contain multiple shot transitions. Representative frames are extracted using keyframe sampling techniques (such as motion change or shot boundary detection), and then a pre-trained visual model is used to identify objects (such as "ice cubes," "glass bottles," and "sunlight"), human actions (such as "raising a glass" and "smiling"), and the overall scene (such as "summer beach" and "outdoor party"), and to generate a scene description in natural language, such as "Young people enjoy iced drinks on a sunny beach."

[0046] Step S12) Perform speech recognition on the audio material to transcribe it into text, and use a sentiment analysis model to extract tone, emotion, and background sound type. For example, an advertisement voiceover "This summer, it should be refreshing!" is processed into corresponding text after speech recognition; at the same time, the system determines its tone as "light and energetic" and its emotional tendency as "positive and cheerful" through acoustic feature analysis (such as fundamental frequency, energy, and speech rate), and identifies the background sound as "ocean waves + upbeat guitar", thereby supplementing non-verbal emotional and atmospheric information.

[0047] Step S13) Use object detection and image classification models on the image material to generate a semantic description that includes the identified object, style, color, and composition. For example, a poster image processed can output: "The main subject is a red canned soda placed in the center of a white background, adopting a minimalist design style, with a red and white contrast as the main color, a symmetrical composition, and emphasizing the product outline and brand logo." This type of description not only covers content elements but also captures visual aesthetic features, which is crucial for matching specific brand tone requirements.

[0048] Step S14) Perform natural language processing on the text material to extract keywords, themes, sentiment descriptions, and semantic summaries. For example, an advertising copy "Limited-time 50% off! Miss it today, or you'll have to wait another year!" can be processed by an NLP model to extract the keywords "limited-time," "50% off," and "promotion," identify the theme as "price promotion," and the sentiment as "urgency + excitement," and generate the summary: "High-discount promotion emphasizing time sensitivity."

[0049] Step S15) The image semantic tags, scene descriptions, tone, emotion, background sound type, identifying objects, style, color, composition semantic descriptions, keywords, themes, sentiment descriptions, and semantic summaries are structurally fused to form content description 11. Content description 11 is stored in a standardized metadata format, which not only preserves the fine-grained features of each modality but also achieves cross-modal association through semantic alignment. For example, a complete material entry may contain fields such as: {"objects":["soft drink","ice cubes"],"scene":"summer outdoor","emotion":"cheerful","style":"fresh and simple","promotion_type":"limited-time discount","audio_mood":"energetic","color_palette":["#FF0000","#FFFFFF"]}.

[0050] The original unstructured multimodal data is transformed into a structured content description that is rich in semantics and can be understood and computed by AI models, thus providing a foundation for the subsequent construction of multimodal joint representations.

[0051] Step S2) Use a multimodal fusion model to convert the multimodal advertising material 10 and content description 11 into a unified multimodal joint representation vector 21, and build a material index library 22.

[0052] Specifically, the method for converting the multimodal advertising material 10 into a unified multimodal joint representation vector 21 using a multimodal fusion model includes:

[0053] Embedding and encoding are performed on video, image, audio, and text materials, as well as their content descriptions, to obtain embedding vectors for each modality.

[0054] The embedding vectors of different modalities are aligned in a unified semantic space through a pre-defined shared semantic space mapping;

[0055] An attention mechanism is used to perform weighted fusion of the aligned multimodal embeddings to obtain a multimodal joint representation vector 21;

[0056] The multimodal joint representation vector 21 is normalized and stored.

[0057] After completing the semantic analysis of the content of various advertising materials 10, it is necessary to further transform them into a computable and comparable numerical representation to support efficient similarity retrieval and intelligent matching. To this end, this method introduces a specially designed multimodal fusion model, which maps the original materials and their structured content descriptions 11 together into a unified semantic vector space, generating a highly expressive multimodal joint representation vector 21. First, the video, image, audio, and text materials and their corresponding content descriptions 11 are embedded and encoded respectively to obtain the embedding vectors of each modality.

[0058] For example, for a "Summer Soda" advertisement video, the system not only uses a visual encoder (such as VideoMAE or TimeSformer) to extract the visual features of its original frame sequence, but also converts the structured description generated in step S1 (such as "scene: beach; mood: cheerful; main color: red and white") into a semantic vector through a text encoder. Similarly, the audio part retains the original acoustic features (encoded by models such as Wav2Vec 2.0) and also integrates its transcribed text and sentiment tag embeddings; the images and text are also dual-encoded using a CLIP image encoder and a language model, respectively.

[0059] Embedsion vectors from different modalities are aligned in a unified semantic space through a pre-defined shared semantic space mapping. Since the original embedding distributions of each modality differ (e.g., image vectors and text vectors have different dimensions or scales), a cross-modal alignment mechanism is employed, such as a loss function based on contrastive learning, or a learnable linear / nonlinear projection layer to map the vectors of each modality to the same high-dimensional semantic space. In this space, the image features of a "red soda bottle," the text description of "refreshing summerdrink," and the acoustic features of upbeat background music are all brought closer to semantically similar regions, achieving cross-modal semantic consistency. For example, in a well-trained model, the concept of "cheerful" exhibits a consistent vector direction regardless of whether it originates from a smiling video, upbeat music, or positive text.

[0060] An attention mechanism is used to weightedly fuse the aligned multimodal embeddings to obtain a multimodal joint representation vector21. Different modalities contribute differently to specific advertisements (e.g., promotional ads rely more on text keywords, while brand ads emphasize visual style). A multi-head cross-attention or gating fusion network is introduced to dynamically evaluate the importance of each modality and assign weights. For example, in an e-commerce ad with "limited-time 50% off" as its core information, the text modality may be given higher attention weight; while in a perfume ad emphasizing atmosphere, the visual and audio modalities dominate. Finally, the modal vectors are fused according to their weights to generate a multimodal joint representation vector21 that comprehensively reflects the overall semantics, emotion, and style of the advertisement.

[0061] Finally, the multimodal joint representation vector 21 is normalized and stored. To facilitate subsequent similarity calculations (such as cosine similarity), all joint representation vectors are L2 normalized to lie on a unit hypersphere. The normalized vectors, along with their corresponding unique material IDs and metadata (such as creation time, modality type, content description, etc.), are written into a vector database (such as FAISS, Milvus, or Pinecone) to construct an efficient and scalable material index 22.

[0062] Heterogeneous and fragmented advertising creatives10 are transformed into vector representations with unified structure and semantic alignment. This not only preserves the complementarity of multimodal information but also provides a solid foundation for subsequent semantic-based precise retrieval, effect prediction, and intelligent recommendation. When a user enters "I want a summer beverage ad that conveys a refreshing feeling, is suitable for young people, and features energetic music," the index can quickly locate historical or newly created creatives that are semantically closest to the target audience, improving the efficiency of creative use and the accuracy of ad placement.

[0063] Please refer to the appendix for details. Figure 4 The methods for constructing the material index library 22 include:

[0064] Step S21) Use the multimodal joint representation vector 21 corresponding to each advertising material 10 as the index key to associate the unique index identifier, content description 11 and modality type information of the advertising material 10.

[0065] After multimodal fusion is completed, each ad creative 10 (regardless of whether it is video, image, audio, or text) is mapped to a high-dimensional, normalized multimodal joint representation vector 21 (e.g., 768-dimensional). The multimodal joint representation vector 21 serves as the digital fingerprint of the creative in the semantic space and is used as the core key value for indexing. Simultaneously, it is strongly associated with a series of metadata, including: a unique index identifier (such as a UUID or internal creative ID): used to accurately locate the original creative file; a structured content description 11: i.e., the semantic metadata generated in step S15, such as "emotion: cheerful," "scene: summer beach," "primary color: red and white," etc.; and modal type information: indicating whether the creative is a combination of "video + audio," "pure text and image," or "AI-generated speech," etc.

[0066] For example, a 30-second short video clip with the ID "A20250615_003" has its joint representation vector V1 associated with the metadata: {"modality":"video+audio","description":{"scene":"urban café","emotion":"relaxed","objects":["coffee cup","laptop"]","style":"minimalist"}}. This binding structure supports efficient vector retrieval while preserving interpretability and business context.

[0067] Step S22) The multimodal joint representation vector 21 is organized and sorted using an approximate nearest neighbor index structure to construct a vector index. Since the advertising material library 10 typically contains hundreds of thousands or even millions of records, using brute-force search to calculate the similarity between the query vector and all material vectors would result in unacceptable latency. Therefore, an ANN indexing algorithm is used to perform hierarchical clustering, quantization, or graph construction on high-dimensional vectors to achieve "approximate but fast" similarity lookup.

[0068] Taking HNSW (Hierarchical Navigable Small World) as an example, by constructing a multi-layered graph structure, it enables rapid navigation at higher levels and fine-grained searching at lower levels, returning Top-K most similar results from millions of creative materials within milliseconds. For instance, when the query vector represents "high-end skincare products + tranquil atmosphere + female audience," the HNSW index can quickly locate semantically similar clusters of creative materials, such as previously run "nighttime repair essence" ad videos or "SPA experience" promotional images, without having to traverse the entire database.

[0069] Step S23) Store the vector index and the unique index identifier of the advertising material 10 in the database, and construct the material index library 22 based on the database.

[0070] The ANN index structure and the source material metadata are persistently stored in a high-performance vector database. This database not only supports vector similarity retrieval but also supports metadata-based hybrid filtering (such as "return only video materials" or "exclude expired brands").

[0071] In a real-world deployment, the content index library 22 might be indexed by Milvus using vectors, while PostgreSQL or MongoDB might be responsible for storing metadata. The two are linked by a unique ID. When an advertising planner initiates a new requirement search, the system first performs a vector query in Milvus to obtain a list of candidate IDs, and then retrieves business information such as the preview link, copyright status, and historical CTR of the corresponding content from the relational database, forming a complete search result 32 pages.

[0072] Step S3) Receive new advertising demand information 31 provided by the user, wherein the demand information 31 includes at least one of content description 11, target audience description and placement scenario description.

[0073] In the early stages of advertising planning, creative teams, marketers, or automated delivery systems propose initial ideas for new advertisements based on business objectives. This requirement information 31 is received and parsed in a structured manner, transforming it into semantic input that can be understood and matched by the AI ​​model, thereby driving subsequent intelligent material retrieval. The requirement information 31 does not need to completely cover all dimensions, but must contain at least one of the following three categories:

[0074] Content Description 11: This refers to the user's natural language expression of the core creative idea, theme, style, or key elements of the advertisement. For example, a marketing specialist might enter: "I want a summer beverage ad that highlights the feeling of 'refreshing and thirst-quenching,' with visuals of splashing water and condensation droplets, a blue-white color scheme, and a brisk pace." This type of description directly corresponds to the visual, auditory, or copywriting style of the advertisement and is the most intuitive semantic anchor in creative matching.

[0075] Target Audience Description: This refers to the profile of the people the advertisement is expected to reach. It can be structured tags (e.g., "women aged 18-25," "Generation Z in first-tier cities," "fitness enthusiasts") or natural language (e.g., "targeting recent graduates, young professionals focused on cost-effectiveness and convenience"). This information helps the system prioritize creatives that have historically performed well with similar audiences. For example, if the target audience is "middle-aged men in third- and fourth-tier cities," the system will be more inclined to return creatives emphasizing emotional connotations such as "affordability," "family," and "durability," rather than content related to "trendy" or "tech."

[0076] Scenario Description: This refers to the specific environment or channel context in which the ad campaign will be displayed, including platform type (e.g., "Douyin feed," "WeChat Moments," "elevator screen"), time period (e.g., "evening commute"), device (e.g., "mobile vertical screen"), and even the competitive environment (e.g., "pre-sale period"). Different scenarios have a significant impact on the format and tone of the creative materials. For example, for the need for "Douyin short video splash screen ads," priority will be given to matching video materials with strong visual impact in the first 3 seconds, with subtitles, and adapted to the vertical screen ratio; while if the scenario is "high-end shopping mall digital screens," high-quality, slow-paced brand image videos without narration are more likely to be recommended.

[0077] The above three types of information can appear in combination. For example, when a brand launches a new product, users may submit the following combined requirements: "Content: Highlight the new product's '0 sugar 0 fat' health attributes and use a fresh green color scheme; Target audience: Urban women aged 25-35 who value health; Scenario: Xiaohongshu information feed and offline gym screens."

[0078] These three parts will be analyzed separately and then merged into a unified demand embedding vector in subsequent steps. Even if users only provide vague or fragmented input, such as simply writing "beverage ads suitable for young people," the input can be reasonably supplemented through default strategies, such as adding typical young audience profiles or popular advertising scenarios. By flexibly receiving multi-dimensional demand information, we can not only find similar materials, but also find high-value materials that are relevant to the right people, in the right scenarios, and convey the right information, significantly improving advertising creation efficiency and targeting accuracy.

[0079] Step S4) Extract the embedding vector of the demand information 31, and search in the material index library 22 according to the similarity between the embedding vector and the multimodal joint representation vector 21 to obtain advertising materials 10 with similarity higher than a preset threshold as candidate results.

[0080] Please refer to the appendix for details. Figure 5 The method for extracting the embedding vector of the demand information 31 includes:

[0081] Step S41) Semantically encode the user-input content description 11 to generate a first text embedding vector. Content description 11 is typically input in free text form, such as: "Highlighting the concepts of 'natural' and 'additive-free,' the visuals are clean and simple, with plant elements." A pre-trained semantic text encoder performs deep semantic understanding on this description, outputting a high-dimensional dense vector (e.g., 768 dimensions), called the first text embedding vector. This vector not only captures keywords (e.g., "natural," "plant") but also understands its contextual semantics (e.g., "clean and simple" implies a minimalist design style), thereby establishing semantic associations with advertisements in the resource library that have similar aesthetics or themes.

[0082] Step S42) Classifies and encodes the target audience description to generate an audience embedding vector. The target audience description may be a structured label (e.g., "Gender: Female, Age: 25–35, City: First-tier city, Interests: Yoga, Organic food") or natural language (e.g., "Urban white-collar women who focus on healthy lifestyles"). The system first parses it into standard demographic and behavioral feature fields, and then converts it into a fixed-dimensional audience embedding vector through a pre-built audience embedding mapping table or a lightweight neural network (e.g., a multilayer perceptron MLP). For example, the system can learn that the combination "25-35-year-old women + health interests" often co-occurs with material features such as "soft color tone," "soothing music," and "product close-up" in historical data, thus mapping it to the corresponding region in the vector space.

[0083] Step S43) Semantically parses the description of the delivery scenario and generates a scenario embedding vector using a scenario pre-trained model. Delivery scenario information, such as "Douyin short video feed, the first 3 seconds need to grab attention" or "offline subway light boxes, mainly static images," includes key constraints such as platform, format, duration, and interaction method. The system uses a specially fine-tuned scenario understanding model to parse these descriptions and generate scenario embedding vectors. This vector implicitly encodes information such as platform characteristics (e.g., Douyin prefers fast-paced, high-contrast content), device adaptation (portrait / landscape), and attention window (the golden time of the first 3 seconds). For example, even if "Xiaohongshu product recommendation videos" and "elevator advertising screens" have the same content theme, their scenario embedding vectors will be significantly different, thus guiding the system to return materials adapted to the characteristics of each media.

[0084] Step S44) The first text embedding vector, audience embedding vector, and scene embedding vector are fused using a cross-modal fusion network to obtain the embedding vector of demand information 31. Since the three types of vectors have different sources and semantic emphases, direct concatenation may lead to information imbalance. Therefore, a cross-modal fusion network is used to dynamically weight and integrate the three. This network can adaptively determine the importance of each dimension based on the input content. For example, when the user only provides "content description 11" without specifying the audience and scene, the model will automatically reduce the weight of the latter two; conversely, if the demand emphasizes "targeting the elderly for television advertising," then the audience and scene vectors will dominate the fusion result. Finally, a unified embedding vector of demand information 31 is output, comprehensively reflecting the overall semantic intent of this advertising task.

[0085] Step S45) Perform the same normalization process on the embedded vector of the demand information 31 as on the multimodal joint representation vector 21. Perform L2 normalization on the fused vector, identical to that on the material library vector, to make it fall on the unit hypersphere. This step is crucial for achieving cross-domain alignment of "demand materials," ensuring fair comparison even between vectors from different generation paths at the same scale.

[0086] Suppose a user inputs the following requirements: "Content: A newly launched plant-based yogurt that emphasizes a pure formula; Target audience: Mothers aged 30-45 who value health; Scenario: WeChat Moments feed, 9:00-11:00 AM."

[0087] The text vectors for "plant-based yogurt" and "pure formula" will be generated using Sentence-BERT encoding; "30-45 year old mothers + health concern" will be mapped to a typical family health audience vector; "WeChat Moments" will be identified through a scene model as medium-length, lifestyle-oriented material suitable for emotional resonance; after integrating the three, the vectors will be normalized to obtain the final query vector.

[0088] The query vector is sent to the material index library 22 to quickly retrieve video materials of the "family kitchen scene, soft lighting, product close-up, and warm narration" type that have performed well in the past. For example, a dairy company's past advertisement "Mom prepares breakfast for her child" has a cosine similarity of 0.87 between its multimodal joint representation vector 21 and the query vector (higher than the preset threshold of 0.75) and is included in the candidate results.

[0089] The method for retrieving data in the material index 22 based on the similarity between the embedded vector and the multimodal joint representation vector 21 includes:

[0090] The normalized embedding vector is used as the query vector and compared with the multimodal joint representation vector 21 of the material to obtain the similarity.

[0091] Based on the similarity in descending order, the top 10 advertising materials with similarity higher than the preset threshold are selected as candidate results.

[0092] The normalized demand information 31 is embedded into a vector and input as a query vector into the material index library 22. Since the material index library 22 has already organized all multimodal joint representation vectors 21 using an approximate nearest neighbor index structure (such as HNSW, IVF, etc.) during the construction phase, large-scale vector comparison can be completed in milliseconds without traversing all materials. Similarity calculation usually uses cosine similarity (since the vectors have been L2 normalized, it is equivalent to the vector dot product), and its value range is [-1, 1]. The closer the value is to 1, the more similar the semantics are.

[0093] For example, when a marketer submits a request for an advertisement for a shampoo featuring "natural ingredients," targeting women aged 25-35, to be used in short videos on Xiaohongshu (Little Red Book), the generated query vector is sent to an index containing 500,000 creative materials. The ANN engine quickly returns the top 100 most similar creative IDs and their similarity scores, such as:

[0094] Material A (ID:SH2024_089): Similarity 0.89, content is "plant extract shampoo + forest background + soft light lens", high historical CTR;

[0095] Material B (ID:SH2023_156): Similarity 0.85, contains "silicone-free" copywriting and features a female KOL, but the color tone is cool;

[0096] Material C (ID:CL2025_021): Similarity 0.72. Although the theme is related, it is a horizontal TV commercial format and has low adaptability.

[0097] Next, all returned results are sorted in descending order of similarity, and a dual filtering strategy is applied: threshold filtering, which retains only results with similarity higher than a preset threshold (e.g., 0.75) and excludes interference items with excessive semantic deviation; and number truncation, which retains at most the top N results (e.g., N=20) while meeting the threshold, in order to balance recall and user experience.

[0098] In the above case, material C was eliminated because its similarity was below 0.75, and the final candidate results were materials A, B, and 18 other highly matched materials. These candidates not only semantically align with user needs, but also naturally inherit the multimodal attributes of the original materials (such as video length, aspect ratio, and audio mood), and can be directly used for rapid prototyping or AI-driven secondary creation.

[0099] Through efficient retrieval and intelligent filtering based on vector similarity, massive and heterogeneous advertising materials are transformed into a pool of creative resources that can be precisely accessed, significantly improving the response speed and content relevance of new advertising production, and preventing high-quality materials from being idle and wasted due to "not being able to find" them.

[0100] Step S5) Generate the predicted effect tendency of the candidate results, filter the candidate results according to the predicted effect tendency, and output them as the search result 32.

[0101] Specifically, it includes:

[0102] The content description 11 and historical performance data of each candidate ad creative 10 are analyzed, and a pre-trained sentiment analysis model is used to predict the emotional response category that the ad creative 10 can evoke.

[0103] Extract the emotional need description from the demand information 31 to obtain the matching degree between the emotional need description and the emotional response category;

[0104] When the matching degree is higher than a preset threshold, the prediction effect tends to be positive; when the matching degree is not higher than the preset threshold, the prediction effect tends to be negative.

[0105] Candidate results that predict negative outcomes are eliminated, thus completing the candidate result selection process.

[0106] By employing a multimodal sentiment understanding model, the model comprehensively considers visual elements of the content (such as color saturation and facial expressions), audio emotion (such as passionate or soothing tone), copy keywords (such as "limited-time offer" and "enjoy quiet moments"), and historical campaign metrics (such as past likes, completion rates, and conversion rates among similar audiences) to predict the most likely emotional response category. These categories typically include, but are not limited to: joy, surprise, trust, urgency, anxiety, warmth, perceived low price, sense of luxury, nostalgia, and anxiety, forming a structured emotional tagging system. For example, a video clip showcasing "family dinner + hot soup + warm color scheme" might be predicted by the model to primarily evoke "warmth" (probability 0.82) and "trust" (probability 0.65); while a clip with fast-paced editing, red background and white text promotional copy, and countdown sound effects is predicted to evoke "urgency" (0.91) and "perceived low price" (0.88).

[0107] Extract the emotional need description from the demand information 31 to obtain the matching degree between the emotional need description and the emotional response category. The demand information 31 provided by the user may directly or indirectly contain emotional appeals. For example: explicit description: "Hope to convey a relaxed and pleasant feeling"; implicit clues: "Summer drinks targeting young people" (implying "cheerful" and "refreshing"); scenario hints: "Launch during the pre-sale period of a major promotion" (implying "sense of urgency" and "high cost performance").

[0108] The natural language understanding module extracts the desired emotional needs labels (such as "joyful" and "surprise") and calculates the matching degree between them and the predicted emotional responses of the candidate materials. The matching degree can be quantified by label overlap, semantic similarity (such as using the distance in the emotional word vector space), or weighted multi-label classification confidence. For example, if the desired emotion is "joyful" and the predicted emotion of the material is "joyful (0.75) + surprise (0.60)," the matching degree can be set to 0.75; if the main emotion of the material is "anxiety" or "seriousness," the matching degree is close to 0.

[0109] Set an emotional matching threshold to determine the predicted effect tendency: when the matching degree is higher than the preset threshold (e.g., 0.6), the predicted effect tendency of the material is determined to be positive, and its emotional tone is considered to be consistent with the business objectives, with good potential for placement; when the matching degree is not higher than the threshold, it is determined to be negative, and even if the semantics are similar, the emotional mismatch may lead to user aversion or poor conversion.

[0110] For example, when promoting a newly launched children's educational toy, the user demand emphasizes "stimulating curiosity and bringing a sense of surprise." One piece of material was identified from the candidate set, which showed a child's eyes lighting up after opening the packaging, accompanied by upbeat background music. The model predicted its main emotions to be "surprise (0.88)" and "curiosity (0.76)," which highly matched the demand (match degree 0.88 > 0.6), so it was marked as positive. Another piece of material, although related to the theme, adopted a calm explanation style and had a relatively static picture. The predicted emotions were "focus (0.70)" and "rationality (0.65)," which lacked the "surprise" element. The match degree was only 0.35, so it was judged as negative.

[0111] Candidate results with a negative predictive bias are filtered out, completing the final selection, and the remaining materials are output as search results 32. After sentiment alignment filtering, only high-quality materials that are both semantically relevant and emotionally resonant are retained. This not only improves the business effectiveness of the recommendation results but also avoids the risk of campaign failure due to "technical matching but emotional inconsistency".

[0112] For example, a beauty brand plans to launch a "Romantic Gift Box" advertisement before Valentine's Day, with the requirement described as: "Creating a sweet and heartwarming atmosphere, targeting couples aged 20-30." The initial search returned 20 semantically relevant materials. One material showed the product in a candlelight dinner scene, with a soft-focus lens and piano music; its sentiment prediction was "romantic (0.90)" and "heartwarming (0.85)", and it was retained due to a match of 0.90. Another material, although containing the keyword "gift box," used a laboratory-style shooting style and emphasized ingredient technology; its sentiment prediction was "professional (0.82)" and "calm (0.75)", and it was removed due to a match of only 0.2. The final search results (32) all focused on the "romantic emotion" theme, improving the efficiency of creative reuse and the accuracy of ad placement.

[0113] On the other hand, this specification provides an AI-based multimodal intelligent retrieval system for advertising creatives; please refer to the appendix. Figure 6 ,include:

[0114] The first receiving module 100 receives multimodal advertising material 10 and extracts the content description 11 of the advertising material 10;

[0115] The conversion module 200 uses a multimodal fusion model to convert the multimodal advertising material 10 and content description 11 into a unified multimodal joint representation vector 21, and constructs a material index library 22;

[0116] The second receiving module 300 receives new advertising demand information 31 provided by the user. The demand information 31 includes at least one of content description 11, target audience description, and placement scenario description.

[0117] The extraction module 400 extracts the embedding vector of the demand information 31, and searches the material index library 22 based on the similarity between the embedding vector and the multimodal joint representation vector 21 to obtain advertising materials 10 with similarity higher than a preset threshold as candidate results.

[0118] The output module 500 generates the predicted effect tendency of the candidate results, filters the candidate results according to the predicted effect tendency, and outputs them as the search result 32.

[0119] Please see Figure 7 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this specification.

[0120] like Figure 7 As shown, the electronic device 1100 may include: at least one processor 1101, at least one network interface 1104, a user interface 1103, a memory 1105, and at least one communication bus 1102. The communication bus 1102 can be used to connect and communicate with the various components mentioned above. The user interface 1103 may include buttons, and optionally may include standard wired or wireless interfaces. The network interface 1104 may include, but is not limited to, a Bluetooth module, an NFC module, or a Wi-Fi module. The processor 1101 may include one or more processing cores. The processor 1101 connects to various parts within the electronic device 1100 using various interfaces and lines, and performs various functions of the routing device and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1105, and by calling data stored in the memory 1105. Optionally, the processor 1101 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 1101 may integrate one or more combinations of CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content that the display screen needs to show; and the modem is used for wireless communication.

[0121] It is understandable that the aforementioned modem may not be integrated into the processor 1101, but may be implemented using a separate chip.

[0122] The memory 1105 may include RAM or ROM. Optionally, the memory 1105 may include a non-transitory computer-readable medium. The memory 1105 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1105 may also be at least one storage device located remotely from the aforementioned processor 1101. As a computer storage medium, the memory 1105 may include an operating system, a network communication module, a user interface module, and application programs. The processor 1101 may be used to call the application programs stored in the memory 1105 and execute the methods in the above-described embodiments.

[0123] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform multiple steps as described in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0124] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the multiple steps described in the above embodiments.

[0125] Where there is no conflict, the technical features in this embodiment and implementation scheme can be combined arbitrarily.

[0126] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes multiple computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating multiple available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0127] When implemented through hardware or firmware, the aforementioned method flow is programmed into the hardware circuit to obtain the corresponding hardware circuit structure and achieve the corresponding function. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit, whose logic function is determined by the user programming the device. Designers can program a digital system onto a PLD themselves, eliminating the need for chip manufacturers to design and fabricate dedicated integrated circuit chips. Furthermore, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, similar to the software compiler used in program development. The original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There is not just one HDL, but many. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of the aforementioned hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logic method flow can be easily obtained.

[0128] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims of this specification.

Claims

1. An AI-based multimodal intelligent retrieval method for advertising creatives, characterized in that, Includes the following steps: Receive multimodal advertising materials and extract the content description of the advertising materials; The multimodal advertising materials and content descriptions are converted into a unified multimodal joint representation vector using a multimodal fusion model, and a material index library is constructed. Receive new advertising request information provided by users, the request information including at least one of content description, target audience description and placement scenario description; The embedding vector of the demand information is extracted, and based on the similarity between the embedding vector and the multimodal joint representation vector, a search is performed in the material index library to obtain advertising materials with a similarity higher than a preset threshold as candidate results. The predicted effect tendency of the candidate results is generated, and the candidate results are filtered according to the predicted effect tendency and output as the search results. The method for generating the prediction effect tendency of the candidate results and filtering the candidate results based on the prediction effect tendency includes: The content description and historical performance data of each candidate ad creative are analyzed, and a pre-trained sentiment analysis model is used to predict the category of emotional response that the ad creative can evoke. Extract the emotional demand description from the demand information, and obtain the matching degree between the emotional demand description and the emotional response category; When the matching degree is higher than a preset threshold, the prediction effect tends to be positive; when the matching degree is not higher than the preset threshold, the prediction effect tends to be negative. Candidate results that predict negative outcomes are eliminated, thus completing the candidate result selection process.

2. The AI-based multimodal intelligent retrieval method for advertising creatives according to claim 1, characterized in that, The advertising material may be a video, image, audio, or text, and the method for extracting the content description of the advertising material includes: Keyframes are extracted from video footage, and a visual recognition model is used to generate image semantic labels and scene descriptions. Speech recognition is performed on audio materials to transcribe them into text, and a sentiment analysis model is used to extract tone, emotion, and background sound type. Using object detection and image classification models on image materials, generate semantic descriptions that include identified objects, style, color, and composition. Natural language processing is performed on the text material to extract keywords, themes, sentiment descriptions, and semantic summaries; The image semantic tags, scene descriptions, tone, emotion, background sound type, identifying objects, style, color, compositional semantic descriptions, keywords, themes, sentiment descriptions, and semantic summaries are structurally integrated to form a content description.

3. The AI-based multimodal intelligent retrieval method for advertising materials according to claim 2, characterized in that, Methods for converting multimodal advertising creatives into a unified multimodal joint representation vector using a multimodal fusion model include: Embedding vectors for each modality are obtained by embedding and encoding video, image, audio, and text materials and their content descriptions respectively. The embedding vectors of different modalities are aligned in a unified semantic space through a pre-defined shared semantic space mapping; An attention mechanism is used to perform weighted fusion of the aligned multimodal embeddings to obtain a joint multimodal representation vector; The multimodal joint representation vector is normalized and stored.

4. The AI-based multimodal intelligent retrieval method for advertising creatives according to claim 3, characterized in that, The methods for extracting the embedding vector of the demand information include: The user input is semantically encoded to generate a first text embedding vector. Classify and encode the target audience description to generate audience embedding vectors; Semantic parsing of the deployment scenario description is performed, and scenario embedding vectors are generated through a scenario pre-trained model; The first text embedding vector, audience embedding vector, and scene embedding vector are fused through a cross-modal fusion network to obtain the embedding vector of demand information; The embedding vector of the demand information is subjected to the same normalization process as the multimodal joint representation vector.

5. The AI-based multimodal intelligent retrieval method for advertising creatives according to claim 4, characterized in that, The method for retrieving data from the material index based on the similarity between the embedded vector and the multimodal joint representation vector includes: The normalized embedding vector is used as the query vector and compared with the multimodal joint representation vector of the material to obtain the similarity. Based on the similarity in descending order, the top preset number of advertising materials with similarity higher than a preset threshold are selected as candidate results.

6. The AI-based multimodal intelligent retrieval method for advertising creatives according to claim 1, characterized in that, Methods for building a material index library include: Each ad creative's multimodal joint representation vector is used as an index key to associate the ad creative's unique index identifier, content description, and modality type information. The multimodal joint representation vector is organized and sorted using an approximate nearest neighbor index structure to construct a vector index; The vector index and the unique index identifier of the advertising material are stored in the database, and a material index library is constructed based on the database.

7. An AI-based multimodal intelligent retrieval system for advertising creatives, characterized in that, include: The first receiving module receives multimodal advertising materials and extracts the content description of the advertising materials; The conversion module uses a multimodal fusion model to convert the multimodal advertising materials and content descriptions into a unified multimodal joint representation vector, and builds a material index library; The second receiving module receives new advertising demand information provided by the user, the demand information including at least one of content description, target audience description and placement scenario description; The extraction module extracts the embedding vector of the demand information, and searches the material index library based on the similarity between the embedding vector and the multimodal joint representation vector to obtain advertising materials with a similarity higher than a preset threshold as candidate results. The output module generates the prediction effect tendency of the candidate results, filters the candidate results based on the prediction effect tendency, and outputs them as search results. The method for generating the prediction effect tendency of the candidate results and filtering the candidate results based on the prediction effect tendency includes: The content description and historical performance data of each candidate ad creative are analyzed, and a pre-trained sentiment analysis model is used to predict the category of emotional response that the ad creative can evoke. Extract the emotional demand description from the demand information, and obtain the matching degree between the emotional demand description and the emotional response category; When the matching degree is higher than a preset threshold, the prediction effect tends to be positive; when the matching degree is not higher than the preset threshold, the prediction effect tends to be negative. Candidate results that predict negative outcomes are eliminated, thus completing the candidate result selection process.

8. An electronic device, characterized in that, Including the processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images