User advertisement recommendation method and system based on fusion neural network

Abstract

The invention relates to the technical field of computers, and discloses a user advertisement recommendation method and system based on a fusion neural network. The method comprises the following steps: performing feature extraction on different modal data in a user behavior sequence; constructing an intra-modal element relation graph, and capturing a dependency relationship between elements by using a sub-graph attention network; a large-scale pre-training multi-mode model is adopted as a teacher network, and a lightweight student network is constructed through knowledge distillation; cross-modal contrast learning is introduced to enhance semantic alignment between modals; and fusing user situation information into a recommendation process by utilizing a context condition embedding technology. According to the invention, high-precision and high-efficiency advertisement recommendation is realized, and the method is especially suitable for a mobile terminal resource limited scene.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a user advertisement recommendation method and system based on a fusion neural network. Background Art

[0002] With the rapid development of the internet and mobile technology, multimodal information (text, images, videos, etc.) has become a major component of user online behavior. In ad recommendation scenarios, accurately understanding and integrating this multimodal information is crucial for improving recommendation quality. Effectively mining user interests and preferences is a major challenge for ad recommendation systems, especially in rich media environments like news feeds and short videos.

[0003] Existing advertising recommendation technologies have the following main problems: limited processing capabilities for multimodal data, difficulty in fully exploring the correlation between elements within the modalities, high computational complexity of large multimodal models, and difficulty in efficient deployment in resource-constrained environments. At the same time, they are not robust enough to modal missing or noise interference, resulting in unstable performance.

[0004] In response to the above technical problems, there is an urgent need to propose a user advertising recommendation method and system that can effectively integrate multimodal information and is suitable for efficient deployment in resource-constrained environments. Summary of the Invention

[0005] The present invention provides a user advertisement recommendation method and system based on a fusion neural network, which solves technical problems in the prior art such as limited multimodal fusion capability, difficulty in deploying complex models in resource-constrained environments, and poor robustness to modality loss.

[0006] The present invention provides a user advertisement recommendation method and system based on a fusion neural network, comprising: In a first aspect, a user advertisement recommendation method based on a fusion neural network comprises the following steps: Extract features from different modal data in user behavior sequences to obtain intra-modal representations; Based on the intra-modal representation, we construct an intra-modal element relationship graph to capture the dependencies between elements, and introduce a multi-scale temporal convolutional structure to simultaneously capture short-term, medium-term, and long-term behavioral sequence patterns. Based on the intra-modal element relationship graph, a large pre-trained multimodal model is used as the teacher network to extract the inter-modal attention mapping relationship, build a lightweight student network, and learn the attention distribution of the teacher network through knowledge distillation; Based on a lightweight student network, we introduce a cross-modal contrastive learning objective into the student network to enhance semantic alignment between modalities. Based on the teacher network and the lightweight student network, the contextual condition embedding technology is introduced to inject the user's current situation information as a conditional variable into the fusion process, and the advertising recommendation results are generated based on the fusion features; The relationship graph determines the weight of the edge by calculating the cosine similarity between elements, and the subgraph attention network processes the relationship graph to calculate the node representation.

[0007] Furthermore, the multimodal feature extraction step includes applying a pre-trained language model to process user text information to generate a text feature vector; applying a convolutional neural network to process the image content of user interaction to generate an image feature vector; combining a spatiotemporal convolutional network and frame sampling technology to process the video content watched by the user to generate a video feature vector; and recording the time information of user behavior to extract a time series feature vector.

[0008] Furthermore, the node representation calculation formula of the subgraph attention network is: ;in, Indicates the Nodes in the layer The representation vector, Indicates the Nodes in the layer The representation vector, Representation node The neighbor set of is the learnable weight matrix, is the activation function, is the attention coefficient, Indicates the node No. The sum of neighbor nodes.

[0009] Furthermore, the calculation formula of the attention coefficient is: ;in, is the attention vector, is the transpose of the attention vector, represents transpose, Represents vector concatenation operation, is the learnable weight matrix for attention calculation, represents the exponential function, is the LeakyReLU activation function, is the sample index, Indicates the node All samples Sum, 、 and Represent the feature vectors of three different nodes respectively.

[0010] Furthermore, the multi-scale temporal convolution structure includes: ;in, 、 and They are short-term, medium-term and long-term behavioral sequence patterns, is the node representation matrix output by the graph attention network, represents a one-dimensional convolution operation, 、 and are the short-term, medium-term, and long-term convolution kernel sizes, 、 and is the corresponding output dimension.

[0011] Furthermore, the loss function of the knowledge distillation is: ;in, is the knowledge distillation loss function, is the number of attention heads, and Teacher and student networks The attention matrix of the attention head, Represents all attention heads From 1 to Sum, is the attention head index, , is the attention matrix of the teacher network, is the attention matrix of the student network, Represents the square of the L2 norm.

[0012] Furthermore, the loss function of the cross-modal contrastive learning is: ;in, is the contrastive learning loss function, is the modal representation of the anchor negative sample, is the modal representation of the anchor point positive sample, The modal representation includes a positive sample and a negative sample, is the similarity function, is the temperature parameter, represents the exponential function, represents the logarithmic function, For all samples Summation.

[0013] Furthermore, the modal fusion function of contextual condition fusion is: ;in, To fusion features, is the modal weight under contextual conditions, is the intra-modal feature processing function, For the A modal representation, is the relationship diagram of elements within the modal, is the number of modes, is the modal index, , For all modes From 1 to Sum, Represents a multiplication operation.

[0014] Furthermore, the modal weight calculation formula under the contextual conditions is: ;in, is the modal weight under contextual conditions, Generate a network for parameterized conditional weights, are network parameters, is the context feature vector, For the A modal representation, is the total number of modes, Indicates the Mode 1 to Sum, Represents the exponential function.

[0015] The second aspect is the user advertising recommendation system based on the fusion neural network, including: Multimodal feature extraction module, used to extract features from different modal data in user behavior sequences; Intra-modal relationship modeling module, which is used to construct the intra-modal element relationship graph and process the graph through sub-graph attention network and multi-scale temporal convolution; The knowledge distillation module is used to achieve the transfer of attention knowledge through the teacher network and the student network; the cross-modal contrastive learning module is used to enhance the semantic alignment between modalities; Contextual condition fusion module, used to integrate user context information into modal features; The advertising recommendation module is used to generate advertising recommendation results based on fused features. The system uses knowledge distillation technology to compress complex models into lightweight models, enabling efficient deployment on mobile devices.

[0016] The beneficial effects of the present invention are as follows: the present invention extracts features from different modal data in user behavior sequences to obtain intra-modal representations; constructs an intra-modal element relationship graph to capture inter-element dependencies, and introduces a multi-scale temporal convolutional structure to simultaneously capture short-term, medium-term, and long-term behavior sequence patterns; uses a large pre-trained multimodal model as a teacher network to extract inter-modal attention mapping relationships, constructs a lightweight student network, and learns the attention distribution of the teacher network through knowledge distillation; introduces cross-modal comparative learning objectives in the student network to enhance inter-modal semantic alignment; introduces contextual conditional embedding technology to inject the user's current context information as a conditional variable into the fusion process, and generates advertising recommendation results based on the fused features; wherein the relationship graph determines the edge weights by calculating the cosine similarity between elements, and the subgraph attention network processes the relationship graph to calculate node representations; by introducing technical means such as multimodal feature extraction, intra-modal relationship modeling, multi-level attention distillation, cross-modal comparative learning, and contextual conditional fusion, the problems of limited multimodal data processing capabilities, difficulty in efficient deployment of complex models in resource-constrained environments, and insufficient robustness under modal missing or noise interference in the existing technology are solved. The combination of these technologies significantly improves recommendation accuracy and computational efficiency, enhances modal fusion quality, and improves device adaptability. It enables efficient advertising recommendations on multiple devices and is particularly suitable for scenarios with limited mobile resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of the main steps of the user advertisement recommendation method based on the fusion neural network of the present invention;

[0018] Figure 2 is a flow chart of the multimodal feature extraction steps of the present invention;

[0019] Figure 3 is a flow chart of the intra-modal relationship modeling steps of the present invention;

[0020] Figure 4 is a flow chart of the multi-level attention distillation steps of the present invention;

[0021] Figure 5 is a flow chart of the cross-modal contrastive learning steps of the present invention;

[0022] Figure 6 It is a flow chart of the context condition fusion and recommendation steps of the present invention. DETAILED DESCRIPTION

[0023] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed solely to enable those skilled in the art to better understand and implement the subject matter described herein, and that the functions and arrangements of the elements discussed may be varied without departing from the scope of this specification. Various examples may omit, substitute, or add various processes or components as needed. Furthermore, features described in some examples may be combined in other examples.

[0024] At least one embodiment of the present invention discloses a user advertisement recommendation method and system based on a fusion neural network, including:

[0025] Implementation Method 1

[0026] This implementation is suitable for advertising recommendation scenarios that integrate multimodal information, particularly in rich media environments such as information streams and short videos. This scenario presents the following technical requirements: First, it is necessary to effectively integrate multimodal information (text, images, videos, etc.) in user behavior sequences to achieve high-precision user interest modeling; second, it is necessary to address the limited computing resources of mobile terminals and ensure that the recommendation system can operate efficiently in resource-constrained environments.

[0027] Traditional recommendation technology has shortcomings in the following aspects: first, its ability to integrate and process multimodal data is limited, making it difficult to fully explore the correlation between elements within the modalities; second, complex models are difficult to deploy on mobile terminals with limited computing resources; third, it is impossible to effectively utilize the advantages of large pre-trained models in multimodal understanding.

[0028] To solve the above problems, this embodiment proposes a user advertising recommendation method based on a fusion neural network, which achieves high-precision and high-efficiency advertising recommendation through technical means such as multimodal feature extraction, intra-modal relationship modeling, multi-level attention distillation, cross-modal comparative learning, and contextual condition fusion.

[0029] like Figures 1 to 6 As shown, the user advertisement recommendation method based on the fusion neural network of this embodiment includes the following steps:

[0030] Step 1: Multimodal feature extraction.

[0031] Use modality-specific feature extractors to process multimodal data in user behavior sequences and generate high-quality intra-modal representations. Specifically, this includes:

[0032] Step 1.1: Text feature extraction.

[0033] A pre-trained language model is applied to process textual information from historical user interactions (such as search keywords, browsed titles, and comment content) to generate a text feature vector T. Specifically, the encoder-based BERT model or its lightweight variant can be used. This model contains a multi-layer bidirectional encoder, each consisting of a multi-head self-attention layer and a feedforward neural network layer. For mobile scenarios, compressed model variants such as DistilBERT or TinyBERT, which have fewer parameters, can be used to reduce computational resource requirements while maintaining text comprehension capabilities. In e-commerce advertising scenarios, this model can effectively capture the semantic matching relationship between user search terms and product titles, for example, associating semantically similar descriptions of thin and light notebooks with ultra-thin portable computers.

[0034] Step 1.2: Image feature extraction.

[0035] A convolutional neural network is applied to the image content with which the user has interacted, extracting the image feature vector I. In specific implementations, convolutional neural network architectures such as ResNet or MobileNet can be used. These networks consist of multiple convolutional layers, pooling layers, and residual connections. For mobile scenarios, the lightweight MobileNetV3 network is preferred. This network uses depthwise separable convolutions and squeeze-and-excitation modules to significantly reduce the number of parameters and computational complexity. In practical applications, the network can be pre-trained on large-scale image datasets (such as ImageNet) and then fine-tuned to target advertising image features. For example, in the context of fashion product advertisements, this model can effectively identify and extract visual features such as clothing style, color, and texture, enabling the system to recommend similar or complementary products based on the user's historically viewed clothing styles.

[0036] Step 1.3: Video feature extraction.

[0037] Combining spatiotemporal convolutional networks and frame sampling techniques, features of the video content viewed by the user are extracted to generate a video feature vector V. Specifically, a 3D convolutional neural network (such as C3D or I3D) or SlowFast network architecture can be used, as these networks can simultaneously capture both the spatial and temporal dimensions of a video. In scenarios with limited computing resources, a video feature extraction method based on a 2D convolutional network can be used. For example, this method first uniformly samples the video to obtain key frames, then uses a 2D convolutional neural network to extract features for each key frame. Finally, a temporal aggregation function (such as average pooling or attention weighting) is used to combine frame-level features to obtain a video-level representation. In short video advertising scenarios, this method can effectively capture key visual elements and dynamic changes in the video. For example, it can identify the video genres that users prefer to watch (such as fast-paced sports content or lighthearted and funny content), thereby recommending video ads that are more in line with their interests.

[0038] Step 1.4: Behavior temporal feature extraction.

[0039] Record the time information of user behavior and extract the time series feature vector B.

[0040] Step 2: Modeling intra-modal relationships.

[0041] Construct a relationship graph of elements within a modality, apply a subgraph attention network to capture the dependencies between elements within a modality, and generate enhanced intra-modality representations. Specifically, this includes:

[0042] Step 2.1: Relationship map construction.

[0043] For each mode , building a relationship graph based on the semantic similarity between elements ,in, is a set of nodes (corresponding to elements in the sequence), is a set of edges (representing the relationship between elements). The weight of the edge is determined by calculating the cosine similarity between elements: in, is the cosine similarity between elements, and Respectively and A feature vector of elements.

[0044] Step 2.2: Subgraph Attention Network.

[0045] The relationship graph constructed by subgraph attention network processing is calculated to represent the nodes: in, Indicates the Nodes in the layer The representation vector, Indicates the Nodes in the layer The representation vector, Representation node The neighbor set of is the learnable weight matrix, is the activation function, is the attention coefficient, Indicates the node No. The sum of neighbor nodes.

[0046] In its specific implementation, the network adopts a graph attention network (GAT) structure, which consists of multiple graph attention layers. Each layer first calculates the attention weights between pairs of nodes, and then aggregates information about neighboring nodes based on these weights. To improve efficiency, a sparse attention calculation method can be used, which only considers the edges that actually exist in the graph to avoid full connection calculations. In system implementation, 2-3 layers of graph attention layers are usually stacked, and each layer is followed by a batch normalization layer and a residual connection to accelerate training and improve stability. In actual advertising recommendation applications, the network can effectively model the user's interest associations for different types of content. For example, in the news information advertising scenario, by capturing the relationship between the news topics read by users, the user's interest migration path (such as from technology news to smart home advertisements) can be identified, thereby achieving more accurate interest capture.

[0047] Step 2.3: Multi-scale temporal convolution.

[0048] Apply multi-scale temporal convolutional structures to simultaneously capture short-term, medium-term, and long-term behavioral sequence patterns: ;in, 、 and They are short-term, medium-term and long-term behavioral sequence patterns, is the node representation matrix output by the graph attention network, represents a one-dimensional convolution operation, 、 and are the short-term, medium-term, and long-term convolution kernel sizes, 、 and is the corresponding output dimension.

[0049] Step 2.4: Multi-scale feature fusion.

[0050] Fusion of temporal features at different scales to obtain the final intra-modal representation: in, is the final intra-modal temporal feature representation, represents the feature concatenation operation, and are the learnable weight matrix and bias vector respectively.

[0051] Step 3: Multi-level attention distillation.

[0052] Using a large pre-trained multimodal model as the teacher network, we extract high-quality inter-modal attention mapping relationships and build a lightweight student network through knowledge distillation to achieve efficient multimodal fusion. Specifically, we include:

[0053] Step 3.1: Teacher network construction and training.

[0054] A large pre-trained multimodal model is used to build a teacher network with rich cross-modal understanding capabilities. The teacher network receives multimodal input and calculates the attention matrix between modalities using a multi-head attention method: in, Teacher Network The attention matrix of the attention head, is the query matrix, is the bond matrix, represents transpose, is the dimension of the key vector, Used to normalize the attention score.

[0055] In specific implementations, the teacher network can be built based on pre-trained multimodal models such as CLIP, ViLBERT, or LXMERT. These models usually consist of two parts: a modality-specific encoder and a cross-modal fusion encoder. The modality-specific encoder is responsible for processing single-modal data, such as a text encoder (based on Transformer) and a visual encoder (based on VisionTransformer or ResNet); the cross-modal fusion encoder uses an attention layer to achieve interaction and integration of information from different modalities. In advertising systems, these models can be fine-tuned on advertising-related multimodal datasets to adapt to specific application scenarios. For example, in e-commerce live advertising scenarios, the teacher network can effectively understand the semantic association between the host's explanation content (text) and the displayed products (images), thereby accurately capturing the user's interest in the products.

[0056] Step 3.2: Student network construction.

[0057] Construct a lightweight student network and use a parameter-efficient architecture, such as depthwise separable convolution and low-rank decomposition, to reduce computational complexity. The student network also calculates the multi-head attention matrix: in, For students network The attention matrix of the attention head, and are the query and key matrices of the student network, respectively. is the dimension of the student network key vector.

[0058] In specific implementation, the student network can adopt a simplified multimodal Transformer architecture, including the following key optimizations: (1) Reduce the number of Transformer layers, such as from 12 layers to 4-6 layers; (2) Using depth-wise separable convolution instead of standard convolution operation; (3) Use low-rank decomposition techniques to compress attention computation, such as decomposing the full-size attention matrix into a product of low-rank matrices; (4) Apply parameter sharing strategies, such as sharing some parameters between different layers.

[0059] Furthermore, quantization techniques (such as 8-bit or 4-bit quantization) can be combined to further reduce model size. In mobile advertising scenarios, this lightweight network can significantly reduce inference latency and memory usage while maintaining performance. For example, it can achieve millisecond-level ad content understanding and matching in mobile news apps, providing a near-real-time personalized recommendation experience.

[0060] Step 3.3: Attention Distillation.

[0061] For the attention matrices of the teacher network and the student network, define the distillation loss function: in, is the knowledge distillation loss function, is the number of attention heads, is the attention matrix of the teacher network, is the attention matrix of the student network, Represents the square of the L2 norm.

[0062] Step 3.4: Adapt distillation temperature.

[0063] Introducing adaptive distillation temperature parameters , adjust the distillation strength according to the importance of different attention heads: in, is the adaptive knowledge distillation loss function, For the The weight coefficient of the attention head, , Represents all attention heads From 1 to Summation. in, For the The weight coefficient of the attention head, Indicates the The importance score of each attention head, is the distillation temperature parameter, represents the exponential function, Indicates the The importance score of each attention head, Indicates the The attention heads range from 1 to Summation.

[0064] Step 4: Cross-modal contrastive learning.

[0065] Introducing cross-modal comparative learning objectives into the student network to enhance semantic alignment between modalities and improve fusion quality. Specifically, this includes:

[0066] Step 4.1: Construction of positive and negative samples.

[0067] For each time step in the user behavior sequence , define different modes of the same interaction term as positive sample pairs , the modalities of different interaction items are represented as negative samples. Positive samples represent users’ interactions with different modalities of the same content at the same time point, while negative samples represent users’ interactions with different modalities at different time points or for different content.

[0068] Step 4.2: Similarity calculation.

[0069] Compute the cosine similarity between modality representations: in, and Represent the feature representations of two different modalities, Represents a vector representation operation.

[0070] Step 4.3: Contrastive loss calculation.

[0071] Calculate contrastive learning loss based on InfoNCE loss function: in, is the contrastive learning loss function, is the modal representation of the anchor negative sample, is the modal representation of the anchor point positive sample, The modal representation includes a positive sample or a negative sample, is the similarity function, is the temperature parameter, represents the logarithmic function, is the sample index, For all samples Summation.

[0072] Step 4.4: Difficult sample mining algorithm.

[0073] Implement a difficult sample mining algorithm. For each anchor sample, select the most similar negative sample as the difficult sample to enhance the effect of contrastive learning: in, It is difficult to compare the negative sample loss function. Indicates the negative sample for the anchor point The sample set, To make negative sample indexing difficult, Represents negative samples for anchor points The sum of all hard negative samples.

[0074] Step 5: Contextual condition fusion and recommendation.

[0075] By introducing contextual condition embedding technology, the user's current situation information is injected into the fusion process as a conditional variable to achieve context-aware advertising recommendations. Specifically, it includes:

[0076] Step 5.1: Context feature extraction.

[0077] Collect and process the user's current context information, including time, location, device type, network status, etc., to generate a context feature vector C.

[0078] Step 5.2: Conditional weight generation.

[0079] Based on contextual features, conditional weights for modal fusion are generated: in, is the modal weight under contextual conditions, Generate a network for parameterized conditional weights, are network parameters, is the context feature vector, For the A modal representation, For the A modal representation, is the number of modes, Indicates the Mode 1 to Summation.

[0080] Step 5.3: Modal feature fusion.

[0081] According to the conditional weights, the features of each modality are integrated: in, To fusion features, is the intra-modal feature processing function, A diagram showing the relationship between elements within a modal.

[0082] Step 5.4: Ad candidate generation and ranking.

[0083] Based on the fused features, calculate the user's interest score for the ad candidate set: in, For users to advertise Interest score, is the characteristic representation of the advertisement, Calculates the interest score function. Sorts the ad candidates based on their interest scores and selects the one with the highest score. Advertisements are recommended.

[0084] This implementation achieves efficient and high-precision user advertising recommendations through multimodal fusion and knowledge distillation technology, with the following technical effects:

[0085] Improved recommendation accuracy: Through multimodal feature extraction and intra-modal relationship modeling, this approach effectively captures the multi-dimensional representation of user interests. By integrating contextual conditions, it enables context-aware personalized recommendations, significantly improving ad matching accuracy. In online A / B testing, compared to traditional recommendation methods, this approach achieved a 23.7% increase in click-through rate and an 18.5% increase in conversion rate.

[0086] Computational efficiency optimization: Multi-level attention distillation technology is used to transfer knowledge from a large teacher model to a lightweight student model, enabling efficient model deployment. Compared to the original complex model, the student model achieves a 5.2x increase in inference speed and a 76.3% reduction in memory usage, with no more than a 3.1% performance loss, effectively addressing resource constraints on mobile devices.

[0087] Improved modal fusion quality: Through cross-modal comparative learning and hard-to-sample mining, this method enhances semantic alignment between modalities and addresses inter-modal information redundancy and noise interference. Experiments show that modal fusion quality improves by 31.4% compared to traditional fusion methods, particularly in scenarios with incomplete multimodal information, achieving a 42.6% improvement in robustness.

[0088] Enhanced device adaptability: By optimizing the model architecture and computational process, this implementation is adaptable to devices with varying computing capabilities. Running on minimally configured devices increases latency by only 15%, successfully covering over 92% of user devices and achieving broad application coverage.

[0089] Expanded Application Scenarios: This implementation is particularly well-suited for rich media advertising scenarios such as news feeds and short videos. In testing of these applications, user engagement rates increased by 29.3% and average dwell time increased by 18.7%, significantly boosting advertisers' return on investment.

[0090] This implementation is applied to a mobile application on an e-commerce platform that provides product display, short video content, and graphic information flow services. The system needs to recommend personalized ads in real time while users browse, while meeting the requirements of ad relevance and resource constraints. This scenario has the following characteristics: User behavior data includes multiple modalities, such as text search history, product image browsing, and short video viewing; User interests change dynamically, such as switching from browsing winter clothing to browsing home goods; Mobile devices have limited computing resources and need to meet real-time requirements while ensuring recommendation quality. The performance of devices used by different users varies greatly, ranging from high-end flagship phones to entry-level devices.

[0091] 4.2 User Behavior Data Example: The user behavior sequence data is shown in Table 1, which contains the user's multimodal interaction records in the past 7 days:

[0092] Table 1: Example of user behavior sequence data.

[0093] 4.3.1 Multimodal feature extraction example.

[0094] The user behavior data in Table 1 is subjected to feature extraction and combined with intra-modal relationship modeling. The processing results are shown in Table 2:

[0095] Table 2: Multimodal feature extraction and relationship modeling results.

[0096] In actual implementation, after processing through the subgraph attention network, it is possible to capture the migration trend of users' interests from thin and light notebooks to mechanical keyboards, and find that users are currently mainly interested in computer accessories.

[0097] 4.3.2 Knowledge distillation and modality fusion examples.

[0098] This system uses a CLIP-based multimodal model as the teacher network, constructs a lightweight student network through knowledge distillation, and implements multimodal fusion recommendation. Table 3 shows the key results of the entire process:

[0099] Table 3: Examples of knowledge distillation and recommendation results.

[0100] As shown in Table 3, knowledge distillation significantly improves the accuracy of the lightweight student network while maintaining low computational requirements. The recommendation results based on the fused features also accurately reflect the changes in user interests.

[0101] 4.4.1 Overall system performance evaluation.

[0102] This method was comprehensively compared with the traditional method in online testing, and the results are shown in Table 4:

[0103] Table 4: Comprehensive evaluation of system performance.

[0104] As can be seen from Table 4, this method has achieved significant improvements in the four dimensions of recommendation accuracy, user experience, computational efficiency, and modality fusion quality.

[0105] 4.4.2 Adaptability test for different devices.

[0106] To verify the adaptability of this method on different devices, we conducted inference performance tests on a variety of typical mobile devices. The results are shown in Table 5:

[0107] Table 5: Performance comparison of different devices.

[0108] As can be seen from Table 5, this method achieves significant performance improvements on various devices, especially on entry-level and older devices. The reduction in inference time makes real-time recommendations possible, greatly improving the user experience and expanding the system's device coverage.

[0109] 4.4.3 Multimodal fusion quality comparison experiment.

[0110] The multimodal fusion quality of this method is compared with three existing methods. The test includes complete information scenarios and incomplete information scenarios (simulating the lack of data in one modality). The results are shown in Table 6:

[0111] Table 6: Multimodal fusion quality comparison experiment. --

[0112] As can be seen from Table 6, this method achieves the best performance in both complete information scenarios and various modality missing scenarios. In particular, it shows stronger robustness when processing incomplete information, with an average performance drop of only 7.8%, which is much lower than other methods.

[0113] The above example verification results show that this implementation successfully solves the problem of balancing accuracy and efficiency in multimodal advertising recommendations, achieving the dual goals of high-quality recommendations and efficient deployment on mobile devices.

[0114] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Ordinary technicians in this field can also make more forms of equivalent embodiments based on the inspiration of this embodiment, all of which are protected by this embodiment.

Claims

1. A user advertisement recommendation method based on a fusion neural network, characterized in that: The following steps are involved: Extract features from different modal data in user behavior sequences to obtain intra-modal representations; Based on the intra-modal representation, we construct an intra-modal element relationship graph to capture the dependencies between elements, and introduce a multi-scale temporal convolutional structure to simultaneously capture short-term, medium-term, and long-term behavioral sequence patterns. Based on the intra-modal element relationship graph, a large pre-trained multimodal model is used as the teacher network to extract the inter-modal attention mapping relationship, build a lightweight student network, and learn the attention distribution of the teacher network through knowledge distillation; Based on a lightweight student network, we introduce a cross-modal contrastive learning objective into the student network to enhance semantic alignment between modalities. Based on the teacher network and the lightweight student network, the contextual condition embedding technology is introduced to inject the user's current situation information as a conditional variable into the fusion process, and the advertising recommendation results are generated based on the fusion features; The relationship graph determines the weight of the edge by calculating the cosine similarity between elements, and the subgraph attention network processes the relationship graph to calculate the node representation.

2. The user advertisement recommendation method based on fusion neural network according to claim 1, characterized in that: The multimodal feature extraction steps include applying a pre-trained language model to process user text information to generate a text feature vector; applying a convolutional neural network to process the image content of user interaction to generate an image feature vector; combining a spatiotemporal convolutional network and frame sampling technology to process the video content watched by the user to generate a video feature vector; and recording the time information of user behavior to extract a time series feature vector.

3. The user advertisement recommendation method based on fusion neural network according to claim 1, characterized in that: The node representation calculation formula of the subgraph attention network is: ;in, Indicates the Nodes in the layer The representation vector of Indicates the Nodes in the layer The representation vector of Representation node The neighbor set of is the learnable weight matrix, is the activation function, is the attention coefficient, Indicates the node No. The sum of neighboring nodes.

4. The user advertisement recommendation method based on fusion neural network according to claim 3, characterized in that: The calculation formula of the attention coefficient is: ;in, is the attention vector, is the transpose of the attention vector, represents transpose, Represents vector concatenation operation, is the learnable weight matrix for attention calculation, represents the exponential function, is the LeakyReLU activation function, is the sample index, Indicates the node All samples Sum, 、 and Represent the feature vectors of three different nodes respectively.

5. The user advertisement recommendation method based on fusion neural network according to claim 1, characterized in that: The multi-scale temporal convolution structure includes: ;in, 、 and They are short-term, medium-term and long-term behavioral sequence patterns, is the node representation matrix output by the graph attention network, represents a one-dimensional convolution operation, 、 and are the short-term, medium-term, and long-term convolution kernel sizes, 、 and is the corresponding output dimension.

6. The user advertisement recommendation method based on fusion neural network according to claim 1, characterized in that: The loss function of the knowledge distillation is: ;in, is the knowledge distillation loss function, is the number of attention heads, and Teacher and student networks The attention matrix of the attention head, Represents all attention heads From 1 to Sum, is the attention head index, , is the attention matrix of the teacher network, is the attention matrix of the student network, Represents the square of the L2 norm.

7. The user advertisement recommendation method based on fusion neural network according to claim 1, characterized in that: The loss function of the cross-modal contrastive learning is: ;in, is the contrastive learning loss function, is the modal representation of the anchor negative sample, is the modal representation of the anchor point positive sample, The modal representation includes a positive sample and a negative sample, is the similarity function, is the temperature parameter, represents the exponential function, represents the logarithmic function, For all samples Summation.

8. The user advertisement recommendation method based on fusion neural network according to claim 1, characterized in that: The modal fusion function of contextual condition fusion is: ;in, To fusion features, is the modal weight under contextual conditions, is the intra-modal feature processing function, For the A modal representation, is the relationship diagram of elements within the modal, is the number of modes, is the modal index, , For all modes From 1 to Sum, Represents a multiplication operation.

9. The user advertisement recommendation method based on fusion neural network according to claim 8, characterized in that: The modal weight calculation formula under the context conditions is: ;in, is the modal weight under contextual conditions, Generate a network for parameterized conditional weights, are network parameters, is the context feature vector, For the A modal representation, is the total number of modes, Indicates the Mode 1 to Sum, Represents the exponential function.

10. A user advertisement recommendation system based on a fusion neural network, configured to execute the user advertisement recommendation method based on a fusion neural network according to any one of claims 1 to 9, characterized in that: include: Multimodal feature extraction module, used to extract features from different modal data in user behavior sequences; Intra-modal relationship modeling module, which is used to construct the intra-modal element relationship graph and process the graph through sub-graph attention network and multi-scale temporal convolution; Knowledge distillation module, which is used to transfer attention knowledge through teacher network and student network; A cross-modal contrastive learning module to enhance semantic alignment between modalities; Contextual condition fusion module, used to integrate user context information into modal features; The advertising recommendation module is used to generate advertising recommendation results based on fused features. The system uses knowledge distillation technology to compress complex models into lightweight models, enabling efficient deployment on mobile devices.

Images