User advertisement recommendation method and system based on fusion neural network

Abstract

The application relates to the technical field of computers and discloses a user advertisement recommendation method and system based on a fusion neural network; the method comprises the following steps: performing feature extraction on different modal data in a user behavior sequence; constructing an element relationship graph within a mode and applying a subgraph attention network to capture the dependency relationship between elements; adopting a large pre-training multi-modal model as a teacher network, constructing a lightweight student network through knowledge distillation; introducing cross-modal contrast learning to enhance semantic alignment between modes; and using context conditional embedding technology to integrate user context information into a recommendation process. The application realizes high-precision and high-efficiency advertisement recommendation and is particularly suitable for mobile terminal resource limited scenes.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a user advertising recommendation method and system based on fused neural networks. Background Technology

[0002] With the rapid development of the internet and mobile technologies, multimodal information (text, images, videos, etc.) has become a major component of users' online behavior. In advertising recommendation scenarios, accurately understanding and integrating this multimodal information is crucial for improving recommendation quality. Especially in rich media environments such as news feeds and short videos, effectively mining user interests and preferences has become a major challenge for advertising recommendation systems.

[0003] Existing advertising recommendation technologies mainly suffer from the following problems: limited ability to process multimodal data, difficulty in fully exploring the correlation between elements within a modality, high computational complexity of large multimodal models, difficulty in efficient deployment in resource-constrained environments, and insufficient robustness to modality loss or noise interference, resulting in unstable performance.

[0004] To address the aforementioned technical challenges, there is an urgent need for a user advertising recommendation method and system that can effectively integrate multimodal information and is suitable for efficient deployment in resource-constrained environments. Summary of the Invention

[0005] This invention provides a user advertising recommendation method and system based on fusion neural networks, which solves the technical problems in the prior art such as limited multimodal fusion capabilities, difficulty in deploying complex models in resource-constrained environments, and poor robustness due to modal missing features.

[0006] This invention provides a user advertising recommendation method and system based on a fusion neural network, including:

[0007] The first aspect is a user ad recommendation method based on fusion neural networks, which includes the following steps:

[0008] Feature extraction is performed on different modal data in user behavior sequences to obtain intramodal representations;

[0009] Based on intramodal representation, an intramodal element relationship graph is constructed to capture the dependencies between elements, and a multi-scale temporal convolutional structure is introduced to capture short-term, medium-term and long-term behavioral sequence patterns simultaneously.

[0010] Based on the intramodal element relationship graph, a large-scale pre-trained multimodal model is used as the teacher network to extract the intermodal attention mapping relationship and construct a lightweight student network. The attention distribution of the teacher network is learned through knowledge distillation.

[0011] Based on a lightweight student network, a cross-modal contrastive learning objective is introduced into the student network to enhance semantic alignment between modalities;

[0012] By introducing contextual conditional embedding technology based on teacher networks and lightweight student networks, user current context information is injected as a conditional variable into the fusion process, and advertising recommendation results are generated based on the fusion features.

[0013] The relation graph determines the edge weights by calculating the cosine similarity between elements, and the subgraph attention network processes the relation graph to calculate node representations.

[0014] Furthermore, the multimodal feature extraction steps include applying a pre-trained language model to process user text information and generate text feature vectors; applying a convolutional neural network to process image content of user interaction and generate image feature vectors; combining spatiotemporal convolutional networks and frame sampling techniques to process video content viewed by the user and generate video feature vectors; and recording the time information of user behavior to extract temporal feature vectors.

[0015] Furthermore, the node representation calculation formula for the subgraph attention network is as follows: ;in, Indicates the first Layer nodes The representation vector, Indicates the first Layer nodes The representation vector, Represents a node The neighborhood group, The weight matrix is ​​a learnable matrix. For activation function, Attention coefficient Indicates a node The Sum of neighboring nodes.

[0016] Furthermore, the formula for calculating the attention coefficient is as follows: ;in, For attention vectors, This is the transpose of the attention vector. Indicates transpose. This represents a vector concatenation operation. The learnable weight matrix is ​​computed for attention. Represents an exponential function. The LeakyReLU activation function is used. For sample index, Indicates a node All samples Summation, , and These represent the feature vectors of three different nodes.

[0017] Furthermore, the multi-scale temporal convolutional structure includes: ;in, , and These represent short-term, medium-term, and long-term behavioral sequence patterns. The node representation matrix output by the graph attention network. This represents a one-dimensional convolution operation. , and These represent the kernel sizes for short-term, medium-term, and long-term convolutions, respectively. , and This corresponds to the output dimension.

[0018] Furthermore, the loss function for knowledge distillation is: ;in, For the knowledge distillation loss function, For the number of attention heads, and The network for teachers and students is divided into two parts. Attention matrix of attention heads, Indicates attention to all heads From 1 to Summation, For attention head index, , For the attention matrix of the teacher network, For the attention matrix of the student network, This represents the square of the L2 norm.

[0019] Furthermore, the loss function for the cross-modal contrastive learning is: ;in, To compare the learning loss function, Modal representation for anchor negative samples, Modal representation for anchor point positive samples. The modal representation includes one positive sample and one negative sample. For similarity function, For temperature parameters, Represents an exponential function. Represents the logarithmic function. This indicates that for all samples Sum.

[0020] Furthermore, the modal fusion function for contextual conditional fusion is: ;in, As a feature of fusion, Modal weights under context conditions, This is the intramodal feature processing function. For the first The representation of each modality, This is a graph showing the relationships between elements within a mode. The number of modes, For modal indexing, , Indicates all modes From 1 to Summation, It represents the multiplication operation.

[0021] Furthermore, the formula for calculating the modal weights under the aforementioned context conditions is as follows: ;in, Modal weights under context conditions, For parameterized conditional weight generation networks, For network parameters, For context feature vectors, For the first The representation of each modality, The total number of modes, Indicates the first Mode 1 to Summation, This represents an exponential function.

[0022] Secondly, a user advertising recommendation system based on fusion neural networks includes:

[0023] The multimodal feature extraction module is used to extract features from different modalities of user behavior sequences;

[0024] The intramodal relation modeling module is used to construct an intramodal element relation graph and process the graph through a subgraph attention network and multi-scale temporal convolution.

[0025] The knowledge distillation module is used to achieve the transfer of attentional knowledge through teacher and student networks; the cross-modal contrastive learning module is used to enhance semantic alignment between modalities.

[0026] The contextual fusion module is used to integrate user contextual information into modal features;

[0027] The advertising recommendation module is used to generate advertising recommendation results based on fusion features. The system uses knowledge distillation technology to compress complex models into lightweight models, enabling efficient deployment on mobile devices.

[0028] The beneficial effects of this invention are as follows: This invention extracts features from different modal data in user behavior sequences to obtain intra-modal representations; constructs an intra-modal element relationship graph to capture inter-element dependencies, and introduces a multi-scale temporal convolutional structure to simultaneously capture short-term, medium-term, and long-term behavior sequence patterns; uses a large-scale pre-trained multimodal model as the teacher network to extract inter-modal attention mapping relationships, constructs a lightweight student network, and learns the attention distribution of the teacher network through knowledge distillation; introduces a cross-modal contrastive learning objective in the student network to enhance inter-modal semantic alignment; introduces contextual conditional embedding technology to inject user current context information as a conditional variable into the fusion process, and generates advertising recommendation results based on the fusion features; wherein, the relationship graph determines the edge weights by calculating the cosine similarity between elements, and the subgraph attention network processes the relationship graph to calculate node representations; by introducing multimodal feature extraction, intra-modal relationship modeling, multi-level attention distillation, cross-modal contrastive learning, and contextual conditional fusion, this invention solves the problems of limited multimodal data processing capabilities, difficulty in efficiently deploying complex models in resource-constrained environments, and insufficient robustness under modal missing or noise interference in existing technologies. By combining these technologies, recommendation accuracy and computational efficiency are significantly improved, modality fusion quality is enhanced, and device adaptability is increased, enabling efficient ad recommendations on multiple devices, especially suitable for mobile scenarios with limited resources. Attached Figure Description

[0029] Figure 1 This is a flowchart of the main steps of the user advertisement recommendation method based on fusion neural networks of the present invention;

[0030] Figure 2 This is a flowchart of the multimodal feature extraction steps of the present invention;

[0031] Figure 3 This is a flowchart of the modal intramodal relation modeling steps of the present invention;

[0032] Figure 4 This is a flowchart of the multi-stage attention distillation steps of the present invention;

[0033] Figure 5 This is a flowchart of the cross-modal contrastive learning steps of the present invention;

[0034] Figure 6 This is a flowchart of the contextual condition fusion and recommendation steps of the present invention. Detailed Implementation

[0035] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0036] At least one embodiment of the present invention discloses a user advertising recommendation method and system based on a fusion neural network, including:

[0037] Implementation Method 1

[0038] This implementation method is applicable to advertising recommendation scenarios involving multimodal information fusion, especially in rich media environments such as news feeds and short videos. In this scenario, the following technical requirements exist: on the one hand, it is necessary to effectively integrate multimodal information (text, images, videos, etc.) from user behavior sequences to achieve high-precision user interest modeling; on the other hand, it is necessary to address the problem of limited computing resources on mobile terminals to ensure that the recommendation system can operate efficiently in resource-constrained environments.

[0039] Traditional recommendation technologies have the following shortcomings: First, they have limited ability to fuse and process multimodal data, making it difficult to fully explore the relationships between elements within a modality; second, complex models are difficult to deploy on mobile terminals with limited computing resources; and third, they cannot effectively utilize the advantages of large pre-trained models in multimodal understanding.

[0040] To address the aforementioned issues, this implementation proposes a user ad recommendation method based on fusion neural networks. Through techniques such as multimodal feature extraction, intramodal relation modeling, multi-level attention distillation, cross-modal contrastive learning, and contextual condition fusion, it achieves high-precision and high-efficiency ad recommendation.

[0041] like Figures 1 to 6 As shown, the user advertisement recommendation method based on a fusion neural network in this embodiment includes the following steps:

[0042] Step 1: Multimodal feature extraction.

[0043] A modality-specific feature extractor is used to process multimodal data in user behavior sequences, generating high-quality intramodal representations. Specifically, this includes:

[0044] Step 1.1: Text feature extraction.

[0045] A pre-trained language model is applied to process textual information from users' historical interactions (such as search keywords, browsing titles, and comment content) to generate text feature vectors T. Specifically, an encoder-based BERT model or its lightweight variant can be used. This model contains multiple bidirectional encoders, each consisting of a multi-head self-attention layer and a feedforward neural network layer. For mobile scenarios, compressed model variants such as DistilBERT or TinyBERT, with fewer parameters, can be used to reduce computational resource requirements while maintaining text understanding capabilities. In e-commerce advertising scenarios, this model can effectively capture the semantic matching relationship between user search terms and product titles, such as associating semantically similar descriptions of thin and light laptops with ultra-thin portable computers.

[0046] Step 1.2: Image feature extraction.

[0047] A convolutional neural network (CNN) is applied to the image content that the user has interacted with to extract the image feature vector I. In specific implementations, CNN architectures such as ResNet or MobileNet can be used. These networks consist of multiple convolutional layers, pooling layers, and residual connections. For mobile scenarios, the lightweight MobileNetV3 network is preferred. This network uses depthwise separable convolutions and squeeze-and-excitation modules, significantly reducing the number of parameters and computational cost. In practical applications, the network can be pre-trained on a large-scale image dataset (such as ImageNet) and then fine-tuned for the features of the advertising images. For example, in fashion product advertising scenarios, this model can effectively identify and extract visual features such as clothing style, color, and texture, enabling the system to recommend similar or complementary products based on the user's historical browsing history.

[0048] Step 1.3: Video feature extraction.

[0049] This method combines spatiotemporal convolutional networks and frame sampling techniques to extract features from video content viewed by users, generating a video feature vector V. Specifically, 3D convolutional neural networks (such as C3D or I3D) or SlowFast network architectures can be used, as these networks can simultaneously capture both spatial and temporal information from the video. In scenarios with limited computational resources, a video feature extraction method based on 2D convolutional networks can be employed. This involves first uniformly sampling the video to obtain keyframes, then using a 2D convolutional neural network to extract features from each keyframe, and finally merging the frame-level features using a temporal aggregation function (such as average pooling or attention weighting) to obtain a video-level representation. In short video advertising scenarios, this method can effectively capture key visual elements and dynamic changes in the video, such as identifying the video styles preferred by users (e.g., fast-paced sports content or lighthearted comedy content), thereby recommending video ads that better match user interests.

[0050] Step 1.4: Extraction of behavioral temporal features.

[0051] Record the time information of user behavior and extract the time-series feature vector B.

[0052] Step 2: Modeling intramodal relationships.

[0053] Construct an intra-modal element relationship graph, apply a subgraph attention network to capture the dependencies between intra-modal elements, and generate an enhanced intra-modal representation. Specifically, this includes:

[0054] Step 2.1: Relationship graph construction.

[0055] For each mode Constructing a relationship graph based on the semantic similarity between elements ,in, It is a set of nodes (corresponding to the elements in the sequence). Let this be a set of edges (representing the relationships between elements). The weights of the edges are determined by calculating the cosine similarity between the elements: in, Cosine similarity between elements and The first The and the first The feature vector of each element.

[0056] Step 2.2: Subgraph Attention Network.

[0057] The constructed relational graph is processed using a subgraph attention network, and node representations are calculated: in, Indicates the first Layer nodes The representation vector, Indicates the first Layer nodes The representation vector, Represents a node The neighborhood group, The weight matrix is ​​a learnable matrix. For activation function, Attention coefficient Indicates a node The Sum of neighboring nodes.

[0058] In its implementation, the network employs a Graph Attention Network (GAT) structure, comprising multiple graph attention layers. Each layer first calculates the attention weights between node pairs, then aggregates information about neighboring nodes based on these weights. To improve efficiency, a sparse attention computation method can be used, considering only the actual edges in the graph and avoiding fully connected computations. In the system implementation, typically 2-3 graph attention layers are stacked, each followed by a batch normalization layer and residual connections to accelerate training and improve stability. In practical advertising recommendation applications, this network can effectively model the interest associations of users towards different types of content. For example, in news and information advertising scenarios, by capturing the relationships between news topics read by users, it can identify the user's interest migration path (such as migrating from technology news to smart home ads), thereby achieving more accurate interest capture.

[0059] Step 2.3: Multi-scale temporal convolution.

[0060] By employing multi-scale temporal convolutional structures, we can simultaneously capture short-term, medium-term, and long-term behavioral sequence patterns. ;in, , and These represent short-term, medium-term, and long-term behavioral sequence patterns. The node representation matrix output by the graph attention network. This represents a one-dimensional convolution operation. , and These represent the kernel sizes for short-term, medium-term, and long-term convolutions, respectively. , and This corresponds to the output dimension.

[0061] Step 2.4: Multi-scale feature fusion.

[0062] By fusing temporal features at different scales, the final intra-modal representation is obtained: in, This is the final intra-modal temporal feature representation. This indicates a feature concatenation operation. and These are the learnable weight matrix and bias vector, respectively.

[0063] Step 3: Multi-level attention distillation.

[0064] Using a large-scale pre-trained multimodal model as the teacher network, high-quality intermodal attention mapping relationships are extracted. A lightweight student network is constructed through knowledge distillation to achieve efficient multimodal fusion. Specifically, this includes:

[0065] Step 3.1: Teacher network construction and training.

[0066] A teacher network is constructed using a large-scale pre-trained multimodal model, which possesses rich cross-modal understanding capabilities. The teacher network receives multimodal input and calculates the inter-modal attention matrix using a multi-head attention method. in, For Teachers' Network Attention matrix of attention heads, For querying the matrix, The key matrix, Indicates transpose. Let be the dimension of the key vector. Used to normalize attention scores.

[0067] In practical implementation, the teacher network can be built based on pre-trained multimodal models such as CLIP, ViLBERT, or LXMERT. These models typically consist of two parts: a modality-specific encoder and a cross-modal fusion encoder. The modality-specific encoder is responsible for processing unimodal data, such as a text encoder (based on Transformer) and a visual encoder (based on VisionTransformer or ResNet); the cross-modal fusion encoder uses an attention layer to achieve the interaction and integration of information from different modalities. In advertising systems, these models can be fine-tuned on advertising-related multimodal datasets to adapt to specific application scenarios. For example, in e-commerce live streaming advertising scenarios, the teacher network can effectively understand the semantic relationship between the anchor's explanation (text) and the displayed products (images), thereby accurately capturing users' interests in the products.

[0068] Step 3.2: Student Network Construction.

[0069] A lightweight student network is constructed using a parameter-efficient architecture, such as depthwise separable convolution and low-rank factorization, to reduce computational complexity. The student network also computes the multi-head attention matrix. in, For student network Attention matrix of attention heads, and These are the query and key matrices of the student network, respectively. Let be the dimension of the student network key vector.

[0070] In practical implementation, the student network can adopt a simplified multimodal Transformer architecture, which includes the following key optimizations:

[0071] (1) Reduce the number of Transformer layers, such as from 12 layers to 4-6 layers;

[0072] (2) Use depthwise separable convolution instead of standard convolution operations;

[0073] (3) Use low-rank decomposition techniques to compress attention computation, such as decomposing the full-size attention matrix into a product of low-rank matrices;

[0074] (4) Apply parameter sharing strategies, such as sharing some parameters between different layers.

[0075] Furthermore, quantization techniques (such as 8-bit or 4-bit quantization) can be combined to further reduce the model size. In mobile advertising scenarios, this lightweight network can significantly reduce inference latency and memory usage while maintaining performance. For example, it can achieve millisecond-level ad content understanding and matching in mobile news apps, providing a near real-time personalized recommendation experience.

[0076] Step 3.3: Attention distillation.

[0077] For the attention matrices of the teacher network and the student network, the distillation loss function is defined as follows: in, For the knowledge distillation loss function, For the number of attention heads, For the attention matrix of the teacher network, For the attention matrix of the student network, This represents the square of the L2 norm.

[0078] Step 3.4: Adaptive distillation temperature.

[0079] Introducing adaptive distillation temperature parameters Adjust the distillation intensity according to the importance of different attention points: in, For the adaptive knowledge distillation loss function, For the first The weighting coefficient of each attention head. , Indicates attention to all heads From 1 to Sum. in, For the first The weighting coefficient of each attention head. Indicates the first The importance score of each attention head For distillation temperature parameters, Represents an exponential function. Indicates the first The importance score of each attention head Indicates the first Each attention point from 1 to Sum.

[0080] Step 4: Cross-modal comparative learning.

[0081] Introducing cross-modal contrastive learning objectives into student networks enhances inter-modal semantic alignment and improves fusion quality. Specifically, this includes:

[0082] Step 4.1: Constructing positive and negative samples.

[0083] For each time step in the user behavior sequence Different modalities of the same interaction item are defined as positive sample pairs. Different modal representations of interaction items are called negative samples. Positive samples represent different modal interactions of users with the same content at the same time, while negative samples represent modal interactions of users at different time points or with different content.

[0084] Step 4.2: Similarity calculation.

[0085] Calculate the cosine similarity between modal representations: in, and These represent the feature representations of two different modalities. Vector representation operations.

[0086] Step 4.3: Comparison loss calculation.

[0087] Calculate the contrastive learning loss based on the InfoNCE loss function: in, To compare the learning loss function, Modal representation for anchor negative samples, Modal representation for anchor point positive samples. This includes a modal representation of either a positive or a negative sample. For similarity function, For temperature parameters, Represents the logarithmic function. For sample index, This indicates that for all samples Sum.

[0088] Step 4.4: Difficult sample mining algorithm.

[0089] To implement a hard sample mining algorithm, for each anchor sample, the most similar negative sample is selected as the hard sample to enhance the effect of contrastive learning. in, For the contrastive learning loss function of difficult negative samples, This indicates that for anchor point negative samples The sample set, For the difficult negative sample index, Indicates negative samples at anchor points Summing all the hard-to-bear samples.

[0090] Step 5: Contextual condition fusion and recommendation.

[0091] Contextual condition embedding technology is introduced, injecting the user's current context information as a condition variable into the fusion process to achieve context-aware ad recommendation. Specifically, this includes:

[0092] Step 5.1: Contextual feature extraction.

[0093] Collect and process the user's current context information, including time, location, device type, network status, etc., and generate a context feature vector C.

[0094] Step 5.2: Conditional weight generation.

[0095] Based on contextual features, conditional weights for modality fusion are generated: in, Modal weights under context conditions, For parameterized conditional weight generation networks, For network parameters, For context feature vectors, For the first The representation of each modality, For the first The representation of each modality, The number of modes, Indicates the first Mode 1 to Sum.

[0096] Step 5.3: Modal feature fusion.

[0097] Based on conditional weights, features from each modality are fused: in, As a feature of fusion, This is the intramodal feature processing function. This is a graph showing the relationships between elements within a mode.

[0098] Step 5.4: Ad candidate generation and sorting.

[0099] Based on the fusion features, calculate the user's interest score for the ad candidate set: in, For users to advertise Interest score, This is a characteristic representation of advertising. A function is used to calculate the interest score. Ad candidates are then sorted based on their interest scores, and the highest-scoring candidate is selected. Recommend an advertisement.

[0100] This implementation method achieves efficient and high-precision user ad recommendation through multimodal fusion and knowledge distillation techniques, and has the following technical effects:

[0101] Improved Recommendation Accuracy: By extracting multimodal features and modeling intramodal relationships, multi-dimensional representations of user interests are effectively captured; context-aware personalized recommendations are achieved through context fusion, significantly improving ad matching accuracy. In online A / B testing, compared to traditional recommendation methods, click-through rate increased by 23.7% and conversion rate increased by 18.5%.

[0102] Computational efficiency optimization: By employing multi-level attention distillation technology, knowledge from the large teacher model is transferred to the lightweight student model, achieving efficient model deployment. Compared to the original complex model, the student model achieves a 5.2x increase in inference speed and a 76.3% reduction in memory usage, with a performance loss of no more than 3.1%, effectively addressing the resource constraints of mobile devices.

[0103] Modality fusion quality improvement: Through cross-modal contrastive learning and hard sample mining, the semantic alignment between modalities is enhanced, solving the problems of information redundancy and noise interference between modalities. Experiments show that the modality fusion quality is improved by 31.4% compared with traditional fusion methods, especially with a 42.6% improvement in robustness in scenarios with incomplete multimodal information.

[0104] Enhanced Device Adaptability: By optimizing the model architecture and computational process, this implementation can adapt to terminal devices with varying computing capabilities. Latency increases by only 15% on the lowest-configuration devices, successfully covering over 92% of user devices and achieving broad technology application coverage.

[0105] Application Scenarios Expanded: This implementation method is particularly suitable for rich media advertising scenarios such as news feeds and short videos. In application tests in these scenarios, user interaction rate increased by 29.3%, and average dwell time increased by 18.7%, resulting in a significant increase in return on investment for advertisers.

[0106] This implementation method is applied to the mobile application of an e-commerce platform, which simultaneously provides product display, short video content, and image and text information flow services. The system needs to recommend personalized advertisements in real time during the user's browsing process, while meeting the requirements of advertisement relevance and resource constraints. This scenario has the following characteristics:

[0107] User behavior data includes multiple modalities, such as text search history, product image browsing, and short video viewing;

[0108] User interests change dynamically, such as shifting from browsing winter clothing to browsing home furnishings;

[0109] Mobile devices have limited computing resources, so it is necessary to meet real-time requirements while ensuring the quality of recommendations.

[0110] The performance of devices used by different users varies greatly, ranging from high-end flagship phones to entry-level devices.

[0111] 4.2 Example of User Behavior Data: The user behavior sequence data is shown in Table 1, which contains the user's multimodal interaction records over the past 7 days:

[0112] Table 1: Example of user behavior sequence data.

[0113]

[0114] 4.3.1 Example of multimodal feature extraction.

[0115] Feature extraction was performed on the user behavior data in Table 1, and in-modal relation modeling was combined with the results, as shown in Table 2:

[0116] Table 2: Results of multimodal feature extraction and relation modeling.

[0117]

[0118] In practice, after processing with a subgraph attention network, it is possible to capture the trend of user interest shifting from thin and light laptops to mechanical keyboards, and discover that users are currently mainly focused on computer accessories.

[0119] 4.3.2 Examples of knowledge distillation and modal fusion.

[0120] This system uses a CLIP-based multimodal model as the teacher network, constructs a lightweight student network through knowledge distillation, and achieves multimodal fusion recommendation. Table 3 shows the key results of the entire process:

[0121] Table 3: Examples of knowledge distillation and recommendation results.

[0122]

[0123] As shown in Table 3, knowledge distillation significantly improves the accuracy of the lightweight student network while maintaining low computational requirements. The recommendation results based on fusion features also accurately reflect changes in user interests.

[0124] 4.4.1 Overall system performance evaluation.

[0125] This method was comprehensively compared with traditional methods in online testing, and the results are shown in Table 4:

[0126] Table 4: Comprehensive evaluation of system performance.

[0127]

[0128] As shown in Table 4, this method has achieved significant improvements in all four dimensions: recommendation accuracy, user experience, computational efficiency, and modality fusion quality.

[0129] 4.4.2 Adaptability test for different equipment.

[0130] To verify the adaptability of this method to different devices, inference performance tests were conducted on various typical mobile devices, and the results are shown in Table 5:

[0131] Table 5: Performance comparison of different devices.

[0132]

[0133] As shown in Table 5, this method achieves significant performance improvements on various devices. In particular, the reduction in inference time on entry-level and older devices makes real-time recommendations possible, greatly improving the user experience and expanding the device coverage of the system.

[0134] 4.4.3 Multimodal fusion quality comparison experiment.

[0135] The multimodal fusion quality of our proposed method was compared with three existing methods. The tests included scenarios with complete information and scenarios with incomplete information (simulating missing data for a certain modality). The results are shown in Table 6.

[0136] Table 6: Comparison of multimodal fusion quality.

[0137]

[0138] As shown in Table 6, our method achieves the best performance in both scenarios with complete information and scenarios with missing modalities. In particular, it demonstrates stronger robustness when dealing with incomplete information, with an average performance drop of only 7.8%, which is far lower than other methods.

[0139] The above examples demonstrate that this implementation successfully solves the problem of balancing accuracy and efficiency in multimodal advertising recommendation, achieving the dual goals of high-quality recommendation and efficient mobile deployment.

[0140] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A user advertisement recommendation method based on a fusion neural network, characterized in that, The method comprises the following steps: Feature extraction is performed on different modal data in the user behavior sequence to obtain intra-modal representation; Based on the intra-modal representation, an intra-modal element relationship graph is constructed to capture the dependency between elements, and a multi-scale time convolution structure is introduced to capture short-term, medium-term and long-term behavior sequence patterns; Based on the intra-modal element relationship graph, a large pre-trained multi-modal model is used as a teacher network to extract inter-modal attention mapping relationships, construct a lightweight student network, and learn the attention distribution of the teacher network through knowledge distillation; Based on the lightweight student network, a cross-modal contrast learning goal is introduced into the student network to enhance the semantic alignment between modalities; Based on the teacher network and the lightweight student network, a context conditional embedding technology is introduced to inject the current context information of the user into the fusion process as a conditional variable, and generate an advertising recommendation result based on the fusion features; The relationship graph determines the edge weight by calculating the cosine similarity between elements, and the subgraph attention network processes the relationship graph to calculate the node representation. 2.The user advertisement recommendation method based on the fusion neural network according to claim 1, characterized in that, The multi-modal feature extraction step includes applying a pre-trained language model to process user text information to generate a text feature vector; applying a convolutional neural network to process image content of user interaction to generate an image feature vector; combining a spatio-temporal convolution network and a frame sampling technique to process video content watched by the user to generate a video feature vector; and recording time information of user behavior to extract a time sequence feature vector. 3.The user advertisement recommendation method based on the fusion neural network according to claim 1, characterized in that, The node representation calculation formula of the subgraph attention network is: ; wherein, represents the representation vector of the node in the i-th layer, represents the representation vector of the node in the i-th layer, represents the representation vector of the node in the i-th layer, represents the neighbor set of the node , is a learnable weight matrix, is an activation function, is an attention coefficient, represents the summation of the i-th neighbor node of the node . ​ 4.The user advertisement recommendation method based on the fusion neural network according to claim 3, characterized in that, The calculation formula of the attention coefficient is: ; where, is an attention vector, is the transpose of an attention vector, denotes the transpose, denotes the vector concatenation operation, is a learnable weight matrix for attention computation, denotes the exponential function, is a LeakyReLU activation function, is a sample index, denotes the summation over all samples of a node , , and denote the feature vectors of three different nodes, respectively. 5.The user advertisement recommendation method based on the fusion neural network according to claim 1, characterized in that, The multi-scale time sequence convolution structure comprises: ; wherein, , and are short-term, medium-term and long-term behavior sequence patterns, respectively, is a node representation matrix output by the graph attention network, denotes a one-dimensional convolution operation, , and are short-term, medium-term and long-term kernel sizes, respectively, , and are corresponding output dimensions. 6.The user advertisement recommendation method based on the fusion neural network according to claim 1, wherein, The loss function of the knowledge distillation is: ; wherein, is the knowledge distillation loss function, is the number of attention heads, and are the attention matrices of the teacher and student network respectively for the th attention head, denotes the sum over all attention heads from 1 to is the attention head index, , is the attention matrix of the teacher network, is the attention matrix of the student network, denotes the square of the L2 norm.​ 7.The fusion neural network based user advertisement recommendation method of claim 1, wherein, The loss function of the cross-modal contrast learning is: ; wherein, is a contrastive learning loss function, is a modal representation of an anchor negative sample, is a modal representation of an anchor positive sample, is a modal representation comprising one positive sample and one negative sample, is a similarity function, is a temperature parameter, denotes an exponential function, denotes a logarithmic function, denotes a sum over all samples sum. 8.The user advertisement recommendation method based on the fusion neural network according to claim 1, characterized in that, The context condition fusion mode fusion function is: ; wherein, is a fusion feature, is a modality weight under context condition, is a modality-in feature processing function, is a representation of the th modality, is a modality-in element relation graph, is a number of modalities, is a modality index, , denotes a summation over all modalities from 1 to , denotes a multiplication operation. 9.The user advertisement recommendation method based on the fusion neural network according to claim 8, characterized in that, The formula for calculating the modal weights under the given context conditions is as follows: ;in, Modal weights under context conditions, For parameterized conditional weight generation networks, For network parameters, For context feature vectors, For the first The representation of each modality, For the total number of modes, Indicates the first Mode 1 to Summation, This represents an exponential function.

10. A user advertisement recommendation system based on a fusion neural network, configured to perform the user advertisement recommendation method based on a fusion neural network according to any one of claims 1-9. The system comprises: A multi-modal feature extraction module for extracting features from different modal data in the user behavior sequence; An intra-modal relationship modeling module for constructing an intra-modal element relationship graph and processing the graph through a subgraph attention network and a multi-scale time convolution; A knowledge distillation module for transferring attention knowledge through a teacher network and a student network; A cross-modal contrast learning module for enhancing semantic alignment between modalities; A context conditional fusion module for integrating user context information into modal features; An advertising recommendation module for generating an advertising recommendation result based on the fusion features; wherein the system compresses a complex model into a lightweight model through knowledge distillation technology to achieve efficient deployment on a mobile terminal.

Images