A semantic collaborative modeling method and system for multi-modal sequence recommendation

Abstract

The application relates to a semantic collaboration modeling method and system for multi-modal sequence recommendation, and belongs to the technical field of multi-modal sequence recommendation, aiming to solve the problems that existing methods depend on item IDs, are difficult to mine multi-modal semantic collaboration signals, have weak cross-scene migration ability, and have insufficient representation precision. The method comprises the following steps: extracting text information and image information of an item according to user historical interaction behaviors, and extracting multi-level semantic representations through a multi-modal encoder; adopting a multi-head attention module with a one-way mask to capture semantic collaboration signals, and integrating the semantic collaboration signals into initial modal representations; utilizing a hybrid expert structure to perform fine-grained semantic focusing and optimization; obtaining final modal representations of the item through a hybrid expert fusion module; constructing a user multi-modal behavior sequence based on the final modal representations of the item, calculating a preference score after encoding by a sequence recommendation model, and realizing next-step interaction prediction of the user. The application can significantly improve recommendation accuracy, generalization ability and cross-scene knowledge migration efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal sequence recommendation, and in particular to a semantic collaboration modeling method and system for multi-modal sequence recommendation. BACKGROUND

[0002] With the rapid development of information technology, personalized recommendation systems have become a key means to solve the problem of information overload. As an important branch of this field, multi-modal sequence recommendation can more finely model the dynamic evolution of user interest by integrating user historical behavior sequences and multi-modal content of items (such as text, images, etc.), and can improve the accuracy and interpretability of recommendations, showing a wide application prospect in e-commerce, content distribution, and social networks, etc.

[0003] Currently, multi-modal sequence recommendation methods mainly focus on multi-modal feature extraction and fusion. On the one hand, feature extraction-based methods use pre-trained models to obtain semantic representations from multi-modal data such as text and images to enhance the feature expression of items. On the other hand, feature fusion-based methods achieve effective integration of multi-modal information by designing cross-modal interaction mechanisms or introducing learnable fusion weights. Although these methods have improved the recommendation performance to some extent, there are still several key limitations. First, most methods rely on item IDs to capture collaboration signals between user behaviors, failing to fully exploit the semantic associations inherent in multi-modal content, resulting in biased user interest modeling. Second, the strong dependence on item IDs limits the model's ability to transfer knowledge across different recommendation scenarios, especially in cross-domain recommendation and cold start scenarios. Finally, existing fusion mechanisms often lack fine-grained processing of collaboration signals, making it difficult to achieve precise alignment of multi-level semantic information and user preferences, restricting the personalization and generalization performance of recommendations.

[0004] Therefore, there is still an urgent need for a multi-modal sequence recommendation method that can explicitly model fine-grained semantic collaboration signals, reduce dependence on item IDs, and support cross-scenario knowledge transfer, to further improve the accuracy, robustness, and scalability of recommendation systems. SUMMARY

[0005] The present application aims to provide a semantic collaboration modeling method and system for multi-modal sequence recommendation to solve the problems raised in the background art.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] A semantic collaboration modeling method for multi-modal sequence recommendation, comprising:

[0008] According to the user historical interaction behavior, a user historical interaction behavior sequence is constructed; text information and image information of each item in the user historical interaction behavior sequence are extracted and input into a multi-modal encoder for encoding to obtain multi-level semantic representation; the multi-level semantic representation is sorted in time sequence to obtain an initial modal representation sequence; the multi-modal encoder comprises a text encoder and an image encoder;

[0009] An initial modal representation sequence is obtained by sorting the multi-level semantic representation in time sequence; a multi-head attention module with a one-way mask mechanism is used to process the initial modal representation sequence to obtain a semantic coordination signal; the semantic coordination signal is integrated into the initial modal representation sequence; and the semantic coordination signal and the initial modal representation sequence are weighted and fused to obtain a multi-level enhanced modal representation;

[0010] The multi-level enhanced modal representation is input into a hybrid expert structure for processing to obtain a fine-grained semantic representation;

[0011] The fine-grained semantic representation is spliced to obtain a global semantic embedding vector; and the global semantic embedding vector is input into a hybrid expert fusion module for processing to obtain an item final modal representation;

[0012] A user multi-modal behavior sequence is constructed based on the item final modal representation, and the user multi-modal behavior sequence is input into a sequence recommendation model for encoding to obtain a user behavior representation; a preference score is obtained by calculating the inner product of the item final modal representation and the user behavior representation, and the next step interaction of the user is predicted according to the preference score.

[0013] As a preferred scheme of the application, according to the user historical interaction behavior, a user historical interaction behavior sequence is constructed; text information and image information of each item in the user historical interaction behavior sequence are extracted and input into a multi-modal encoder for encoding to obtain multi-level semantic representation; the multi-level semantic representation is sorted in time sequence to obtain an initial modal representation sequence; comprising:

[0014] A user historical interaction behavior sequence is constructed according to the time sequence of the user historical interaction behavior;

[0015] The user historical interaction behavior sequence is composed of I items, and each item includes text information and image information;

[0016] The text information is input into a text encoder, and the image information is input into an image encoder; and the hidden states of multiple intermediate layers and the final output layer of each encoder are extracted simultaneously to obtain the multi-level semantic representation of the item;

[0017] Obtain multi-level semantic representations of all items in the user's historical interaction sequence, and arrange all multi-level semantic representations in chronological order to obtain an initial modal representation sequence; wherein, the initial modal representation sequence includes a text modal representation sequence and an image modal representation sequence.

[0018] As a preferred embodiment of the present invention, the initial modality representation sequence is processed by a multi-head attention module with a one-way masking mechanism to obtain a semantic cooperative signal; including:

[0019] ;

[0020] ;

[0021] In the formula, As an intermediate variable, For layer normalization operation, For the first The initial mode representation sequence of the layer, For the multi-head self-attention layer, This is a point-to-point feedforward network, and Mask is a one-way attention mask. For the first Semantic coordination signals of layers.

[0022] As a preferred embodiment of the present invention, the step of obtaining a multi-level enhanced modal representation by weighted fusion of the semantic cooperative signal and the initial modal representation sequence includes:

[0023] ;

[0024] In the formula, For the first Multi-level enhanced modal representation, It is an adjustable hyperparameter.

[0025] As a preferred embodiment of the present invention, the step of inputting the multi-level enhanced modal representation into a hybrid expert structure for processing to obtain a fine-grained semantic representation includes:

[0026] The hybrid expert structure includes One expert network and one gated network;

[0027] The gated network generates the weights for each expert network based on the multi-level augmented modal representation; the expression is:

[0028] ;

[0029] In the formula, For gating networks to process the first The weights assigned to the nth expert network in a multi-layered enhanced modal representation. is an activation function, and are learnable weights and bias of the gating network, respectively;

[0030] The expert network performs a nonlinear transformation on the multi-level enhanced modal representation; the expression is:

[0031]

[0032] wherein, is the output of the jth expert network in the ith layer as input, is the output of the jth expert network in the ith layer as input, is the output of the jth expert network in the ith layer as input, and is the learnable parameter of the jth expert network in the ith layer, is a nonlinear activation function; The outputs of each of the expert networks are weighted and summed to obtain a fine-grained semantic representation; the expression is:

[0033]

[0034]

[0035] wherein, is the fine-grained semantic representation in the ith layer. As a preferred scheme of the present application, the global semantic embedding vector is input into a hybrid expert fusion module for processing to obtain an item final modal representation; the processing includes:

[0036] The hybrid expert fusion module includes

[0037] fusion expert networks and a gating network; The gating network calculates the weight of each fusion expert network according to the global semantic embedding vector; the expression is:

[0038]

[0039]

[0040] wherein, is the gating weight assigned to the jth fusion expert network, is the global semantic embedding vector, and are learnable weights and bias of the gating network, respectively; Each fusion expert network processes the corresponding fine-grained semantic representation; the expression is:

[0041]

[0042] ​​​​​​​​

[0043] wherein, is the final modality representation of the item, as input, the output of the i-th fusion expert network, and are the learnable weight and bias of the i-th fusion expert network, respectively;

[0044] The final modality representation of the item is obtained by weighted summation of the outputs of all fusion expert networks; the expression is as follows:

[0045] ;

[0046] wherein, is the final modality representation of the item.

[0047] As a preferred solution of the present application, the user multi-modal behavior sequence is constructed based on the final modality representation of the item, and the user multi-modal behavior sequence is input into a sequence recommendation model for encoding to obtain a user behavior representation; comprising:

[0048] The user multi-modal behavior sequence is constructed according to the time sequence of user and item interaction based on the final modality representation of the item; the sequence recommendation model is a Transformer encoder, the user multi-modal behavior sequence is input into the Transformer encoder for time sequence encoding, and a user behavior representation sequence is output; the vector at the last position in the user behavior representation sequence is extracted as the user behavior representation fused with time sequence information.

[0049] As a preferred solution of the present application, the preference score is obtained by calculating the inner product of the final modality representation of the item and the user behavior representation, and the next step interaction of the user is predicted according to the preference score; comprising:

[0050] For each candidate item in the candidate item set, the inner product of the user behavior representation and the final modality representation of the candidate item is calculated as the preference score of the user for the candidate item:

[0051] ;

[0052] wherein, is the user behavior representation, is the preference score of the user for the candidate item , and is the final modality representation of the candidate item ;

[0053] ​Sort all candidate items by preference scores in descending order, and select the top K items by score to form a list of items for the next interaction recommended to the user.

[0054] As a preferred embodiment of the present invention, a training model is also included:

[0055] For each user and their time-ordered sequence of historical user interactions, perform the following operations:

[0056] S1, For users Each position that needs to be predicted in the historical interaction sequence , will go The nth item is used as the input sequence, with the nth item as the input sequence. Items As the positive sample item for the current prediction;

[0057] S2, Calculate User For positive sample items raw preference scores :

[0058] ;

[0059] In the formula, This is a representation of user behavior based on the input sequence. Positive sample items The final modal representation;

[0060] S3, Correcting sample items After performing popularity bias correction, the adjusted positive sample items are obtained. Preference score :

[0061] ;

[0062] In the formula, Positive sample items Popularity;

[0063] S4. Select all users not belonging to the current training batch. For each item in the historical interaction item set, construct a negative sample set; for each negative sample item... After performing popularity bias correction, the adjusted negative sample items are obtained. Preference score :

[0064] ;

[0065] ;

[0066] wherein, is the original preference score of the user u for the negative sample item , is the final modal representation of the negative sample item , is the popularity of the negative sample item j;

[0067] S5, calculate the normalization denominator :

[0068] ;

[0069] wherein, is the set of all items in the current training batch, is the set of items that the user has interacted with historically;

[0070] the predicted probability of the positive sample item is:

[0071] ;

[0072] wherein, is the probability of the user selecting the positive sample item is the next interaction item;

[0073] S6, calculate the training loss based on the multi-class cross-entropy loss function :

[0074] ;

[0075] wherein, is the length of the user multi-modal behavior sequence, is the set of all users in the training set;

[0076] S7, calculate the gradient of the loss function to all learnable parameters in the training model through the back propagation algorithm, use the AdamW optimizer, update the parameters of the training model according to the gradient, repeatedly execute S1 to S6 for iterative training, when the recommendation performance index of the training model on the validation set no longer improves in continuous multiple training periods, it is determined that the training model converges.

[0077] A semantic collaborative modeling system for multi-modal sequence recommendation, comprising:

[0078] The multi-modal collaborative semantic representation module is configured to construct a user historical interaction behavior sequence according to a user historical interaction behavior, extract text information and image information of each item in the user historical interaction behavior sequence, and input the text information and the image information into a multi-modal encoder to obtain multi-level semantic representations; and arrange the multi-level semantic representations in chronological order to obtain an initial modal representation sequence; the multi-modal encoder includes a text encoder and an image encoder;

[0079] The multi-dimensional collaborative signal extraction module is configured to process the initial modal representation sequence by using a multi-head attention module with a one-way mask mechanism to obtain semantic collaborative signals, integrate the semantic collaborative signals into the initial modal representation sequence, and perform weighted fusion on the semantic collaborative signals and the initial modal representation sequence to obtain multi-level enhanced modal representations;

[0080] The expert network mixing module is configured to input the multi-level enhanced modal representations into a mixed expert structure for processing to obtain fine-grained semantic representations;

[0081] The adaptive multi-modal representation fusion module is configured to splice the fine-grained semantic representations to obtain a global semantic embedding vector, input the global semantic embedding vector into a mixed expert fusion module for processing to obtain an item final modal representation, construct a user multi-modal behavior sequence based on the item final modal representation, input the user multi-modal behavior sequence into a sequence recommendation model for encoding to obtain a user behavior representation, calculate an inner product of the item final modal representation and the user behavior representation to obtain a preference score, and predict a next interaction of the user according to the preference score.

[0082] The present application has the advantages that the problems of insufficient collaborative signal mining, weak cross-scene migration ability and limited multi-modal representation precision in the prior art are effectively solved, and the recommendation accuracy and migratability are significantly improved. The present application can not only generate more accurate and personalized user interest representations, but also reuse multi-modal collaborative knowledge between different recommendation scenarios, thereby reducing the cost of retraining models in new tasks by traditional methods, and embodying obvious technical progress and industrial application value. BRIEF DESCRIPTION OF DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0084] Figure 1 It is a flow chart of a multi-modal sequence recommendation oriented semantic collaboration modeling method in the embodiments of the present application.

[0085] Figure 2 A structure diagram of a semantic collaborative modeling system for multi-modal sequence recommendation in an embodiment of the present application;

[0086] Figure 3 A comparison diagram of the traditional sequence recommendation model and the multi-modal sequence recommendation model in an embodiment of the present application. DETAILED DESCRIPTION

[0087] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described below in detail with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0088] As shown in Figure 1 , a first embodiment of the present application provides a semantic collaborative modeling method for multi-modal sequence recommendation, comprising:

[0089] S1, constructing a user historical interaction behavior sequence according to the user historical interaction behavior; extracting the text information and image information of each item in the user historical interaction behavior sequence, and inputting them into a multi-modal encoder for encoding to obtain multi-level semantic representation; arranging the multi-level semantic representation in chronological order to obtain an initial modal representation sequence; the multi-modal encoder comprises a text encoder and an image encoder.

[0090] There is a user set U and an item set V. For any user u, the interaction with the item in chronological order forms a user historical interaction behavior sequence , which can be expressed as , wherein represents the i-th item interacted in history, i also represents the length of the user historical interaction behavior sequence , and the items in the user historical interaction behavior sequence are not isolated IDs, but are associated with rich multimedia information, including text information (such as product title, description) and image information (such as product main picture, detail picture).

[0091] The text information is input into the text encoder In this case, the text encoder is the BERT, RoBERTa model; the unique feature of this text encoder is that it not only obtains the final output result, but also extracts all the hidden states of multiple intermediate layers (from shallow to deep) and the final output layer inside the model, expressed as:

[0092] ;

[0093] In the formula, is the layer index, is the hidden state representation of the text information of the i-th item at the l-th layer, is the text information of the i-th item.

[0094] The image information is input into the image encoder . This image encoder is the ResNet, ViT model, and similarly, the image encoder extracts the hidden states of multiple intermediate layers and the final output layer, expressed as:

[0095] ;

[0096] In the formula, is the hidden state representation of the image information of the i-th item at the l-th layer, is the image information of the i-th item.

[0097] Through the above operation, for each item in the sequence, two sets of representation are obtained:

[0098] a. From the text, it contains the complete information spectrum from shallow syntactic features (such as words, parts of speech) to deep semantic features (such as theme, emotion).

[0099] b. From the image, it contains the complete visual hierarchy from the bottom visual features (such as edges, textures) to the high-level abstract features (such as objects, scenes).

[0100] These constitute the multi-level semantic representation.

[0101] Arrange all the items that the user has interacted with in the order of time according to their respective multi-level semantic representations to form an initial modal representation sequence , including:

[0102] Text modal representation sequence, this sequence is represented as , which represents the process of the user reading the text content over time.

[0103] Image modal representation sequence, this sequence is represented as , which represents the process of the user browsing the visual content over time.

[0104] S2, processing the initial modality representation sequence by using a multi-head attention module with a one-way mask mechanism to obtain a semantic coordination signal; integrating the semantic coordination signal into the initial modality representation sequence; and performing weighted fusion according to the semantic coordination signal and the initial modality representation sequence to obtain a multi-level enhanced modality representation.

[0105] A multi-head self-attention module with a mask mechanism is used to simulate the user's thinking process when it analyzes the i-th behavior of the user:

[0106] The mask mechanism is like a baffle, which ensures that only the first to the i-1th historical items can be focused on, and the i-th and subsequent (future) items cannot be seen at all.

[0107] The multi-head attention module will examine the initial modality representation sequence from multiple different “angles” or “subspaces”, for example, one “angle” may focus on functional association (such as “tennis racket” and “sports socks”), and another “angle” may focus on style association (such as “simple style table” and “chair of the same style”).

[0108] And the multi-head attention module can dynamically and finely extract the most relevant semantic information from the historical context of the initial modality representation sequence to the current item, and finally converge into a new representation, i.e. semantic coordination signal This signal captures the internal relationship between items in the user's interest stream, and integrates the semantic coordination signal into the original modality representation, the purpose is to make the initial modality representation sequence information of the item have semantic association of user behavior history, so as to more effectively model the user's interest.

[0109] The process of calculating the semantic coordination signal follows the classic structure of the Transformer model, and the expression is:

[0110] ;

[0111] ;

[0112] In the formula, is an intermediate variable, is a layer normalization operation, is the initial modality representation sequence of the i-th layer, is a multi-head self-attention layer, is a point-by-point feedforward network, Mask is a one-way attention mask, is the semantic coordination signal of the i-th layer.

[0113] ​​In the above expression, context association is calculated through a masked multi-head self-attention layer Attn, using residual connections (plus the input). To prevent information loss, layer normalization (LN) is used to stabilize the training process; The model's expressive power is enhanced by performing a nonlinear transformation through a pointwise feedforward network (FFN, a small neural network). Residual connections and layer normalization are then used again to output the final semantic co-operation signal. .

[0114] Through the derived semantic cooperative signals Initial mode representation sequence and a predefined hyperparameter that can be trained or tuned on a validation set. The multi-level enhanced modal representation is obtained through the expression. :

[0115] ;

[0116] In the formula, "Weightedness of semantic coordination signals, The value range is typically [0, 1], representing the weight or strength we want the "semantic co-signal" to account for in the final enhanced representation; This indicates that the weighted cooperative signal is added element-wise to the initial modal representation, forming a typical residual connection. This ensures that regardless of the fusion strength, the most original and essential feature information of the item is preserved. All of these can be passed to the next layer without loss, preventing information decay or gradient vanishing during deep network processing.

[0117] S3. Input the multi-level enhanced modal representation into the hybrid expert structure for processing to obtain a fine-grained semantic representation.

[0118] For the multi-level enhanced modal representation that has been obtained The input is fed into a hybrid expert structure, which includes multiple independent expert networks and a gated network.

[0119] First, the gating network acts as an "intelligent scheduler," and its input is a multi-level augmented modal representation. Its internal implementation is a linear transformation layer with parameters as follows: ,in The dimension is This means it maps d-dimensional input features to an N-dimensional space, where each dimension of this N-dimensional vector corresponds to an expert network; the output of the linear transformation is then fed into the Softmax function for normalization, ultimately outputting an N-dimensional probability distribution vector:

[0120] each component of this vector represents the current input multi-level enhanced modal representation the weight that should be assigned to the nth expert network, and the sum of all weights is 1, which realizes adaptive and competitive allocation of expert resources; the expression is:

[0121] ;

[0122] where, is the weight assigned to the nth expert network for processing the nth layer of multi-level enhanced modal representation, is the weight assigned to the nth expert network for processing the nth layer of multi-level enhanced modal representation, is the activation function, and are the learnable weights and biases of the gating network.

[0123] Secondly, the expert network plays the role of a "feature refinement committee", each expert network is a simple single-layer feedforward network in structure, and the expression is:

[0124] ;

[0125] where, is the output of the nth expert network for the nth layer as input, is the output of the nth expert network for the nth layer as input, is the output of the nth expert network for the nth layer as input, is the output of the nth expert network for the nth layer as input, is the output of the nth expert network for the nth layer as input, is the output of the nth expert network for the nth layer as input, are the learnable parameters of the nth expert network, which enable them to learn different, specialized patterns or features in the input data, is the introduction of a nonlinear activation function (such as ReLU or GELU), which gives each expert network the ability to capture complex nonlinear transformations.

[0126] Finally, integration is performed by weighted summation, and the calculation formula of the fine-grained semantic representation is:

[0127] ;

[0128] where, is the nth layer of fine-grained semantic representation, is the total number of expert networks.

[0129] ​The above operation can be understood as: the gating network dynamically combines a "temporary committee" according to the input, and weights and fuses the output results of each "expert", and the result is that, for different inputs, the most relevant feature transformation path can be automatically emphasized, and irrelevant or noisy paths can be suppressed, so that the collaborative semantics is focused in a fine-grained manner, and a refined and optimized fine-grained semantic representation with better discrimination is output , for subsequent tasks.

[0130] S4, splice the fine-grained semantic representation to obtain a global semantic embedding vector; and input the global semantic embedding vector into a hybrid expert fusion module for processing to obtain a final modal representation of the item.

[0131] Splice all the fine-grained semantic representations refined by the hybrid expert structure on all M levels, i.e. , into a comprehensive global semantic embedding vector , to provide a global view containing all abstract level information for subsequent decision-making.

[0132] Global weight distribution is performed by a gating network, which takes a fully connected layer as the core, and the parameters are and , the global semantic embedding vector obtained in the previous step as input, and outputs a gating network weight vector with a dimension of after linear transformation and Softmax function; each scalar weight here is not determined independently by the single level it corresponds to, but is dynamically calculated based on the comprehensive information of the global semantic embedding vector of all fine-grained semantic representations; the expression is:

[0133]

[0134] In the formula, is the gating weight allocated to the th fusion expert network, is the global semantic embedding vector, and are the learnable weights and biases of the gating network, L is the total number of fusion expert networks, and d is the dimension of the global semantic embedding vector.

[0135] Each fusion expert network performs specific processing on the fine-grained semantic representation at each level, and unlike the global perspective of the gating network, each semantic level in the fine-grained semantic representation is equipped with an independent fusion expert network​ , and each fusion expert network is a simple single-layer feedforward network with its own dedicated parameters and and the fine-grained representation is obtained by performing a last nonlinear feature transformation; the expression is:

[0136] ;

[0137] wherein is the i-th fusion expert network, and are the learnable weights and biases of the i-th expert network, denotes the non-activation function, is the i-th layer fine-grained semantic representation; ensuring that each level of semantic information from shallow to deep is specially and independently optimized before fusion. Based on the above gating network and fusion expert network, the weighted fusion product item final modality representation is generated, and the expression is:

[0138]

[0139] ;

[0140] wherein is the item final modality representation, and L is the total number of fusion expert networks.

[0141] The above expression can be understood as: the weighted sum of the outputs of all L fusion expert networks, wherein the contribution of each expert network is determined by the weight assigned by the gating network based on global information, which enables adaptive construction of fusion strategies for different items, for example, some items may rely more on their deep abstract semantics, while for other items, their shallow visual texture may be more discriminative; is the gating weight of the i-th fusion expert. The final output m is a powerful representation that combines the inherent multi-modal characteristics of the item itself, the semantic association of the user sequence context, and is optimized across layers, laying the foundation for subsequent accurate interest prediction. S5, based on the item final modality representation, a user multi-modal behavior sequence is constructed, and the user multi-modal behavior sequence is input into a sequence recommendation model for encoding to obtain a user behavior representation; a preference score is obtained by calculating the inner product of the item final modality representation and the user behavior representation, and the next step of the user's interaction is predicted according to the preference score.

[0142] ​​​​

[0143] transforming the sequence of user historical interaction behaviors into a sequence of user multi-modal behaviors with rich semantic information each element in the sequence is no longer an original ID or a coarse feature, but a final modal representation of the i-th item after fine-grained semantic collaborative perception and cross-level fusion

[0144] The sequence recommendation model is a Transformer encoder, in order to capture the dynamic pattern of user interest evolution over time, the above sequence of user multi-modal behaviors is input into the Transformer encoder, which is a sequence of user multi-modal behaviors composed of T vectors , each vector corresponds to the final modal representation of a historical interaction item, containing fused multi-modal semantic information such as text and image. In order to preserve the time sequence order, position encoding is added to each item representation, and the sequence of user multi-modal behaviors is sent into the Transformer encoder stack, the core of which is the multi-head self-attention mechanism; the representation of each item will interact with the representations of all other items (including itself) in the sequence of user multi-modal behaviors, calculate attention weights, and automatically learn the dependency between any two items in the sequence of user multi-modal behaviors, no matter how far they are apart, so as to globally understand the evolution and context of user interest; each attention sub-layer is followed by a feedforward neural network for nonlinear transformation, and the above "attention-feedforward" structure will be repeatedly stacked for N layers, so that the model can refine more complex sequence patterns layer by layer, and the Transformer encoder finally outputs a sequence of user behavior representations with the same length as the input sequence of user multi-modal behaviors

[0145] The core operation of prediction is to calculate the matching degree between the current interest state of the user and all candidate items, and the final modal representation of each candidate item v is obtained from the entire item set V , then the preference score of the user for each candidate item is calculated by computing the inner product of vectors, the expression is:

[0146] ;

[0147] wherein, is the user behavior representation, ​​​​​​​​Let be the preference score of the candidate item; and in the expression, the inner product can be geometrically interpreted as calculating the similarity between two vectors. The larger the inner product value, the closer the user's behavior representation is to the final modal representation of the candidate item in the vector space, that is, the stronger the user's preference for the item.

[0148] Collect preference scores for all candidate items, sort them in descending order, and determine the top K items with the highest scores as the items most relevant to the user's current interests and most likely to be interacted with next, thus forming the final personalized recommendation list output.

[0149] The method also includes a training model for the prediction process, which employs a multi-class cross-entropy loss function with popularity adjustment, including:

[0150] S51. For each user in the training batch and their historical interaction behavior sequence. The training model employs an autoregressive next-term prediction task; specifically, for each position to be predicted in the sequence... Using all t-1 items preceding a given position as input, the learning objective is to predict the actual interactive item at position t. The item is defined as a positive sample item for the current prediction task.

[0151] S52. Perform score calculation and popularity bias correction; first, calculate the original preference score of user u for the target positive sample item:

[0152] ;

[0153] In the formula, This is a representation of user behavior based on the input sequence. Positive sample items The final modal representation;

[0154] S53. Subsequently, a bias correction term is introduced, which subtracts the popularity of the item from the original score. The logarithm of the result is used to obtain the adjusted preference score for the positive sample items:

[0155] ;

[0156] Here Calculated as positive sample items The frequency of an item in the training set is used to determine its probability of being predicted. The more popular an item is, the higher its probability of being predicted. Subtracting its log popularity weakens the model's inherent advantage due to its popularity, forcing the model to rely on learned real users and semantic matching relationships between items to make predictions.

[0157] S54, select the items not in the user u history interaction set as negative sample items, and perform the same adjustment calculation for negative sample items j, the expression is:

[0158] ;

[0159] ;

[0160] In the formula, is the original preference score of user u to negative sample item , is the final modal representation of negative sample item , is the popularity of negative sample item j;

[0161] S55, adopt the intra-batch negative sampling strategy; all items in the current training batch that do not belong to the user u history interaction set are automatically considered as negative samples for this prediction:

[0162] ;

[0163] In the formula, is the set of all items in the current training batch, is the set of items that user has interacted with in history;

[0164] Based on the adjusted score, the prediction probability of the positive sample item is calculated:

[0165] ;

[0166] In the formula, is the normalization denominator, is the probability of user selecting positive sample item is the probability of the next interaction item;

[0167] S56, the above formula measures the relative matching degree of the positive sample item in all candidate samples (positive samples + intra-batch negative samples) after popularity adjustment, therefore, the final training target, i.e. the multi-class cross entropy loss function gets the training loss :

[0168] ;

[0169] In the formula, is the length of the user multi-modal behavior sequence;

[0170] S57. Calculate the gradient of the loss function with respect to all model parameters (including the encoder, one-way attention mechanism, expert network, etc.) using the backpropagation algorithm, and update the parameters using the AdamW optimizer. The termination condition for model training is usually set as follows: when the recommended performance metric (such as normalized loss cumulative gain) on the validation set no longer improves over multiple consecutive training epochs, the model is considered to have converged. First, calculate the loss. For prediction probability The gradient of the input is then backpropagated to the Transformer encoder, where the gradients of all learnable parameters within the Transformer encoder (including the query, key, and value projection matrices in the self-attention mechanism, the weights and biases of the feedforward network, and layer normalization parameters) are calculated based on their impact on the prediction probability. Next, the gradient is further propagated to the fusion expert networks, where the gradients of the weights and biases of each fusion expert network are determined based on its contribution to the input of the Transformer encoder (i.e., the final representation of the item). Then, the gradient continues to propagate to the hybrid expert structure, where the gradients of the parameters of each expert network and the gating network are calculated sequentially.

[0171] The entire process is efficiently completed at the lowest level by the automatic differentiation engine of deep learning frameworks (such as PyTorch and TensorFlow). Ultimately, every learnable parameter in the model (whether it's in the Transformer encoder, attention mechanism, or various expert networks) receives a gradient value, which precisely indicates the direction and magnitude of adjustment that parameter should take to reduce the overall loss. Subsequently, the AdamW optimizer uses this entire set of gradient information to update all parameters, completing one training iteration.

[0172] like Figure 2 As shown, this is the second embodiment of the present invention. This embodiment provides a semantic collaborative modeling system for multimodal sequence recommendation, including:

[0173] A multimodal collaborative semantic representation module is used to construct a sequence of user historical interaction behaviors based on the user's historical interaction behaviors; extract the text information and image information of each item in the sequence of user historical interaction behaviors and input them into a multimodal encoder for encoding to obtain a multi-level semantic representation; arrange the multi-level semantic representations in chronological order to obtain an initial modal representation sequence; the multimodal encoder includes a text encoder and an image encoder;

[0174] A multi-dimensional collaborative signal extraction module is configured to process the initial modality representation sequence by using a multi-head attention module with a one-way mask mechanism to obtain semantic collaborative signals, integrate the semantic collaborative signals into the initial modality representation sequence, and perform weighted fusion according to the semantic collaborative signals and the initial modality representation sequence to obtain multi-level enhanced modality representations.

[0175] An expert network mixing module is configured to input the multi-level enhanced modality representations into a mixed expert structure for processing to obtain fine-grained semantic representations.

[0176] An adaptive multi-modal representation fusion module is configured to splice the fine-grained semantic representations to obtain global semantic embedding vectors, input the global semantic embedding vectors into a mixed expert fusion module for processing to obtain final modality representations of items, construct user multi-modal behavior sequences based on the final modality representations of items, input the user multi-modal behavior sequences into a sequence recommendation model for encoding to obtain user behavior representations, calculate inner products of the final modality representations of items and the user behavior representations to obtain preference scores, and predict next-step interactions of users according to the preference scores.

[0177] As shown in Figure 3 the third embodiment of the present application, the embodiment provides a comparison of the present method with a conventional sequence recommendation model and a multi-modal sequence recommendation model, including:

[0178] The present application performs system experiments on three public data sets (a musical instrument data set, an office product data set, and an industrial and scientific data set), the experimental design follows the principle of training, validation, and test set division, and ensures the time sequence of user historical interaction behavior sequences to truly reflect dynamic behavior patterns in a recommendation scenario.

[0179] As shown in Figure 3The sequence recommendation models shown for comparison with the present application are: GRU4Rec for processing user short-term behavior sequences to make next-item predictions; SASRec for capturing long-term dependencies in user behavior sequences using self-attention; NOVA for improving sequence modeling stability through normalized attention mechanisms; UniSRec for solving cross-domain / cold-start recommendation problems and learning general item representations; MISSRec for fusing text, image, and other multi-modal content for sequence recommendation; IISAN for extracting and utilizing dynamic user intent from behavior sequences for recommendation; TeDRec for modeling temporal dynamics and filtering interaction noise in sequence recommendation; IISAN-VS for adding visual feature enhanced representation in intent-aware sequence recommendation; PMMRec for dynamically fusing multi-modal features according to user personalized preferences; and HM4SR for automatically filtering redundant noise in multi-modal information through two-level expert mixing and multi-task learning, and explicitly modeling changes in user interest over time to more accurately predict the user's next interaction item.

[0180] The evaluation indicators are "normalized discounted cumulative gain (top five and top ten)" and "hit rate (top five and top ten)" to measure the performance of the model in capturing user interest and recommendation accuracy. The experimental results show that the present application significantly outperforms existing baseline methods on all data sets; on the musical instrument data set, the "normalized discounted cumulative gain (top five)" is improved by 15.0%, the "normalized discounted cumulative gain (top ten)" is improved by 13.8%, the "hit rate (top five)" is improved by 17.1%, and the "hit rate (top ten)" is improved by 18.7%, which is more than 27.5% higher than the multi-modal model (such as IISAN, IISAN-VS) that does not rely on ID, indicating the key role of semantic collaboration signals in the performance of ID-independent recommendation. On the office product data set, the "normalized discounted cumulative gain (top five)", "normalized discounted cumulative gain (top ten)", "hit rate (top five)", and "hit rate (top ten)" are improved by 27.7%, 23.0%, 19.5%, and 28.7% respectively compared to the personalized multi-modal recommendation model (such as PMMRec), fully demonstrating that the present application can accurately capture user interest changes and multi-modal interaction information in large-scale, diverse item sets. On the industrial and scientific data set, the "normalized discounted cumulative gain (top five)" is improved by 21.1%, the "normalized discounted cumulative gain (top ten)" is improved by 17.8%, the "hit rate (top five)" is improved by 15.7%, and the "hit rate (top ten)" is improved by 14.4%, verifying the robustness and generalization ability of the model under different sparsity and modal distribution conditions. In addition, the experimental results also show that multi-modal recommendation methods generally outperform traditional ID-dependent methods, further highlighting the importance of modeling user dynamic interest using multi-modal information.

[0181] In summary, the present application extracts fine-grained semantic collaborative signals explicitly in multi-modal sequence recommendation, integrates potential collaborative information in user historical behavior sequence into multi-modal item representation directly, further refines and optimizes multi-modal representation by adopting hierarchical hybrid expert architecture, and dynamically fuses multi-level semantic collaborative information through representation-specific expert and gating mechanism to realize fine modeling of user interest. The method not only improves recommendation accuracy, eliminates dependence on item ID, but also enhances the knowledge transfer ability of the model in cross-scene, cold start and other environments, effectively solving the problems of insufficient expression ability, insufficient collaborative signal mining and limited generalization ability in existing methods.

[0182] In the above embodiments, all or part of them can be realized by software, hardware, firmware or any other combination. When realized by software, all or part of them can be realized in the form of a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0183] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above integrated module can be realized in the form of hardware or software function module. When the above integrated module is realized in the form of software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk.

[0184] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A semantic collaborative modeling method for multi-modal sequential recommendation, characterized in that, The method comprises the following steps: According to the user's historical interaction behavior, a user historical interaction behavior sequence is constructed; Text information and image information of each item in the user historical interaction behavior sequence are extracted and input into a multi-modal encoder for encoding to obtain multi-level semantic representations; The multi-level semantic representations are sorted in chronological order to obtain an initial modal representation sequence; comprising: According to the time sequence of the user's historical interaction behavior, a user historical interaction behavior sequence is constructed; The user historical interaction behavior sequence is composed of I items, and each item includes text information and image information; The text information is input into a text encoder, and the image information is input into an image encoder; and the hidden states of multiple intermediate layers and the final output layer of each encoder are extracted to obtain the multi-level semantic representations of the items; The multi-level semantic representations of all items in the user historical interaction behavior sequence are obtained, and all multi-level semantic representations are arranged in chronological order to obtain an initial modal representation sequence; wherein the initial modal representation sequence includes a text modal representation sequence and an image modal representation sequence; The multi-modal encoder includes a text encoder and an image encoder; A multi-head attention module with a one-way mask mechanism is used to process the initial modal representation sequence to obtain a semantic coordination signal; comprising: wherein, is an intermediate variable, is a layer normalization operation, is a first layer of an initial modal representation sequence, is a multi-head self-attention layer, is a point-wise feed-forward network, Mask is a unidirectional attention mask, is a first layer of a semantic synergistic signal; The semantic coordination signal is integrated into the initial modal representation sequence; the semantic coordination signal and the initial modal representation sequence after integrating the semantic coordination signal are weighted and fused to obtain a multi-level enhanced modal representation; comprising: In the formula, is the first layer multi-level enhanced modal representation, is a tunable hyper-parameter; The multi-level enhanced modal representation is input into a hybrid expert structure for processing to obtain a fine-grained semantic representation; The fine-grained semantic representations are spliced to obtain a global semantic embedding vector; and the global semantic embedding vector is input into a hybrid expert fusion module for processing to obtain an item final modal representation; Based on the item final modal representation, a user multi-modal behavior sequence is constructed, and the user multi-modal behavior sequence is input into a sequence recommendation model for encoding to obtain a user behavior representation; the inner product of the item final modal representation and the user behavior representation is calculated to obtain a preference score, and the next step of the user's interaction is predicted according to the preference score.

2. The semantic collaborative modeling method for multi-modal sequence recommendation according to claim 1, characterized in that, The multi-level enhanced modal representation is input into a hybrid expert structure for processing to obtain a fine-grained semantic representation; comprising: The hybrid expert structure comprises one expert network and one gating network; The gating network generates the weight of each expert network according to the multi-level enhanced modal representation; the expression is: wherein, is a gating network for processing the first is a weight assigned to the n-th expert network of the multi-layered enhanced modal representation, is an activation function, and are a learnable weight and bias of the gating network, respectively. The expert network performs a nonlinear transformation on the multi-level enhanced modal representation; the expression is: wherein is a function of the first input to the layer of the first expert network, and is a learnable parameter of the first expert network, is a non-linear activation function; The output of each expert network is weighted and summed according to the weight to obtain a fine-grained semantic representation; the expression is: In the formula, is the first layer fine-grained semantic representation.

3. The method of claim 2, wherein, The global semantic embedding vector is input into a hybrid expert fusion module for processing to obtain an item final modal representation; comprising: The hybrid expert fusion module comprises a fusion expert network and a gating network; The gating network calculates the weight of each fusion expert network according to the global semantic embedding vector; the expression is: wherein, is the gating weight assigned to the th fusion expert network, is the global semantic embedding vector, and are the learnable weight and bias of the gating network, respectively. Each fusion expert network processes the corresponding fine-grained semantic representation; the expression is: wherein is a function of the input to the output of the and are the learnable weights and bias of the fusion expert network, respectively The outputs of all fusion expert networks are weighted and summed to obtain an item final modal representation; the expression is: In the formulae, is the final modal representation of the item.

4. The method of claim 3, wherein, The method comprises the following steps: The user multi-modal behavior sequence is constructed based on the final modal representation of the item, and the user multi-modal behavior sequence is input into a sequence recommendation model for encoding to obtain a user behavior representation. The user multi-modal behavior sequence is constructed based on the final modal representation of the item in the time sequence of the user interacting with the item.

5. The method of claim 4, wherein, The sequence recommendation model is a Transformer encoder, the user multi-modal behavior sequence is input into the Transformer encoder for time sequence encoding, and a user behavior representation sequence is output. For each candidate item in the candidate item set , compute an inner product of the user behavior representation and a final modality representation of the candidate item as a preference score of the candidate item by the user: wherein, is a user behavior representation, is a user preference score for a candidate item , is a final modal representation of a candidate item . The inner product of the final modal representation of the item and the user behavior representation is calculated to obtain a preference score, and the next step interaction of the user is predicted according to the preference score.

6. The method of claim 5, wherein the method is a semantic collaborative modeling method for multi-modal sequence recommendation. The preference scores of all candidate items are sorted in descending order, and the top K items in the score ranking are selected to form a next step interaction item list recommended to the user. The model is trained as follows: S1, for each position in the history interaction behavior sequence of the user that needs to be predicted , the previous items as input sequence, the item as the positive sample item of the current prediction; S2, computing user to positive sample items of original preference scores : wherein, is a representation of user behavior based on the input sequence, is a positive sample item final modality representation; S3, positive sample item performing popularity bias correction to obtain an adjusted preference score for the positive sample item :​ wherein is the prevalence of positive sample items ​ S4, select all items not belonging to the user from the current training batch items of the historical interaction item set, construct a negative sample set; for each negative sample item perform popularity bias correction to obtain an adjusted negative sample item preference score : wherein, is the original preference score of the user u for the negative sample item , is the final modality representation of the negative sample item , is the popularity of the negative sample item j; S5, compute normalization denominator : wherein, is a set of all items in the current training batch, is a user a set of items that the user has interacted with historically; positive sample item the predicted probability of wherein selecting positive sample items for a user probability of next interaction item; S6、based on the multi-class cross-entropy loss function to calculate the training loss : wherein is the length of the sequence of multi-modal behavior of the user, is the set of all users in the training set; For each user and the time-ordered user historical interaction behavior sequence, the following operations are performed:

7. A semantic collaborative modeling system for multi-modal sequential recommendation, according to the semantic collaborative modeling method for multi-modal sequential recommendation of any one of claims 1-6, characterized in that, S7, the loss function is calculated by the back propagation algorithm to obtain the gradient of all learnable parameters in the training model, the AdamW optimizer is adopted, the parameters of the training model are updated according to the gradient, and the iteration training of S1 to S6 is repeatedly performed, when the recommendation performance index of the training model on the validation set no longer improves in continuous multiple training periods, it is determined that the training model converges. The method comprises the following steps: A multi-modal collaborative semantic representation module is used to construct a user historical interaction behavior sequence according to user historical interaction behavior. Text information and image information of each item in the user historical interaction behavior sequence are extracted and input into a multi-modal encoder for encoding to obtain multi-level semantic representations; the multi-level semantic representations are arranged in time sequence to obtain an initial modal representation sequence; the multi-modal encoder comprises a text encoder and an image encoder. A multi-dimensional collaborative signal extraction module is used to process the initial modal representation sequence by using a multi-head attention module with a one-way mask mechanism to obtain semantic collaborative signals; the semantic collaborative signals are integrated into the initial modal representation sequence; and the semantic collaborative signals and the initial modal representation sequence are weighted and fused to obtain multi-level enhanced modal representations. An expert network hybrid module is used to input the multi-level enhanced modal representations into a hybrid expert structure for processing to obtain fine-grained semantic representations. An adaptive multi-modal representation fusion module is used to splice the fine-grained semantic representations to obtain a global semantic embedding vector; and the global semantic embedding vector is input into a hybrid expert fusion module for processing to obtain an item final modal representation. The user multi-modal behavior sequence is constructed based on the final modal representation of the item, and the user multi-modal behavior sequence is input into a sequence recommendation model for encoding to obtain a user behavior representation. The inner product of the final modal representation of the item and the user behavior representation is calculated to obtain a preference score, and the next step interaction of the user is predicted according to the preference score.

Images