A method for determining a text case, a device for determining a text case, an electronic device, and a computer-readable storage medium

Abstract

The application provides a method for determining a script, a device for determining a script, an electronic device and a computer program product; the method comprises: obtaining a target image related to a first user, a candidate script set corresponding to the target image and behavior data of the first user in a first time period; performing feature extraction on the target image to obtain a first type of features related to vision; preprocessing the candidate script set to obtain a second type of features related to script semantics; determining a third type of features related to script operation behavior of the first user from the behavior data; and determining a recommended script of the target image based on the first type of features, the second type of features and the third type of features. Through the application, a recommended script meeting the image connotation and personalized needs of a user can be generated.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a method for determining a document, a device for determining a document, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Currently, image tag generation is typically based on computer vision algorithms, such as using convolutional neural networks (CNNs) to extract visual features (color, texture, object category, etc.) from images, and then generating tags describing the image content (such as "beach," "cat," "sunset"). These techniques can only capture the low-level visual information of an image; therefore, the image tags generated by these techniques lack personalization. Summary of the Invention

[0003] This application provides a method for determining text, a device for determining text, an electronic device, and a computer-readable storage medium. Through this application, recommended text that meets the connotation of images and the personalized needs of users can be generated.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for determining text, the method comprising: Acquire a target image related to the first user, a set of candidate texts corresponding to the target image, and behavioral data of the first user within a first time period; Feature extraction is performed on the target image to obtain a first type of visually related features; the first type of features includes color features, texture features, object category features, scene features, and emotion category features; The candidate text set is preprocessed to obtain a second type of features related to the semantics of the text; wherein, the second type of features includes the latent semantic features corresponding to each text and the semantic features obtained by encoding each text. A third type of feature related to the first user's text manipulation behavior is determined from the behavioral data; Based on the first type of features, the second type of features, and the third type of features, the recommended text for the target image is determined.

[0005] This application provides a document determination device, which includes: The acquisition unit is used to acquire a target image related to the first user, a set of candidate texts corresponding to the target image, and behavioral data of the first user within a first time period. The processing unit is used to extract features from the target image to obtain a first type of visually related features; the first type of features includes color features, texture features, object category features, scene features, and emotion category features; The processing unit is further configured to preprocess the candidate copy set to obtain a second type of features related to the semantics of the copy; wherein, the second type of features includes the latent semantic features corresponding to each copy and the semantic features obtained by encoding each copy. The processing unit is further configured to determine a third type of feature related to the first user's text operation behavior from the behavioral data; The processing unit is further configured to determine the recommended text for the target image based on the first type of features, the second type of features, and the third type of features. This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the document determination method provided in the embodiments of this application.

[0006] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the document determination method provided in this application.

[0007] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the document determination method provided in this application.

[0008] The embodiments of this application have the following beneficial effects: Based on image visual features (color, texture, object semantics), user behavior sequence features (historical preferences, real-time interaction) and text semantic features, this application can generate recommended text that meets the connotation of the image and the personalized needs of users, thereby improving the accuracy of recommendations. Attached Figure Description

[0009] Figure 1 This is a first flowchart illustrating the document determination method provided in the embodiments of this application; Figure 2 This is a second flowchart illustrating the document determination method provided in the embodiments of this application. Figure 3 This is a schematic diagram of the structure of the document determination device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0010] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] Before explaining this application, this section will describe the text recommendation methods based on image content analysis and traditional collaborative filtering in the related technologies: The steps of text recommendation methods based on image content analysis and traditional collaborative filtering in related technologies include the following: Image labels are obtained by extracting features from the visual content of the image.

[0015] The image tags are used to search the preset copywriting library. The matching rule is "the tag keywords directly correspond to the fixed copywriting template" (e.g., the tag "snow mountain" corresponds to the copywriting "travel far to the human world to witness the magnificent beauty of the snow mountain").

[0016] Using a collaborative filtering algorithm, based on the text clicked by users in the past, we calculate the similarity between texts and recommend similar texts.

[0017] Among the related technologies, collaborative filtering algorithms include user-based collaborative filtering (UserCF) and item-based collaborative filtering (ItemCF). UserCF recommends items liked by similar users to the target user by calculating the similarity between users; ItemCF recommends other items similar to the interacted items to the user by calculating the similarity between items. The core of these algorithms lies in calculating similarity through the user-item interaction matrix.

[0018] The similarity calculation formula for ItemCF is as follows: (1) Where N(i) represents the set of users who interact with copy i; N(j) represents the set of users who interact with copy j; Formula (1) is based only on the co-occurrence frequency of user-copy interaction.

[0019] Embodiments of this application provide a method for determining a document, which is applied to an electronic device, with reference to... Figure 1 As shown, the method includes the following steps: Step 101: Obtain the target image related to the first user, the candidate text set corresponding to the target image, and the behavioral data of the first user in the first time period.

[0020] In this embodiment of the application, the target image may be uploaded by the first user, indicated by the first user, or touched by the first user, that is, the target image is an image operated by the first user.

[0021] In this embodiment of the application, the candidate text set may include a set of texts historically generated by the first image or the first user; wherein the similarity between the first image and the target image is within a first threshold range or greater than the first threshold.

[0022] In this embodiment of the application, the behavioral data includes historical behavioral data and real-time behavioral data, namely, the first user's text selection records, editing records, and browsing duration data within a first range.

[0023] Here, the first range includes the past month, the past first week, or the current moment.

[0024] In this application embodiment, the electronic device may include mobile terminal devices such as mobile phones, tablets, laptops, personal digital assistants (PDAs), cameras, wearable devices, etc., as well as fixed terminal devices such as desktop computers.

[0025] Step 102: Extract features from the target image to obtain the first type of visually relevant features.

[0026] The first category of features includes color features, texture features, object category features, scene features, and emotion category features.

[0027] In this embodiment of the application, a 50-layer Residual Network (ResNet-50) is used to extract the color and texture features of the target image and output a 2048-dimensional feature vector representing the color and texture features.

[0028] In this embodiment, a Faster Region-based Convolutional Neural Network (Faster R-CNN) architecture is used to identify object categories (such as "cat" and "coffee cup") and scenes (such as "park" and "office"), and to generate bounding box coordinates and confidence scores representing object category features and scene features.

[0029] In this embodiment, an image emotion classifier is constructed by combining a pre-trained Visual Geometry Group 19-layer (VGG19) network with transfer learning, and outputs emotion labels (such as "pleasure", "tranquility", "shock") and corresponding probability values ​​that represent emotion category features.

[0030] Step 103: Preprocess the candidate copy set to obtain the second type of features related to the semantics of the copy.

[0031] The second type of feature includes the latent semantic features corresponding to each copy and the semantic features obtained by encoding each copy.

[0032] In this embodiment of the application, a pre-trained language model (Bidirectional Encoder Representation from Transformers, BERT) based on the Transformer architecture is used to encode the copy library and generate a 768-dimensional semantic vector representing the semantic features of each copy.

[0033] In this embodiment of the application, synonym expansion is performed using WordNet to construct a semantic association network (such as the association between "travel" and "journey" and "hiking"), and the associated semantic features of each word or sentence in the text are generated based on the constructed semantic association network; here, the potential semantic features include associated semantic features.

[0034] In this embodiment of the application, a CNN-Long Short Term Memory network (LSTM) hybrid model is trained to identify the style of the text (literary, humorous, inspirational, etc.) and output a style probability distribution vector; here, the latent semantic features include the features corresponding to the style probability distribution vector.

[0035] Step 104: Identify the third type of features related to the first user's text operation behavior from the behavioral data.

[0036] In this embodiment of the application, the third type of feature includes historical behavior features and real-time behavior features.

[0037] Here, the steps for determining historical behavioral characteristics include: collecting the first user's copywriting selection records, editing operations (such as keyword modification frequency), and browsing duration data within the first time period; modeling time-series features through an LSTM network to capture preference change trends (for example, the first user has recently frequently used "pet + healing" copywriting); LSTM can identify this trend and generate time-series features (such as time-series weights).

[0038] The steps for determining real-time behavioral characteristics include: acquiring the first user's clicks, dwell time, likes, and comments on the candidate text in real time; calculating recent activity characteristics (such as activity weights) using a sliding window algorithm; and calculating the activity weights using the following formula: (2) Among them, w i Weights are assigned to behavior types (e.g., likes = 2, comments = 3, clicks = 1), t i Interaction duration (in seconds).

[0039] Here, the third type of features includes the temporal features generated by LSTM and the activity features calculated by the sliding window algorithm.

[0040] Step 105: Based on the first type of features, the second type of features, and the third type of features, determine the recommended text for the target image.

[0041] This application maps the first type of features, the second type of features, and the third type of features to a unified semantic space, and determines the recommended text for the target image based on the mapped features.

[0042] It should be noted that the method for determining the copy is a cyclical process. The electronic device can first recommend a copy based on the information currently acquired, and then update the recommended copy in real time based on the behavior of the first user.

[0043] This application provides a method for determining text copy; the method includes: acquiring a target image related to a first user, a set of candidate text copy corresponding to the target image, and behavioral data of the first user within a first time period; extracting features from the target image to obtain a first type of visually related features; the first type of features includes color features, texture features, object category features, scene features, and emotion category features; preprocessing the set of candidate text copy to obtain a second type of semantically related features; wherein the second type of features includes latent semantic features corresponding to each text copy and semantic features obtained by encoding each text copy; determining a third type of features related to the first user's text copy operation behavior from the behavioral data; and determining recommended text copy for the target image based on the first type of features, the second type of features, and the third type of features. In other words, this application, based on image visual features (color, texture, object semantics), user behavior sequence features (historical preferences, real-time interactions), and text semantic features, can generate recommended text copy that meets the connotation of the image and the personalized needs of the user, thus improving the accuracy of recommendations and achieving multi-dimensional semantic matching.

[0044] In some embodiments, the recommended text for the target image is determined based on the first type of features, the second type of features, and the third type of features, including the following steps: Step A1: Based on the first type of features, the second type of features, and the third type of features, filter out reference texts from the candidate text set.

[0045] Step A2: Obtain the fused features of the first type of features, the second type of features, and the third type of features.

[0046] In this embodiment, feature fusion includes feature concatenation, weighted summation, attention mechanism fusion, pyramid fusion, deconvolution upsampling fusion, and multi-branch interactive fusion.

[0047] This application uses the CLIP model to map image visual features (color, texture, object semantics), user behavior sequence features (historical preferences, real-time interactions), and text semantic features to a unified semantic space, achieving cross-modal alignment. This fundamentally solves the problem of the disconnect between "image content - user preferences - text semantics" in related technologies, ensuring that recommended text simultaneously satisfies both the image's connotation and the user's personalized needs, which is the basis for improving recommendation accuracy.

[0048] Step A3: Determine the matching score between the fusion feature and each reference text.

[0049] Step A4: Based on the first type of features, the second type of features, and the third type of features, determine the attention score for each reference text.

[0050] Step A5: Based on the matching score and attention score, determine the recommended text for the target image from the second text.

[0051] This application can sort based on matching score and attention score, and select the text with the highest matching degree or the highest attention score as the recommended text for the target image.

[0052] In some embodiments, step A1, filtering reference texts from the candidate text set based on the first type of features, the second type of features, and the third type of features, includes the following steps: Step A11: Map the first type of features and the second type of features into a high-dimensional representation.

[0053] This application introduces Contrastive Language-Image Pre-training (CLIP), which aligns visual features with the text semantic space to generate a 512-dimensional visual-semantic joint embedding vector. For example, when given an image of a snow-capped mountain, the CLIP model can output a feature vector containing semantics such as "magnificent," "cold," and "natural."

[0054] This application achieves zero-sample alignment between visual features and text semantics through CLIP model pre-training, which improves matching accuracy without the need for additional labeled data.

[0055] Step A12: Determine the first weight based on the high-dimensional representation.

[0056] Among them, the first weight corresponds to visual semantic similarity.

[0057] Step A13: Determine the second weight based on the third type of feature.

[0058] The second weight is the weight corresponding to the first user's current interaction behavior.

[0059] Step A14: Determine the third weight based on behavioral data.

[0060] The third weight represents the weight of the temporal changes in the first user's preference.

[0061] This application introduces a triple dynamic weighting system based on the ItemCF algorithm in related technologies, resulting in an improved ItemCF algorithm (D-ItemCF). The triple weighting system includes a first weight (i.e., visual semantic similarity). The second weight (i.e., the weight of real-time user interaction) ) and the third weight (i.e., the time decay factor) ).

[0062] Here, visual semantic similarity It can calculate cross-modal similarity between images and text based on CLIP embedding. The specific calculation formula is as follows: (3) Here, user real-time interaction weight It can dynamically adjust the copy based on the current user's historical interaction data. The specific calculation formula is as follows: (4) Among them, t current The current viewing time of the text, t avg m represents the average viewing time for similar copywriting, and m represents the number of likes. avg This represents the average number of likes.

[0063] Here, the time decay factor It takes into account the temporal changes in user preferences. The specific calculation formula is as follows: (5) Where T is the current time, t interact The time of the user's last interaction. This is the attenuation coefficient.

[0064] Step A15: Based on the first weight, the second weight, and the third weight, calculate the similarity between the texts operated by the first user and the candidate text set to obtain the reference texts.

[0065] Among them, the similarity between the reference text and the text operated by the first user is greater than the first threshold.

[0066] The formula for calculating the similarity between two documents in this application is as follows: (6) Where N(i) represents the set of users who interact with copy i; N(j) represents the set of users who interact with copy j.

[0067] When recommending content, this application combines real-time user interaction data (browsing time, likes) with a time decay factor to improve the algorithm's response speed to the minute level.

[0068] This application introduces visual semantic similarity weights (based on cross-modal embedding), real-time interaction weights (dependent on dynamic data such as browsing time and number of likes), and time decay factors (to capture the temporal changes in user preferences), enabling the recommendation algorithm to upgrade from "lagging response" to "real-time adaptation." This solves the core problem in related technologies that recommendation strategies cannot follow changes in user interests and improves the recommendation effect in dynamic scenarios.

[0069] In some embodiments, step A2, obtaining the fused features of the first type of features, the second type of features, and the third type of features, includes the following steps: Step A21: Obtain the heterogeneous graph neural network.

[0070] Among them, the heterogeneous graph neural network includes a heterogeneous graph with three nodes: user, image, and text.

[0071] This application will first construct a heterogeneous graph containing three types of nodes: "user-image-text". The edge types in the heterogeneous graph include user-image, image-text, and user-text. Among them, the edge type corresponding to user-image is the image uploaded / viewed by the user; the edge type corresponding to image-text is the image-text pair that has been successfully matched in history; and the edge type corresponding to user-text is the text that the user has selected / interacted with.

[0072] This application uses a three-way graph model of "user-image-copy" to capture third-order associations (such as the implicit similarity between "user A-snow mountain image-healing copy" and "user B-forest image-artistic copy").

[0073] Step A22: Input the first type of features, the second type of features, and the third type of features into the heterogeneous graph neural network to obtain the output fused features.

[0074] This application improves the LightGCN model in related technologies to obtain an improved LightGCN model; multiple features are fused through the cross-modal gating mechanism in the improved LightGCN model to obtain fused features.

[0075] For the improved LightGCN model: The message passing mechanism can be characterized by the following formula: (7) in, Let N(v) be the feature vector of node v at layer k, and N(v) be the set of neighbors of node v. Let be the feature vector of node u at layer k, N(u) be the set of neighbors of node u, and w(k) be the trainable weight matrix.

[0076] For the cross-modal feature fusion scheme, after each layer of message passing, a gating mechanism is used to fuse visual features, user behavior features, and text semantic features: (8) in, For the Sigmoid function, For GNN output features, Visual features; is the text feature; w1 is the preset weight parameter.

[0077] This application aggregates high-order connectivity relationships among three types of nodes (such as the third-order association of "user-image-text") through a heterogeneous graph neural network, and fuses cross-modal features through a gating mechanism. It overcomes the limitations of traditional matrix factorization, uncovers implicit semantic associations (such as the scene association of "coffee image + good morning text"), solves the problem of insufficient diversity in recommendation results, and expands the recommendation boundary.

[0078] In some embodiments, step A4, determining the attention score for each reference text based on the first type of features, the second type of features, and the third type of features, includes the following steps: Step A41: Calculate the first attention weight of the first type of feature and the second type of feature corresponding to the reference text.

[0079] Step A42: Based on the third type of features, determine the second attention weight that represents the preferences of the first user.

[0080] This application employs a multi-dimensional attention mechanism to determine the attention score for each reference text, specifically: For image-text semantic alignment (calculating the attention weights between visual features and text semantics), the formula is: (9) Among them, W a The weight matrix is ​​a learnable weight matrix; For the visual features of image i, The semantic features of copy j.

[0081] User preference attention (combining the relevance of historical user preferences with current recommendations), the formula is: (10) in, Create feature vectors for user profiles.

[0082] Step A43: Calculate the popularity score of the reference text among the second user.

[0083] Among them, the similarity between the attribute information of the second user and the attribute information of the first user is greater than the second threshold.

[0084] In some embodiments, first attribute information of a first user and second attribute information of multiple candidate users are obtained; similarity is calculated between the first attribute information and the multiple second attribute information to obtain a third user; and a second user is determined from the third user based on a second type of feature.

[0085] This application employs User Collaborative Filtering (U-CF+) optimization to determine the second user, specifically: This application introduces user attribute similarity (gender, age, historical style preference) as an auxiliary factor: (11) in, This represents gender similarity (same gender = 1, otherwise = 0). The similarity is based on age difference (1 for an age difference ≤ 5 years, decreasing by 0.2 for every 5 years increase). The cosine similarity is the historical style preference; w1, w2, and w3 are pre-set weights.

[0086] Furthermore, based on formula (12), the similarity between the two users is calculated.

[0087] (12) Step A44: Determine the attention score for each reference copy based on the first attention weight, the second attention weight, and the popularity score.

[0088] In this embodiment of the application, the attention score for each reference text is determined based on the following formula: (13) Among them, popularity is the score of the copy's popularity among similar user groups; , and This parameter is assigned a pre-set weight.

[0089] This application uses an attention mechanism to comprehensively calculate the semantic alignment between image and text, the fit of user preferences, and the popularity of the text to generate a comprehensive score. Compared with related technologies that rely on only a single dimension (such as image tag matching), this application uses an attention mechanism to comprehensively calculate the semantic alignment between image and text, the fit of user preferences, and the popularity of the text to generate a comprehensive score.

[0090] In some embodiments, the scheme to be protected by this application includes the following steps: Obtain the interaction data between the first user and the recommended text; based on the interaction data, adjust the recommended text corresponding to the target image.

[0091] This application can employ reinforcement learning for closed-loop processing, specifically: The state space is set as follows: (14) The motion space is set as follows: (15) The reward function is set as follows: (16) Among them, W1, W2, W3 and W4 are pre-set weight parameters.

[0092] Policy update: Update the network parameters of the recommendation policy using the Proximal Policy Optimization (PPO) algorithm.

[0093] Incremental model update: Real-time data caching: User interaction data is stored in a sliding window cache (the most recent 1000 records), triggering model fine-tuning once per hour; Dynamic parameter adjustment: Multimodal fusion weights W1, W2, and W3 are optimized through a genetic algorithm to maximize recommendation accuracy.

[0094] By continuously optimizing the recommendation strategy through the PPO algorithm, the drop in recommendation accuracy is reduced by more than 50% in scenarios with sudden changes in user preferences (such as switching from "photography" to "food").

[0095] This application uses user interaction behaviors (clicks, edits, and dwell times) as reward signals for reinforcement learning, and dynamically adjusts the recommendation strategy through the PPO algorithm. This enables the recommendation model to self-evolve, allowing the system to quickly adapt to scenarios with sudden changes in user preferences (such as a shift from travel to pet-related topics), thus solving the problem of lagging model iteration in traditional solutions.

[0096] This application realizes personalized recommendation through multimodal fusion. It uses the CLIP model to achieve visual-semantic cross-modal mapping, models the temporal features of user behavior through LSTM, and dynamically adjusts the weights of each modality by combining an attention mechanism to generate copywriting recommendations that fit the user's personality.

[0097] This application enhances the dynamic adaptability of recommendation algorithms by introducing dynamic weight factors (such as browsing time and number of likes) to improve collaborative filtering algorithms. Combined with reinforcement learning closed loop, the model parameters are updated based on real-time user interaction feedback, enabling the recommendation strategy to evolve in sync with the user's current interests, thereby improving dynamic adaptability and tracking changes in user preferences in real time.

[0098] This application improves the semantic matching accuracy between images and text by constructing a "user-image-text" association graph through a graph neural network, aggregating high-order semantic relationships using a graph neural network (GNN), and introducing a sentiment analysis module to model the sentiment dimension of image visual features and text, thereby achieving multi-dimensional semantic matching.

[0099] This application mines implicit associations to enhance recommendation diversity. It captures implicit preference patterns in user groups (such as the scene association of "coffee image + good morning text") through a three-way graph neural network, and combines it with a dynamic collaborative filtering algorithm to recommend texts with similar semantic clusters, rather than being limited to direct label matching. This significantly enhances the diversity of recommendation results and mines implicit semantic associations.

[0100] This application employs a four-layer technical architecture for text recommendation, which includes multimodal feature fusion, dynamic collaborative filtering, semantic enhancement ranking, and real-time feedback optimization. Figure 2 This is a flowchart illustrating a method for determining a document according to an embodiment of this application. The method includes the following steps: Figure 2 As shown, Step 201: User uploads image; Step 202: Based on the multi-model feature extraction layer, perform visual feature extraction, user behavior feature extraction, and text semantic feature preprocessing.

[0101] Step 203: Update the model parameters based on the first feature obtained from visual feature extraction, the second feature obtained from user behavior feature extraction, and the third feature obtained from preprocessing text semantic features.

[0102] Step 204: Optimize based on graph structure.

[0103] Step 205: Generate recommendation results based on the attention ranking layer.

[0104] Step 206: Combine the recommendation results with user interaction feedback, and update the model parameters based on the feedback.

[0105] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0106] For example, in a scenario where a user uploads an image of a "coffee cup under the sunset," the following operations are performed: 1. Feature extraction: including visual features and user behavior features. ResNet extracts color (warm tones) and texture (wooden desktop). Faster R-CNN identifies "coffee cup" and "sunset". CLIP generates semantic embeddings ("leisure", "afternoon tea", "healing"). LSTM analysis of historical records shows that users have recently frequently used "coffee + literary style" copywriting. The average time to browse similar copywriting is 8 seconds (higher than the average of 5 seconds).

[0107] 2. Collaborative Filtering: D-ItemCF calculates text similarity and introduces visual semantic similarity (0.85), interaction weight (1.6), and time decay factor (0.9); U-CF+ finds 3 similar users, all of whom prefer "coffee + literary style" text.

[0108] 3. Graph Neural Network: In the three-dimensional graph, the image and the "literary copywriting" node are connected through three paths. LightGCN aggregates and outputs enhanced features; the gating mechanism integrates visual features (warm color tone) and copywriting semantics ("slow time") to generate a matching score.

[0109] 4. Ranking and Feedback: The attention mechanism scored 0.92, and the recommended copy was "Take coffee as poetry and turn the sunset into sugar". When the user clicked and edited the keyword "poetry" → "story", the reinforcement learning module recorded the feedback and updated the keyword weight of the copy generator.

[0110] Embodiments of this application provide a document determination apparatus, which can be used to implement... Figure 1 The method for determining the text provided in the corresponding embodiment is referred to Figure 3 As shown, the copy-determining device 300 includes: The acquisition unit 301 is used to acquire a target image related to the first user, a set of candidate texts corresponding to the target image, and behavioral data of the first user in a first time period. The processing unit 302 is used to extract features from the target image to obtain a first type of visually related features; the first type of features includes color features, texture features, object category features, scene features, and emotion category features; The processing unit 302 is also used to preprocess the candidate copy set to obtain a second type of features related to the semantics of the copy; wherein, the second type of features includes the latent semantic features corresponding to each copy and the semantic features obtained by encoding each copy. Processing unit 302 is further configured to determine a third type of feature related to the copywriting operation behavior of the first user from the behavioral data; The processing unit 302 is also used to determine the recommended text for the target image based on the first type of features, the second type of features, and the third type of features.

[0111] In other embodiments of this application, the processing unit 302 is used to filter out reference texts from the candidate text set based on the first type of features, the second type of features and the third type of features; The acquisition unit 301 is used to acquire the fused features of the first type of features, the second type of features, and the third type of features; Processing unit 302 is used to determine the matching score between the fusion feature and each reference text; Processing unit 302 is used to determine the attention score of each reference text based on the first type of features, the second type of features, and the third type of features; Processing unit 302 is used to determine recommended copy for the target image from the second copy based on the matching score and attention score.

[0112] In other embodiments of this application, the processing unit 302 is used to map the first type of features and the second type of features into a high-dimensional representation; Processing unit 302 is used to determine a first weight based on a high-dimensional representation; wherein the first weight corresponds to visual semantic similarity; Processing unit 302 is used to determine a second weight based on the third type of feature; wherein the second weight is the weight corresponding to the current interaction behavior of the first user; Processing unit 302 is used to determine a third weight based on behavioral data; wherein the third weight represents the weight of the temporal change of the first user's preference; The processing unit 302 is used to calculate the similarity between the text operated by the first user and the candidate text set based on the first weight, the second weight and the third weight, to obtain the reference text; the similarity between the reference text and the text operated by the first user is greater than the first threshold.

[0113] In other embodiments of this application, the acquisition unit 301 is used to acquire a heterogeneous graph neural network; wherein, the heterogeneous graph neural network includes a heterogeneous graph with user, image and text as nodes; The processing unit 302 is used to input the first type of features, the second type of features and the third type of features into the heterogeneous graph neural network to obtain the output fused features.

[0114] In other embodiments of this application, the processing unit 302 is used to calculate the first attention weight of the first type of feature and the second type of feature corresponding to the reference text; Processing unit 302 is used to determine a second attention weight representing the preferences of the first user based on the third type of features; Processing unit 302 is used to calculate the popularity score of the reference text among the second user; the similarity between the attribute information of the second user and the attribute information of the first user is greater than a second threshold. Processing unit 302 is used to determine the attention score of each reference copy based on the first attention weight, the second attention weight, and the popularity score.

[0115] In other embodiments of this application, the acquisition unit 301 is used to acquire the first attribute information of the first user and the second attribute information of multiple candidate users; Processing unit 302 is used to calculate the similarity between the first attribute information and multiple second attribute information to obtain the third user; Processing unit 302 is used to determine a second user from a third user based on a second type of feature.

[0116] In other embodiments of this application, the acquisition unit 301 is used to acquire interaction data between the first user and the recommended text; The processing unit 302 is used to adjust the recommended text corresponding to the target image based on the interaction data.

[0117] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0118] It should be noted that, in the embodiments of this application, if the above-mentioned method for determining the recommended text is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0119] Figure 4 This is a schematic structural diagram of an electronic device provided in an embodiment of this application. The electronic device may be a document determination device. Figure 4 The illustrated electronic device 400 includes a processor 410, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0120] Optionally, such as Figure 4 As shown, the electronic device 400 may further include a memory 420. The processor 410 can retrieve and run computer programs from the memory 420 to implement the methods described in the embodiments of this application.

[0121] The memory 420 can be a separate device independent of the processor 410, or it can be integrated into the processor 410.

[0122] Optionally, such as Figure 4 As shown, the electronic device 400 may also include a transceiver 430, which the processor 410 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0123] The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include an antenna, and the number of antennas may be one or more.

[0124] Optionally, the electronic device 400 may specifically be a document determination device in the embodiments of this application, and the electronic device 400 may implement the corresponding processes implemented by the document determination device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0125] This application also provides a computer program product, including a computer program that can be executed by the processor 410 of the electronic device 400 to perform the steps described in any of the foregoing methods.

[0126] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0127] As one embodiment, the processor may include one or more general-purpose central processing units (CPUs). Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used for processing data (e.g., executing instructions).

[0128] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be ROM, Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory. The volatile memory can be Random Access Memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0129] This application also provides a computer-readable storage medium for storing computer programs.

[0130] Optionally, the computer-readable storage medium can be applied to the document determination device / electronic device in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the document determination device / electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0131] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0132] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0133] The foregoing has provided a detailed description of the document determination method, document determination device, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0134] It should be understood that the phrases "an embodiment," "an embodiment," "an embodiment of this application," "the foregoing embodiment," "some implementations," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, the phrases "an embodiment," "an embodiment," "an embodiment of this application," "the foregoing embodiment," "some implementations," or "some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0135] Unless otherwise specified, any step performed by the document determining device / electronic device in the embodiments of this application may be performed by the processor of the document determining device / electronic device. Unless otherwise specified, the embodiments of this application do not limit the order in which the document determining device / electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0137] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0138] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0139] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0140] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0141] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0142] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0143] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0144] The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0145] It should be noted that in the various embodiments involved in this application, all steps or some steps may be performed, as long as a complete technical solution can be formed.

[0146] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining copywriting, characterized in that, The method includes: Acquire a target image related to the first user, a set of candidate texts corresponding to the target image, and behavioral data of the first user within a first time period; Feature extraction is performed on the target image to obtain a first type of visually related features; the first type of features includes color features, texture features, object category features, scene features, and emotion category features; The candidate text set is preprocessed to obtain a second type of features related to the semantics of the text; wherein, the second type of features includes the latent semantic features corresponding to each text and the semantic features obtained by encoding each text. A third type of feature related to the first user's text manipulation behavior is determined from the behavioral data; Based on the first type of features, the second type of features, and the third type of features, the recommended text for the target image is determined.

2. The method according to claim 1, characterized in that, The step of determining the recommended text for the target image based on the first type of features, the second type of features, and the third type of features includes: Based on the first type of features, the second type of features, and the third type of features, reference texts are filtered out from the candidate text set; Obtain the fused features of the first type of features, the second type of features, and the third type of features; Determine the matching score between the fusion feature and each reference text; Based on the first type of features, the second type of features, and the third type of features, the attention score of each reference text is determined; Based on the matching score and the attention score, a recommended text for the target image is determined from the second text.

3. The method according to claim 2, characterized in that, The process of filtering reference texts from the candidate text set based on the first type of features, the second type of features, and the third type of features includes: Map the first type of features and the second type of features into a high-dimensional representation; Based on the high-dimensional representation, a first weight is determined; wherein, the first weight corresponds to visual semantic similarity; Based on the third type of feature, a second weight is determined; wherein, the second weight is the weight corresponding to the current interaction behavior of the first user; Based on the behavioral data, a third weight is determined; wherein, the third weight represents the weight of the temporal changes in the first user's preferences; Based on the first weight, the second weight, and the third weight, the similarity between the text operated by the first user and the candidate text set is calculated to obtain the reference text; the similarity between the reference text and the text operated by the first user is greater than the first threshold.

4. The method according to claim 2, characterized in that, The process of obtaining the fused features of the first type of features, the second type of features, and the third type of features includes: Obtain a heterogeneous graph neural network; wherein the heterogeneous graph neural network includes a heterogeneous graph with user, image and text as nodes; The first type of feature, the second type of feature, and the third type of feature are input into the heterogeneous graph neural network to obtain the output fused features.

5. The method according to claim 2, characterized in that, The process of determining the attention score for each reference text based on the first type of features, the second type of features, and the third type of features includes: Calculate the first attention weight of the first type of feature and the second type of feature corresponding to the reference text; Based on the third type of feature, a second attention weight representing the preferences of the first user is determined; Calculate the popularity score of the reference text among the second user; the similarity between the attribute information of the second user and the attribute information of the first user is greater than a second threshold; An attention score is determined for each reference copy based on the first attention weight, the second attention weight, and the popularity score.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the first attribute information of the first user and the second attribute information of multiple candidate users; The similarity between the first attribute information and multiple second attribute information is calculated to obtain the third user; Based on the second type of features, the second user is determined from the third user.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the interaction data between the first user and the recommended text; Based on the interaction data, the recommended text corresponding to the target image is adjusted.

8. A document determination device, characterized in that, The device includes: The acquisition unit is used to acquire a target image related to the first user, a set of candidate texts corresponding to the target image, and behavioral data of the first user within a first time period. The processing unit is used to extract features from the target image to obtain a first type of visually related features; the first type of features includes color features, texture features, object category features, scene features, and emotion category features; The processing unit is further configured to preprocess the candidate copy set to obtain a second type of features related to the semantics of the copy; wherein, the second type of features includes the latent semantic features corresponding to each copy and the semantic features obtained by encoding each copy. The processing unit is further configured to determine a third type of feature related to the first user's text operation behavior from the behavioral data; The processing unit is further configured to determine recommended text for the target image based on the first type of features, the second type of features, and the third type of features.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the document determination method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the document determination method according to any one of claims 1 to 7.

Images