Product marketing text generation method and device based on multi-modal feature fusion

Abstract

The embodiment of the invention provides a product marketing text generation method and device based on multi-modal feature fusion, and relates to the field of artificial intelligence. The method comprises the steps that customer information features and product information features are determined, the customer information features are determined based on basic data, transaction information and browsing information of customers, the product information features are determined based on product promotion information, and the product promotion information comprises text information and image information; carrying out feature fusion on the customer information features and the product information features by adopting a self-attention mechanism, and determining multi-modal features; the multi-modal features are input into a pre-trained text coding layer, a product marketing text of the customer is generated, and the text coding layer is used for generating the product marketing text based on the input multi-modal features. According to the method provided by the invention, the large-scale precise generation of the product marketing text of the customer is realized, and the experience of the customer is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and device for generating product marketing text based on multimodal feature fusion. Background Art

[0002] With the continuous development of Internet technology, people's lives are becoming more and more digital, and e-marketing has emerged. Through computer communication technology, digital interactive media, modern communication technology and other Internet technologies, product marketing, promotion and sales are carried out to achieve marketing goals. Product marketing texts can help customers understand products and are an important means of successfully selling products.

[0003] At present, the digital operations of banks mainly rely on manual writing models. Writing staff write product marketing texts by understanding product information and customer needs, and combining market research and marketing goals.

[0004] However, manual writing requires high writing literacy from the writer, and it is difficult to achieve large-scale personalization, resulting in low work efficiency. Therefore, how to generate precise product marketing texts for different customers has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a method and device for generating product marketing text based on multimodal feature fusion, which is used to solve the technical problem that it is difficult to generate product marketing texts for customers in a large-scale and personalized manner.

[0006] In a first aspect, the present application provides a method for generating product marketing text based on multimodal feature fusion, comprising:

[0007] Determine customer information features and product information features, wherein the customer information features are determined based on the customer's basic data, transaction information, and browsing information, and the product information features are determined based on product recommendation information, wherein the product recommendation information includes text information and image information;

[0008] A self-attention mechanism is used to fuse the customer information features and the product information features to determine multimodal features;

[0009] The multimodal features are input into a pre-trained text encoding layer to generate a product marketing text for the customer, wherein the text encoding layer is used to generate the product marketing text based on the input multimodal features.

[0010] In a second aspect, the present application provides a product marketing text generation device based on multimodal feature fusion, comprising:

[0011] A determination module, used to determine customer information features and product information features, wherein the customer information features are determined based on the customer's basic data, transaction information and browsing information, and the product information features are determined based on product recommendation information, wherein the product recommendation information includes text information and image information;

[0012] A fusion module, used to fuse the customer information features and the product information features using a self-attention mechanism to determine a multimodal feature;

[0013] A generation module is used to input the multimodal features into a pre-trained text encoding layer to generate a product marketing text for the customer, and the text encoding layer is used to generate the product marketing text based on the input multimodal features.

[0014] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect.

[0018] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0019] The product marketing text generation method and device based on multimodal feature fusion provided in the present application obtain multimodal features by fusing customer information features determined by the customer's basic data, transaction information and browsing information, and product information features determined based on product promotion information. The multimodal features are input into a pre-trained text encoding layer to generate the customer's product marketing text, thereby realizing large-scale and precise generation of customer's product marketing text and improving the customer experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] Figure 1 A process diagram of a method for generating product marketing text based on multimodal feature fusion provided in an embodiment of the present application Figure 1 ;

[0022] Figure 2 A process diagram of a method for generating product marketing text based on multimodal feature fusion provided in an embodiment of the present application Figure 2 ;

[0023] Figure 3 A schematic diagram of the structure of a product marketing text generation device based on multimodal feature fusion provided in this application;

[0024] Figure 4 A schematic diagram of the structure of a product marketing text generation device based on multimodal feature fusion provided in an embodiment of the present application.

[0025] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0027] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, products or devices.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0031] It should be noted that the product marketing text generation method, device, equipment, and medium based on multimodal feature fusion provided in this application can be used in the field of artificial intelligence, and can also be used in any field other than artificial intelligence. The application field of the product marketing text generation method, device, equipment, and medium based on multimodal feature fusion in this application is not limited.

[0032] With the continuous development of Internet technology, people's lives are becoming more and more digital, and e-marketing has emerged. Through computer communication technology, digital interactive media, modern communication technology and other Internet technologies, product marketing, promotion and sales are carried out to achieve marketing goals. Product marketing texts can help customers understand products and are an important means of successfully selling products.

[0033] At present, the digital operations of banks mainly rely on manual writing models. Before writing, staff need to first understand customer needs, customer preferences, etc. They also need to understand product information and write product marketing texts based on market research and marketing goals.

[0034] However, the interests of Internet users are becoming increasingly fragmented and personalized, and the traditional digital operation of uniformly pushing homogeneous product marketing texts has become difficult to work; the manual writing method has high requirements on the writing skills of the writers, and it is difficult to achieve large-scale personalization, resulting in low work efficiency; therefore, how to generate precise product marketing texts for different customers has become a technical problem that needs to be solved urgently.

[0035] The product marketing text generation method, device, equipment, and medium based on multimodal feature fusion provided in the present application fuse the customer information features and product information features of each customer, and then generate the product marketing text corresponding to each customer based on the fused multimodal features, thereby avoiding the technical problem of low product marketing efficiency caused by uniformly pushing homogeneous product marketing texts and difficulty in achieving large-scale personalized and accurate push of marketing texts.

[0036] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0037] Figure 1 Schematic diagram of the process of generating product marketing text based on multimodal feature fusion provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes:

[0038] S101. Determine customer information characteristics and product information characteristics;

[0039] Among them, the customer information features are determined based on the customer's basic data, transaction information and browsing information, and the product information features are determined based on the product promotion information, wherein the product promotion information includes text information and image information.

[0040] Basic customer data refers to data about customers and their related characteristics and relationships, including basic information and key attributes of customers, which are used to identify customers, understand customer needs, provide personalized services, and conduct market analysis; transaction information refers to various information directly related to customer economic behavior; browsing information refers to the traces left by customers when browsing web pages. Product recommendation information is used to indicate a series of information and materials about products.

[0041] According to the basic data, transaction information, browsing information and product promotion information of customers, basic data includes personal information, account information, etc. For example, personal information can be education, home address, contact information, age, etc.; account information can be asset status, whether to hold a credit card, whether to apply for a credit product, etc.; transaction information can be, for example, transaction time, transaction amount, payee / remitter information, transfer method (transfer / collection), cumulative number of transactions on the day, cumulative transaction amount on the day, number of transactions on the month, cumulative transaction amount on the month, etc.; browsing information can be, for example, the duration, frequency, browsing content, stay interface, reasons for exiting browsing, etc. of browsing each product. In product promotion information, text information can be, for example, product description, features and advantages, customer evaluation, etc., and image information can be, for example, marketing brochures, etc. For example, features can be extracted through methods such as convolutional neural networks and long short-term memory networks to determine customer information features and product information features. By obtaining multiple types of data, the analysis dimension is enriched, and the user's behavioral characteristics are better analyzed to provide personalized product marketing texts for the customer.

[0042] In a possible implementation, the specific process of determining the customer information features is described in detail: obtaining the customer's basic data, transaction data, and browsing data; inputting the transaction data and browsing data into a pre-trained information extraction network to obtain the customer's transaction information and browsing information; and using the customer's basic data, transaction information, and browsing information as customer information features.

[0043] Among them, the information extraction network is trained based on the Long Short-Term Memory (LSTM) network.

[0044] Obtaining basic customer data , transaction data and browse data The transaction data and browsing data are input into the information extraction network pre-trained based on the long short-term memory network to extract information and obtain the customer's transaction information h y and browse information z .

[0045] The specific formula is as follows:

[0046]

[0047] Then, through splicing, the transaction information h y and browse information z Embedded into the basic data X, we get the customer information feature I. The formula is as follows:

[0048]

[0049] By inputting transaction data and browsing data into the information extraction network, transaction information and browsing information are obtained, and customer information features are obtained after being spliced ​​with basic data. The long-term dependencies in the sequence data are effectively handled, the accuracy of customer information features is improved, and the user's behavioral characteristics are better analyzed to provide personalized product marketing text for the customer.

[0050] S102, using a self-attention mechanism to fuse the customer information features and the product information features to determine a multimodal feature;

[0051] Among them, the self-attention mechanism is used to capture complex dependencies within the sequence; feature fusion is to combine data from different sources or different types of features; multimodal features are vectors that integrate customer information features and product information features.

[0052] In a possible implementation, the above-mentioned self-attention mechanism is used to fuse the customer information features and the product information features to determine the multimodal features. The following is described in detail: the corresponding query, key, and value are calculated according to the parameter matrices corresponding to the query, key, and value, and the customer information features and the product information features; the feature dimension parameters corresponding to the customer information features are determined, and the multimodal features are determined according to the feature dimension parameters and the query, key, and value.

[0053] Among them, the parameter matrix is ​​used to indicate the trainable parameter matrix; the feature dimension parameter is used to indicate the number of feature dimensions.

[0054] The query, key, and value calculation process is as follows:

[0055]

[0056] Where Q is the query, W is Q is the parameter matrix corresponding to the query; K is the key, W K is the parameter matrix corresponding to the key; V is the value, W V is the parameter matrix corresponding to the value; ReLU is the activation function; S is the product information feature; I is the customer information feature.

[0057] The multimodal features are calculated using the following formula:

[0058]

[0059] Among them, U is the multimodal feature; softmax is the softmax function; d k is the feature dimension parameter.

[0060] By performing feature fusion through the self-attention mechanism, more comprehensive and complete multimodal features are determined, thereby improving the accuracy of product marketing text generation for customers.

[0061] S103: Input the multimodal features into a pre-trained text encoding layer to generate a product marketing text for the customer.

[0062] Among them, the text encoding layer is used to generate product marketing text based on the input multimodal features.

[0063] In one possible implementation, the text encoding layer includes a decoder, which describes in detail inputting the multimodal features into a pre-trained text encoding layer to generate the product marketing text of the customer:

[0064] Initializing the decoder to determine the initial state of the decoder;

[0065] Among them, the initial state is used to indicate the initial time step information of the encoder.

[0066] Use LSTM as the decoder, initialize it through LSTM, and get the initial state S of the decoder t .

[0067] Inputting the initial state and the multimodal features into the pre-trained text encoding layer to determine the distribution probabilities corresponding to a plurality of words in the vocabulary;

[0068] The vocabulary is obtained when training the text encoding layer; the distribution probability is used to indicate the possibility of selecting each word at the current time step.

[0069] In a possible implementation, the above-mentioned inputting the initial state and the multimodal features into the pre-trained text encoding layer to determine the distribution probabilities corresponding to multiple words in the vocabulary is described in detail: the initial state and the multimodal features are input into the first linear layer to extract intermediate features; the intermediate features are input into the second linear layer to obtain the original score corresponding to each word in the vocabulary; the original score corresponding to each word is converted to determine the distribution probabilities corresponding to multiple words in the vocabulary.

[0070] The formula is as follows:

[0071]

[0072] Among them, p v is the distribution probability of the vocabulary; softmax is the softmax function, which is used to convert the original score into a distribution probability; V1 and b1 are the trainable parameters of the first linear layer; V2 and b2 are the trainable parameters of the second layer.

[0073] By determining the distribution probabilities corresponding to multiple words in the vocabulary, the accuracy of product marketing text generation is improved, and the user experience is enhanced.

[0074] At least one target word is determined based on the distribution probabilities corresponding to the plurality of words, and a product marketing text for the customer is generated based on the at least one target word.

[0075] At least one target word is determined based on the distribution probabilities corresponding to the multiple words. For example, a word with the highest probability may be determined as the target word, and a product marketing text for the customer is generated based on the at least one target word.

[0076] Generate customer product marketing texts through multimodal features, personalized product marketing texts, improve customer experience and stimulate customer interest.

[0077] The method for generating product marketing text based on multimodal feature fusion provided in this embodiment fuses customer information features with product information features, and then inputs the fused multimodal features into a pre-trained text encoding layer, analyzes customer needs and interests based on customer information features, and analyzes product attribute features based on product information features, mines out attribute features in the product that meet customer interests, and generates product marketing text corresponding to each customer, thereby improving customer experience and realizing large-scale personalized and precise generation of customer product marketing texts. Precision marketing achieves higher returns at a lower cost, improves the overall marketing return rate, and improves the work efficiency of staff.

[0078] Figure 2 Schematic diagram of the process of generating product marketing text based on multimodal feature fusion provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, the product information features include text features and image features. Figure 1 Based on the embodiment, the process of determining product information features is described in detail:

[0079] S201, obtaining text information of a product;

[0080] Get text information from product promotion information, including product description, features and benefits, customer reviews, etc.

[0081] S202, splitting the text information to obtain multiple clauses, and determining the word segmentation vectors corresponding to multiple word segmentations contained in each clause;

[0082] Split the text information, generate pre-trained word vector files for multiple word segments in the text information through unsupervised methods, and map the pre-trained word vector files into a word vector matrix. Map each word segment in the text information to the word vector matrix to obtain the word segment vector corresponding to each word segment. ij .

[0083] For example, the text information T consists of N clauses, each of which consists of M words. ;Right now , where x i Indicates clause i, c ij Represents participle j in clause i.

[0084] S203: for any one of the multiple clauses, determine first contextual semantic information corresponding to each word in the clause according to the word vector of each word in the clause;

[0085] The word vector e ij Input into a two-layer bidirectional long short-term memory network to obtain the first context semantic information of the word segmentation corresponding to the word segmentation vector in the text information.

[0086] The specific calculation process is as follows:

[0087]

[0088]

[0089]

[0090] in, is the long short-term memory network, is the vector to the LSTM network, is the forward hidden layer state obtained after two layers of forward long short-term memory network, is the reverse hidden state obtained after the word vector passes through two layers of reverse long short-term memory network, It is the first context semantic information of the current word segmentation, which is the output of a two-layer bidirectional long short-term memory network.

[0091] S204, calculating the segmentation weight of each segmentation in the clause according to the first contextual semantic information, and determining the clause feature according to the segmentation weight and the first contextual semantic information;

[0092] The attention mechanism is used to calculate the weight of each word in the clause where the word is located. The formula is as follows:

[0093]

[0094]

[0095] Among them, W w and b w is a trainable parameter, q w is a randomly initialized feature vector, which will be iteratively updated during the training process; ij The weight of the jth word in the i-th clause; tanh is the activation function, u ij is the candidate value vector.

[0096] Then use the weighted sum method to obtain the vector representation of the text clause based on the word segmentation vector. The calculation formula is as follows:

[0097]

[0098]

[0099] Among them, s h Features of the clause.

[0100] S205, determining the text feature according to the clause features of the multiple clauses and the second context semantic information corresponding to each clause;

[0101] The clause features of multiple clauses are input into a two-layer bidirectional long short-term memory network to obtain the second context semantic information corresponding to each clause. Then, based on the second context semantic information and the attention mechanism, the weight of each clause in the text information is calculated. Based on the second context semantic information and the clause weight, the text feature H is determined.

[0102] The calculation process for determining text features is similar to the calculation process for determining clause features and will not be described in detail here.

[0103] S206, determining initial image information of the product;

[0104] Determine the initial image information of the product, such as a marketing brochure.

[0105] S207, determining coordinate information of the target object according to the initial image information;

[0106] The target object is used to indicate the image information of the product; and the coordinate information is usually used to indicate the position of the target object in the image.

[0107] The initial image information is input into the target detection tool to obtain the position of the target object in the initial image information.

[0108] S208, performing cropping processing on the initial image information of the product according to the coordinate information of the target object;

[0109] The initial image information is first cropped. The cropped image only contains the target object, which can reduce the interference of irrelevant features and make feature extraction more focused and efficient, thereby reducing the consumption of computing resources, improving processing speed, and facilitating accurate feature extraction and analysis.

[0110] S209: extracting features from the cropped initial image information according to the coordinate information of the target object to obtain image features of the product.

[0111] By using the image feature extraction tool, feature extraction is performed on the cropped initial image information according to the coordinate information of the target object to obtain the image feature P of the product.

[0112] In a possible implementation, after extracting the text feature H and the image feature P, the text feature and the image feature may be fused by splicing to obtain the product information feature S.

[0113] The expression is as follows:

[0114]

[0115] The product marketing text generation method based on multimodal feature fusion provided in the embodiment of the present application obtains multiple clauses and word segments in each clause by splitting the text information of the product, and analyzes the context information of the word segments and clauses to extract features and determine text features, thereby improving the correlation between word segmentation features and clause features, capturing implicit information, and improving the completeness and accuracy of text features; by cropping the image information and then extracting the image features according to the coordinate information of the target object, it is ensured that the feature extraction process is more focused on the key parts of the image information, thereby improving the accuracy of image extraction.

[0116] Figure 3 A schematic diagram of the structure of a product marketing text generation device based on multimodal feature fusion provided in this application, such as Figure 3 As shown, the product marketing text generation device 30 based on multimodal feature fusion provided in this embodiment includes:

[0117] Determination module 301, used to determine customer information features and product information features, the customer information features are determined based on the customer's basic data, transaction information and browsing information, the product information features are determined based on product recommendation information, and the product recommendation information includes text information and image information;

[0118] A fusion module 302 is used to fuse customer information features and product information features using a self-attention mechanism to determine multimodal features;

[0119] The generation module 303 is used to input the multimodal features into a pre-trained text encoding layer to generate a product marketing text for the customer. The text encoding layer is used to generate the product marketing text based on the input multimodal features.

[0120] In a possible implementation, the fusion module 302 is also used to calculate the corresponding query, key, and value according to the parameter matrices corresponding to the query, key, and value, as well as the customer information features and product information features; determine the feature dimension parameters corresponding to the customer information features, and determine the multimodal features according to the feature dimension parameters and the query, key, and value.

[0121] In a possible implementation, the text encoding layer includes a decoder, and the generation module 303 is further used to initialize the decoder and determine the initial state of the decoder, where the initial state is used to indicate the initial time step information of the encoder;

[0122] Input the initial state and multimodal features into a pre-trained text encoding layer to determine the distribution probabilities corresponding to multiple words in the vocabulary, where the vocabulary is obtained when training the text encoding layer;

[0123] At least one target word is determined based on the distribution probabilities corresponding to the plurality of words, and a product marketing text for the customer is generated based on the at least one target word.

[0124] In a possible implementation, the determination module 301 is further used to obtain text information of the product;

[0125] Split the text information into multiple clauses, and determine the word vectors corresponding to the multiple words contained in each clause;

[0126] For any one of the multiple clauses, determine first context semantic information corresponding to each word in the clause according to the word vector of each word in the clause;

[0127] Calculate the segmentation weight of each segmentation in the clause according to the first context semantic information, and determine the clause feature according to the segmentation weight and the first context semantic information;

[0128] The text feature is determined according to the clause features of the multiple clauses and the second context semantic information corresponding to each clause.

[0129] In a possible implementation, the determination module 301 is further used to determine initial image information of the product;

[0130] Determine the coordinate information of the target object according to the initial image information, where the target object is used to indicate the image information of the product;

[0131] According to the coordinate information of the target object, the initial image information of the product is cropped;

[0132] According to the coordinate information of the target object, feature extraction is performed on the cropped initial image information to obtain the image features of the product.

[0133] In a possible implementation, the fusion module 303 is further used to input the initial state and the multimodal features into the first linear layer to extract the intermediate features;

[0134] Input the intermediate features into the second linear layer to obtain the raw scores corresponding to each word in the vocabulary;

[0135] The original scores corresponding to each word are converted to determine the distribution probabilities corresponding to multiple words in the vocabulary.

[0136] In a possible implementation, the determination module 301 is further used to obtain the basic data, transaction data and browsing data of the customer;

[0137] Input the transaction data and browsing data into the pre-trained information extraction network to obtain the customer's transaction information and browsing information;

[0138] The customer's basic data, transaction information and browsing information are used as customer information features.

[0139] The product marketing text generation device based on multimodal feature fusion provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0140] Figure 4 This is a schematic diagram of the structure of a product marketing text generation device based on multimodal feature fusion provided in an embodiment of the present application. Figure 4 As shown, the electronic device 40 provided in this embodiment includes: at least one processor 401 and a memory 402. Optionally, the device 40 also includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus 404.

[0141] In a specific implementation process, at least one processor 401 executes the computer-executable instructions stored in the memory 402, so that at least one processor 401 executes the above method.

[0142] The specific implementation process of the processor 401 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0143] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0144] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0145] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0146] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0147] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0148] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0149] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0150] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0151] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0152] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0153] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A product marketing text generation method based on multimodal feature fusion, characterized in that: include: Determine customer information features and product information features, wherein the customer information features are determined based on the customer's basic data, transaction information, and browsing information, and the product information features are determined based on product recommendation information, wherein the product recommendation information includes text information and image information; A self-attention mechanism is used to fuse the customer information features and the product information features to determine multimodal features; The multimodal features are input into a pre-trained text encoding layer to generate a product marketing text for the customer, wherein the text encoding layer is used to generate the product marketing text based on the input multimodal features.

2. The method according to claim 1, characterized in that The self-attention mechanism is used to fuse the customer information features and the product information features to determine the multimodal features, including: Calculate the corresponding query, key, and value according to the parameter matrices corresponding to the query, key, and value and the customer information characteristics and product information characteristics; Determine a feature dimension parameter corresponding to the customer information feature, and determine the multimodal feature based on the feature dimension parameter and the query, key, and value.

3. The method according to claim 1, characterized in that The text encoding layer includes a decoder, and the multimodal features are input into a pre-trained text encoding layer to obtain the product marketing text of the customer, including: Initializing the decoder to determine an initial state of the decoder, wherein the initial state is used to indicate initial time step information of the encoder; Inputting the initial state and the multimodal features into the pre-trained text encoding layer to determine the distribution probabilities corresponding to a plurality of words in a vocabulary, wherein the vocabulary is obtained when training the text encoding layer; At least one target word is determined based on the distribution probabilities corresponding to the plurality of words, and a product marketing text for the customer is generated based on the at least one target word.

4. The method according to claim 1, characterized in that: The product information feature includes a text feature, and the determining of the product information feature includes: Get text information of products; Splitting the text information to obtain multiple clauses, and determining the word segmentation vectors corresponding to the multiple word segments contained in each clause; For any one of the multiple clauses, determining first contextual semantic information corresponding to each word in the clause according to the word vector of each word in the clause; Calculate the word segmentation weight of each word in the clause according to the first contextual semantic information, and determine the clause feature according to the word segmentation weight and the first contextual semantic information; The text feature is determined according to clause features of the plurality of clauses and second context semantic information corresponding to each clause.

5. The method according to claim 4, characterized in that The product information feature further includes: an image feature, and the method further includes: Determining initial image information of the product; Determine coordinate information of a target object according to the initial image information, wherein the target object is used to indicate image information of the product; Performing cropping processing on the initial image information of the product according to the coordinate information of the target object; According to the coordinate information of the target object, feature extraction is performed on the cropped initial image information to obtain the image features of the product.

6. The method according to claim 3, characterized in that The step of inputting the initial state and the multimodal features into the pre-trained text encoding layer to determine the distribution probabilities corresponding to a plurality of words in the vocabulary table comprises: Inputting the initial state and the multimodal features into a first linear layer to extract intermediate features; Inputting the intermediate features into the second linear layer to obtain the raw scores corresponding to each word in the vocabulary; The original scores corresponding to each word are converted to determine the distribution probabilities corresponding to the multiple words in the vocabulary.

7. The method according to claim 1, characterized in that Determining the customer information characteristics includes: Obtain customers’ basic data, transaction data, and browsing data; Inputting the transaction data and browsing data into a pre-trained information extraction network to obtain the transaction information and browsing information of the customer; The basic data, transaction information and browsing information of the customer are used as customer information features.

8. A product marketing text generation device based on multimodal feature fusion, characterized in that: include: A determination module, used to determine customer information features and product information features, wherein the customer information features are determined based on the customer's basic data, transaction information and browsing information, and the product information features are determined based on product recommendation information, wherein the product recommendation information includes text information and image information; A fusion module, used to fuse the customer information features and the product information features using a self-attention mechanism to determine a multimodal feature; A generation module is used to input the multimodal features into a pre-trained text encoding layer to generate a product marketing text for the customer, and the text encoding layer is used to generate the product marketing text based on the input multimodal features.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images