Method, computing device and storage medium for generating blessing text

Abstract

The embodiment of the present invention relates to a method, a computing device and a storage medium for generating a blessing text. The method includes generating a blessing tag embedding vector and a blessing prompt embedding vector respectively based on the blessing tag and the blessing prompt input by the user; splicing the blessing tag embedding vector and the blessing prompt embedding vector to obtain a first embedding vector for generating a blessing text; extracting the relationship between each word segment in the first embedding vector through the constructed language model, so as to calculate the attention weight about the first embedding vector, and the language model is constructed based on a multi-head attention mechanism; and calculating and outputting the text content of the next character in the blessing text to be generated based on the weight matrix, the blessing tag embedding vector and the attention weight, until the blessing text is generated. In this way, the appeal of the blessing text can be effectively improved and more personalized.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of data processing, and more specifically to a method, a computing device, and a storage medium for generating a blessing text. Background Art

[0002] Conventional methods for generating blessing texts, such as generating blessing texts through templates and a predetermined vocabulary library, use similar or identical generation parameters or templates for different scenarios and / or different blessing objects.

[0003] The above-mentioned traditional methods for generating blessing texts all have shortcomings. On the one hand, due to the limitations of fixed templates and vocabulary, the generated blessing texts lack personalization and creativity; on the other hand, due to the differences in blessing festivals, blessing scenes and blessing objects, different blessing tones are required to cater to the recipients of the blessings. The traditional blessing text generation method cannot accurately convey the true emotions and intentions of the blesser, and it is also difficult to cope with changes in blessing scenes and blessing objects.

[0004] In summary, the traditional method for generating blessing texts has the following shortcomings: it is impossible to generate personalized blessing texts according to changes in blessing objects and / or blessing scenes, and it is impossible to accurately convey the true emotions and intentions of the blesser. Summary of the invention

[0005] In response to the above problems, the present invention provides a method, computing device and storage medium for generating blessing texts, which can effectively improve the true meaning of the blessing texts and generate personalized and more appealing blessing texts for the blessing objects and / or blessing scenes.

[0006] According to a first aspect of the present invention, a method for generating a blessing text is provided, comprising: generating a blessing tag embedding vector and a blessing prompt embedding vector respectively based on a blessing tag and a blessing prompt input by a user; concatenating the blessing tag embedding vector and the blessing prompt embedding vector to obtain a first embedding vector for generating the blessing text; extracting the relationship between each word segment in the first embedding vector through a constructed language model so as to calculate the attention weight about the first embedding vector, wherein the language model is constructed based on a multi-head attention mechanism; and calculating and outputting the text content of the next character in the blessing text to be generated based on the weight matrix, the blessing tag embedding vector and the attention weight, until the blessing text generation is completed.

[0007] According to a second aspect of the present invention, a computing device is provided, comprising: at least one processing unit; at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the steps of the method according to the first aspect.

[0008] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a machine, the method according to the first aspect is implemented.

[0009] According to a fourth aspect of the present invention, there is further provided a computer program product, comprising a computer program, wherein when the computer program is executed by a machine, the method of the first aspect of the present invention is performed.

[0010] In some embodiments, based on the weight matrix, the blessing label embedding vector and the attention weight, calculating and outputting the text content of the next character in the blessing text to be generated includes calculating the predicted output value vector of each head based on the attention weight of each head of attention regarding the first embedding vector; adjusting the weight matrix based on the blessing label embedding vector; splicing the predicted output value vector of each head, and converting the spliced ​​predicted output value vector through the adjusted weight matrix to obtain the output value vector of the next character; and outputting the text content of the next character of the blessing text to be generated based on the output value vector of the next character.

[0011] In some embodiments, based on the blessing tag embedding vector, adjusting the weight matrix includes: adjusting the value associated with the blessing tag embedding vector in the weight matrix to 0.

[0012] In some embodiments, generating a blessing tag embedding vector and a blessing prompt embedding vector includes: generating a blessing tag embedding vector based on one-hot encoding, determining the number of values ​​corresponding to each blessing tag, and the value of each value corresponding to the blessing tag; making each value corresponding to the blessing tag occupy one vector length bit; and making the length of the generated blessing tag embedding vector the same as the length of the blessing prompt embedding vector.

[0013] In some embodiments, obtaining a first embedded vector for generating a blessing text includes: determining that the internal sequence of the first embedded vector is: a starting vector column, a blessing label embedded vector column, a blessing prompt embedded vector column, and an ending vector column; or determining that the internal sequence of the first embedded vector is: a starting vector column, a blessing prompt embedded vector column, a blessing label embedded vector column, and an ending vector column; wherein the blessing prompt embedded vector column contains multiple sequentially connected vector columns.

[0014] In some embodiments, the blessing tag is related to the blessing scene, blessing object and / or blessing tone; the blessing tag is also used to indicate the reference range and / or initial weight of the blessing word segmentations stored in the training database associated with the language model during the current blessing text generation process.

[0015] In some embodiments, the blessing tag is related to the blessing scene, blessing object and / or blessing tone, including: the blessing scene includes: festival type, festival customs and / or festival season; the blessing tag is related to the blessing object includes: the type of object, the gender of the object, the personality of the object, the occupation of the object, the unit of the object, the position of the object, the personality of the object and / or the birthday of the object; the blessing tone includes cheerfulness, solemnity, passion, plainness, seriousness and / or network style.

[0016] Therefore, the present invention can splice the blessing label embedding vector and the blessing prompt embedding vector so that the language model input end can splice the blessing label embedding vector and the blessing prompt embedding vector in the vector embedding link to obtain the first embedding vector for generating the blessing text, so that the input data of the language model, in addition to the conventional prompts, also splices the information of the label vector, so that the blessing text content generated by the language model can better match the recipient of the blessing text, and have more abundant emotions and more personalized blessing text generation to adapt to each different blessing text recipient.

[0017] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0019] Figure 1 A schematic diagram of a system for implementing a method for generating a blessing text according to an embodiment of the present invention is shown.

[0020] Figure 2 A flow chart of a method for generating a blessing text according to an embodiment of the present invention is shown.

[0021] Figure 3 A flow chart of a method for calculating the next character in a blessing text according to an embodiment of the present invention is shown.

[0022] Figure 4 A schematic structural diagram of a first embedding vector according to an embodiment of the present invention is shown.

[0023] Figure 5 A schematic structural diagram of another first embedding vector according to an embodiment of the present invention is shown.

[0024] Figure 6 A schematic diagram of a weight matrix according to an embodiment of the present invention is shown.

[0025] Figure 7 A block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0028] As described above, the traditional method for generating blessing texts has the following shortcomings: it is impossible to generate personalized blessing texts according to changes in blessing objects and / or blessing scenes, and it is impossible to accurately convey the true emotions and intentions of the blesser.

[0029] In order to at least partially solve one or more of the above problems and other potential problems, an exemplary embodiment of the present invention proposes a solution for generating a blessing text. In the solution of the present invention, a blessing label embedding vector and a blessing prompt embedding vector are generated respectively based on the blessing label and blessing prompt input by the user; the relationship between each word segment in the first embedding vector is extracted through the constructed language model to calculate the attention weight about the first embedding vector, and the language model is constructed based on the multi-head attention mechanism; and based on the weight matrix, the blessing label embedding vector and the attention weight, the text content of the next character in the blessing text to be generated is calculated and output until the blessing text is generated; thus, the present invention can splice the blessing label embedding vector and the blessing prompt embedding vector at the input end of the language model in the vector embedding link to obtain a first embedding vector for generating the blessing text, so that the input data of the language model, in addition to the conventional prompt, also splices the information of the label vector, so that the blessing text content generated by the language model can better match the recipient of the blessing text, and has more abundant emotions and more personalized blessing text generation to adapt to each different blessing text recipient.

[0030] Figure 1 FIG. 1 is a schematic diagram of a system 100 (referred to as system 100 ) for implementing a method for generating a blessing text according to an embodiment of the present invention. Figure 1 As shown in , the system 100 includes a computing device 110 and a server 130, a network 140, a database 150 and a terminal device 160. The computing device 110, the server 130, the database 150 and the terminal device 160 can all perform data exchange through the network 140 (eg, the Internet, a local area network, a wide area network, etc.).

[0031] The terminal device 160 may be a hardware device or software that supports network connection for data interaction and data processing, such as a mobile device, a personal terminal, a desktop, a smart watch, a tablet computer, an interactive device, etc., and a user can perform interactive operations on its display interface (such as initiating a blessing text generation request, sending a blessing text, receiving a blessing text, replying to a blessing text, etc.).

[0032] Regarding the database 150, it stores blessing text training data, blessing object information (for example, multiple employees under a group user can become blessing objects, and the database 150 stores information on multiple employees of the entire group user), and also stores word segmentation information and label information of multiple blessing texts, so that the server 130 and / or the computing device 110 can generate requests based on the blessing texts, generate blessing prompts, blessing labels, etc.

[0033] Regarding server 130, it can be, for example, an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud database, cloud storage, network services, cloud communications, middleware services, domain name services, security services, as well as big data and artificial intelligence platforms.

[0034] The server 130 can provide a variety of services, such as obtaining a blessing text generation request from a user, generating a blessing prompt based on the user's blessing text generation request, generating a blessing prompt and a blessing label based on the blessing text and the blessing text generation request stored in the database 150, and receiving and executing a blessing text sending request. The server 130 can also provide the blessing prompt, blessing label, and blessing text generation request to the computing device 110.

[0035] Regarding the computing device 110, it is used, for example, to train the language model for blessing text generation of the embodiment of the present invention, for example, the method for blessing text generation of the embodiment of the present invention is deployed thereon. The computing device 110 may have one or more processing units, including dedicated processing units such as GPU, FPGA and ASIC, and general processing units such as CPU. In addition, one or more virtual machines may also be running on each computing device 110. In some embodiments, the computing device 110 and the server 130 may be integrated together or may be separately arranged from each other. In some embodiments, the computing device 110, for example, includes an embedding vector generation module 112, an embedding vector splicing module 114, an attention calculation module 116, and a blessing text generation module 118.

[0036] The embedding vector generating module 112 is used to generate a blessing tag embedding vector and a blessing prompt embedding vector respectively based on the blessing tag and blessing prompt input by the user.

[0037] As for the embedding vector concatenation module 114, it is used to concatenate the blessing tag embedding vector and the blessing prompt embedding vector to obtain a first embedding vector for generating a blessing text.

[0038] Regarding the attention calculation module 116, it is used to extract the relationship between each word in the first embedding vector through the constructed language model, so as to calculate the attention weight about the first embedding vector. The language model is constructed based on a multi-head attention mechanism.

[0039] Regarding the blessing text generation module 118, it is used to calculate and output the text content of the next character in the blessing text to be generated based on the weight matrix, the blessing label embedding vector and the attention weight until the blessing text generation is completed.

[0040] Figure 2 FIG. 2 is a flow chart of a method 200 for generating a blessing text according to an embodiment of the present invention. The method 200 for generating a blessing text may be as follows: Figure 1 The computing device 110 shown may also be executed in Figure 7 The method 200 is executed at the electronic device 700 shown. It should be understood that the method 200 for generating a blessing text may also include additional steps not shown and / or may omit the steps shown, and the scope of the present invention is not limited in this respect.

[0041] In step 202 , the computing device 110 generates a blessing tag embedding vector and a blessing prompt embedding vector respectively based on the blessing tag and blessing prompt input by the user.

[0042] Regarding the blessing tags and blessing prompts input by the user, for example, based on the blessing text generation request initiated by the user, the blessing tags and blessing prompts corresponding to the blessing text generation request are generated. For example, based on the blessing text generation request initiated by the user, the blessing tags related to the blessing text generation request are extracted; or tag options are recommended to the user, and the blessing tags are determined according to the user's selection.

[0043] Regarding the type of blessing tags, for example, the type of terminal to which the blessing is sent (app push, SMS push, email push), the type of festival (New Year, Lantern Festival, Mid-Autumn Festival), the target group (old employees, young employees), the size of the enterprise (small customers, large customers), the nature of the enterprise (state-owned and state-holding enterprises, private enterprises, private enterprises, foreign-funded enterprises, institutions, associations and private non-enterprise units, etc.); and also for example, the type of industry (such as computer / Internet / communication / electronics, such as finance / banking / insurance, such as pharmaceuticals / medical care, such as advertising / media, such as real estate / construction, such as education / training, such as service industry, such as logistics / transportation, such as energy / raw materials, such as government / non-profit organizations, such as trade / consumption / manufacturing / operation, such as others).

[0044] For example, keywords related to blessing objects, blessing scenes and / or blessing tones are extracted from the blessing text generation request, and blessing tags are matched to the blessing text generation request based on the extracted keywords. For another example, blessing prompts are generated based on the text input by the user; or blessing prompt options are recommended based on the text input by the user, and blessing prompts are generated in response to the user's selection and / or input.

[0045] For example, based on the information of the initiator of the blessing request, the information of the blessing object is determined, so as to determine the blessing label and / or blessing prompt of the blessing text to be generated based on the information of the blessing object. For example, the initiator of the blessing request, Zhang San, initiates a blessing request for the blessing object, Xiao Ming, on the Mid-Autumn Festival. Based on Zhang San's user information, Zhang San's colleague Xiao Ming is found in the database 150, and it is determined that Xiao Ming is male, a new employee of the company, unmarried and single, and his relatives are all in cities other than the workplace. Based on the above information of Xiao Ming, the labels of the blessing text to be generated are generated, such as "new employee", "out-of-town", "living alone", "single male", "Mid-Autumn Festival" and other blessing labels.

[0046] In addition, blessing labels can be obtained by language annotation based on pre-collected blessing text data, and the type of blessing labels can also be adjusted based on user feedback. Blessing labels can reflect the common characteristics of blessing texts in certain aspects. The blessing text database 150 stores a large number of pre-collected blessing texts with blessing labels, and can also store blessing texts generated by the language model during the training process and the use process. These blessing text data can be used for the training of the language model and for the generation of blessing texts.

[0047] The above scheme can generate the content of the request based on the blessing text input by the user based on the blessing tag system, recommend blessing tags to users and / or generate blessing tags, so as to obtain blessing tags for the blessing text to be generated, so that the language model can generate blessing text accurately and efficiently.

[0048] In addition, it is also possible to recommend blessing prompts and / or recommended blessing prompts to users based on the content of the blessing text generation request input by the user, so that users can more easily and quickly obtain blessing prompts about the blessing text to be generated; the above scheme, especially for group users with multiple blessing objects, can automatically extract the blessing labels and blessing prompts of the blessing text to be generated for each blessing object among multiple blessing objects, so as to generate customized blessing texts for multiple blessing objects.

[0049] For example, Zhang San, the initiator of the blessing request, inputs a blessing text generation request "Generate New Year blessing prompts for each employee of Company A". In response to the request, information about all employees of Company A is queried from database 150, and based on the information of each employee (each blessing object), a blessing label and blessing prompt for each blessing object is generated. Since the information of each employee is different (such as position, gender, age, family members, place of origin, personality, etc.), the blessing label and blessing prompt of the blessing text to be generated for each blessing object are different. This can achieve customized blessing text generation and enable each blessing object to receive a blessing text that makes them feel happier.

[0050] In some embodiments, the blessing tag is related to the blessing scene, the blessing object and / or the blessing tone. In some embodiments, the blessing scene includes: festival type, festival customs and / or festival season. In some embodiments, the blessing tag is related to the blessing object including: the type of object, the gender of the object, the personality of the object, the occupation of the object, the unit of the object, the position of the object, the personality of the object and / or the birthday of the object. In some embodiments, the blessing tone includes cheerful, solemn, passionate, plain, serious and / or network style.

[0051] For easy understanding, some blessing tags are listed below: the blessing tag "Mid-Autumn Festival" means that the blessing text is used for the Mid-Autumn Festival, the blessing tag "Dragon Boat Festival" means that the blessing text is used for the Dragon Boat Festival, the blessing tag "Elders" means that the blessing object is the elders, the blessing tag "Leaders" means that the blessing object is the company leaders, the blessing tag "Summer" means that the blessing scene is summer, the blessing tag "Birthday" means that the blessing scene is a birthday, the blessing tag "Female" means that the blessing object is female, the blessing tag "Lively" means that the blessing tone needs to be lively, and the blessing tag "Birthday" means that the blessing scene is the birthday of the blessing object.

[0052] In some embodiments, the blessing tag is related to the blessing scene, blessing object and / or blessing tone; the blessing tag is also used to indicate the reference range and / or initial weight of the blessing word segmentations stored in the training database associated with the language model during the current blessing text generation process.

[0053] Regarding blessing word segmentation (i.e., word unit, token), for example, when generating text through a language model, the language model can be used to better learn and refer to objects through the method of prompts + auxiliary reference text, so that the generated content is more in line with user needs. In the training database 150, a number of blessing texts are stored, each of which is marked with a blessing label. Through the blessing label of the blessing text to be generated, the language model can focus on the segmentation (i.e., word unit, token) related to the provided blessing label, and reset the segmentation weights that are irrelevant or contrary to the blessing label to 0.

[0054] For example, if the blessing tag is "Dragon Boat Festival", the weights of tokens related to other festivals such as New Year and Winter Solstice are reset to 0; if the blessing tag is "single male", the weights of tokens outside this group are reset to 0; if the blessing tag is "solemn", the weights of tokens with other blessing tones are reset to 0. For another example, if the blessing tags for the blessing text generation request are "spring", "outing", "lively", and "employee", then based on this group of blessing tags, the initial probability of the related word segmentation (word unit) is increased to facilitate faster calculation of each character in the blessing text to be generated.

[0055] Therefore, the above scheme fully considers various factors such as blessing scene, blessing object and blessing tone to construct a blessing label system, which can make the blessing text to be generated more targeted, thereby instructing the language model to generate a blessing text that is more touching to the blessing object. In addition, the blessing label can accurately point to the auxiliary reference text (or word segmentation, word unit, etc.) in the text generation process, so that the language model can generate each character in the blessing text to be generated more quickly and accurately.

[0056] In step 204 , the computing device 110 concatenates the blessing tag embedding vector and the blessing prompt embedding vector to obtain a first embedding vector for generating the blessing text.

[0057] In some embodiments, generating a blessing tag embedding vector and a blessing prompt embedding vector includes: generating a blessing tag embedding vector based on one-hot encoding, determining the number of values ​​corresponding to each blessing tag, and the value of each value corresponding to the blessing tag; making each value corresponding to the blessing tag occupy one vector length bit; and making the length of the generated blessing tag embedding vector the same as the length of the blessing prompt embedding vector.

[0058] For example, if the length of the prompt embedding vector is 768, then the length of the blessing tag embedding vector is also 768. Therefore, the above scheme does not change the embedding vector structure of the prompt in the traditional scheme, but generates an embedding vector for the blessing tag separately, so that the length of the blessing tag embedding vector is the same as the length of the prompt embedding vector, which is convenient for various operations, splicing, stacking and embedding between the blessing tag embedding vector and the prompt embedding vector.

[0059] For example, to determine the value of the blessing tag embedding vector, a blessing tag embedding vector has a length of 768 bits, and can be used to contain information of multiple blessing tags, such as blessing text generation request 001, whose blessing tags are tag A, tag B and tag C, where the values ​​of tag A, tag B and tag C are shown in Table 1.

[0060] Table 1:

[0061]

[0062] Based on the above label values, a blessing label embedding vector 001 is generated for the blessing text generation request 001, and its corresponding vector is [1,1,1,1,1,0,0,1,1,…0] (a total of 768 bits, and unused null value bits can all be represented by 0). The final generated blessing label embedding vector also needs to undergo softmax normalization processing, etc.

[0063] Through the above scheme, when constructing the first embedding vector, the blessing tag can be embedded based on the existing embedding vector method, so that the obtained blessing tag embedding vector can be directly spliced ​​with the input prompt embedding vector of the language model without changing the basic length structure of the vector. The blessing tag embedding vector is regarded as an ordinary word segmentation embedding vector, so that in subsequent calculations, the language model does not need to make any adjustments, and the blessing text generation result can be obtained based on the first embedding vector, and the compatibility is very high; at the same time, the characteristic data of the blessing tag can be retained to guide the language model to more efficiently generate blessing texts that are adapted to the blessing text request.

[0064] In some embodiments, obtaining the first embedding vector for generating the blessing text includes: the computing device 110 determines that the internal sequence of the first embedding vector is: a start vector column, a blessing label embedding vector column, a blessing prompt embedding vector column, and an end vector column. The blessing prompt embedding vector column includes a plurality of sequentially connected vector columns.

[0065] For example, see Figure 4 , which illustrates a first embedding vector. The structure of the first embedding vector is as follows Figure 4 From left to right, they are the starting vector, the blessing label embedding vector, the blessing prompt embedding vector composed of several word segmentation embedding vectors, and the ending vector.

[0066] In some embodiments, alternatively, obtaining the first embedding vector for generating the blessing text includes: the computing device 110 determines that the internal sequence of the first embedding vector is: a start vector column, a blessing prompt embedding vector column, a blessing label embedding vector column, and an end vector column. The blessing prompt embedding vector column includes a plurality of sequentially connected vector columns.

[0067] For example, see Figure 5 , which illustrates another first embedding vector. The structure of the first embedding vector is as follows Figure 5 From left to right, they are the starting vector, the blessing prompt embedding vector composed of several word segmentation embedding vectors, the blessing label embedding vector and the ending vector.

[0068] Among them, the blessing prompt embedding vector is composed of several word segmentation embedding vectors. The number of word segmentation embedding vectors depends on the number of tokens after the blessing prompt is segmented. The start vector and the end vector are the two beginning and end filling vectors of the first embedding vector of the input language model (such as <pos> and< / pos> corresponding vector).

[0069] In addition, the number of blessing label vectors can be one or more, and whether the number of blessing label vectors needs to be expanded is determined based on the length of the blessing label. For example, if the length of the prompt embedding vector is 768, then the length of each blessing label vector is also 768. If the 768-bit vector cannot fully express the blessing label vector, a blessing label vector with a length of 768 can be generated.

[0070] It is worth noting that in the above scheme, the blessing label vector can only be located after the start embedding column or before the end embedding column. It cannot be located before the start embedding column, otherwise the start semantics will be problematic, resulting in the inability to reuse the existing language model (the semantics of the start embedding in the existing language model is the start. If the position is wrong, the vector will be meaningless). It also cannot be located anywhere in the middle of several word embedding vector columns of the blessing prompt embedding vector, which will destroy the position encoding and original semantics of the blessing prompt in the language model, thereby affecting the accuracy of the text generation results.

[0071] In step 206, the computing device 110 extracts the relationship between each word segment in the first embedding vector via the constructed language model so as to calculate the attention weight about the first embedding vector, and the language model is constructed based on the multi-head attention mechanism.

[0072] Regarding language models, for example, there are GPT (Generative Pre-trained Transformer) series models, BERT (Bidirectional Encoder Representations from Transformers) models, Wenxin Yiyan, Tongyi Qianwen, New Bing, etc. The language model is built based on the Transformer model, and the multi-head attention mechanism of the Transformer model is adopted.

[0073] In step 208, the computing device 110 calculates and outputs the text content of the next character in the blessing text to be generated based on the weight matrix, the blessing tag embedding vector and the attention weight until the blessing text is generated.

[0074] The following will combine Figure 3 The method for calculating the next character in the blessing text according to the embodiment of the present invention is described in detail, which will not be repeated here.

[0075] In the above scheme, the present invention can generate a request and / or input a blessing tag according to the user's blessing text by constructing a blessing tag system, generate a blessing tag embedding vector with the same length as the language model embedding vector, and splice the blessing tag embedding vector and the blessing prompt embedding vector to obtain a first embedding vector for generating the blessing text, so that the input data of the language model is spliced ​​with the information of the label vector in addition to the traditional prompt, so that the language model can obtain relevant auxiliary reference text (or word segmentation, word unit, etc.) in the text generation process through the precise pointing of the blessing tag, so that the language model can generate each character in the blessing text to be generated more quickly and accurately, so that the generated blessing text content is more in line with the recipient of the blessing text, and has more abundant emotions and more personalized blessing text generation to adapt to each different blessing text recipient.

[0076] Figure 3 1 shows a flow chart of a method 300 for calculating the next character in a blessing text according to an embodiment of the present invention. The method 300 for calculating the next character in a blessing text may be as follows: Figure 1 The computing device 110 shown may also be executed in Figure 7 The method 300 for calculating the next character in the blessing text may also include additional steps not shown and / or may omit the steps shown, and the scope of the present invention is not limited in this respect.

[0077] In step 302, the computing device 110 calculates a predicted output value vector of each head based on the attention weight of each head's attention with respect to the first embedding vector.

[0078] In step 304 , the computing device 110 adjusts the weight matrix based on the blessing tag embedding vector.

[0079] In some embodiments, based on the blessing tag embedding vector, adjusting the weight matrix includes: adjusting the value associated with the blessing tag embedding vector in the weight matrix to 0.

[0080] Therefore, adjusting the value associated with the blessing label embedding vector in the weight matrix to 0 can cut off the influence of the blessing label on the generated content, so that the blessing label does not directly affect the generation of the blessing text. Instead, the blessing label embedding vector instructs the language model to more quickly determine the word segmentation with a high correlation with the blessing text to be generated, thereby completing the generation of the blessing text through the language model and the word segmentation (word unit, token) with high similarity in the training database.

[0081] There is an assumption in the multi-head attention mechanism that each vector has a certain relationship with other vectors, especially the padding vectors at the beginning and end. However, based on the generation mechanism of the blessing tag, our blessing tag embedding vector does not actually have such a relationship with other embedding vectors. Therefore, in order to make the result of weight calculation not affected by the blessing tag embedding vector, the present invention improves the attention mechanism. For example, taking the attention layer of the Transformer model as an example, it mainly has three matrices, the query matrix Q, the key matrix K and the value matrix V. The attention matrix Z is obtained by the following formula (1):

[0082] (1)

[0083] Among them, formula (1) is the core calculation formula of the attention mechanism; Q, K, V: these three letters represent query, key, and value, which are the three main components of the attention mechanism. In different application scenarios, Q, K, and V can be different data representations. For example, in NLP (natural language processing), they can be the embedding vectors of different words in a sentence.

[0084] in, Represents the dot product operation of the query (Q) and the key (K). Specifically, if Q and K are both matrices, then this step is a matrix multiplication of the Q matrix with the transpose (T) of the K matrix, and the result is a new matrix whose elements represent the similarity between each query and each key.

[0085] in, This is a scaling operation to prevent the gradient of the softmax function from disappearing due to the dot product result being too large. is the dimension of the key vector. This scale factor is an important detail of the attention mechanism that helps the model remain stable during training.

[0086] Among them, softmax In Represents the scaled dot product result mentioned above. The softmax function is used to convert the scaled similarity value into a probability distribution, that is, the relative importance of each key to the current query. The output of the softmax function is a new matrix whose elements in each row sum to 1, and each element represents the attention weight of the corresponding key.

[0087] Finally, the output of the softmax function (attention weight) is multiplied by the value (V) matrix to get the final attention output. This step actually applies the attention weight to the value. According to the attention weight of each key, the value is weighted and summed to get a new representation that takes into account all key information.

[0088] In step 306 , the computing device 110 concatenates the predicted output value vectors of each head, and transforms the concatenated predicted output value vectors via the adjusted weight matrix to obtain an output value vector of the next character.

[0089] In step 308 , the computing device 110 outputs the text content of the next character of the blessing text to be generated based on the output value vector of the next character.

[0090] Continuing with the above example, if the multi-head attention matrix has 8, the final multi-head attention formula (2) is:

[0091] (2)

[0092] Among them, Multi-head attention represents the calculation result of multi-head attention.

[0093] Among them, Z1, Z2, …, Z8: These Zs represent the outputs obtained after applying the attention mechanism on different "heads". Specifically, for each head, a query, key, and value matrix are calculated, and a weighted value matrix is ​​obtained through the attention mechanism (such as the softmax dot product attention mentioned above); here Z1, Z2, …, Z8 are the output matrices obtained by 8 different heads.

[0094] Among them, Concatenation (Z1, Z2, …, Z8) means concatenating the output matrices of all heads along a certain dimension (usually the last dimension). The result of concatenation is a larger matrix that contains the information captured by all heads.

[0095] Where W is a linear transformation matrix that converts the concatenated matrix back to the same dimension as the original input data (or any dimension required). This step is necessary because the concatenated matrix dimension may be larger than the dimension of the original input data, and other parts of the model may expect to receive outputs of the same dimension as the original input data.

[0096] It should be noted that if the matrix of the relevant operation is cut here (i.e. the blessing label dimension is removed), there is no problem in this step. However, the output of the attention layer is the input of the position encoding layer, and the position encoding layer is missing a dimension, which will cause problems. The incorrect structure makes the tensor unable to operate. We can modify the output tensor here.

[0097] Therefore, in the present invention, a new method is proposed: when passing through the attention layer, the weights (weight) of the blessing label embedding vector are all set to 0, that is, W[1,j] and W[i,1]=0, i, j are all row and column coordinates of the blessing label embedding vector. Then the calculation results of the blessing label embedding vector and other vectors are also 0, which means that there is no relationship between the blessing label embedding vector and the prompt embedding vector. Of course, we can also set the 1st row and 1st column of the initial weight matrix of QKV to 0 at the beginning of pre-training, but because QKV is learned, if the weight value W is not set, then its 1st row and 1st column will still become other values. For example, please refer to Figure 6 , indicating that the rows and columns related to the blessing label embedding vector in the weight matrix are set to 0.

[0098] Through the above scheme, the network structure of the language model is changed instead of adding network depth, which is different from general transfer learning (fine-tuning). For example, an original language model XX is selected. In its transformer model part, the main layers with weight values ​​include: word embedding layer; query, key, value and output linear layers of the self-attention mechanism layer; two linear layers of the feedforward neural network layer; output linear layer. Although the word embedding layer is paired with the large model, it is obtained before training the large model and can be reused directly. We can also reuse the self-attention layer, but we need to add rows with row number 1 and columns with column number 1, and their values ​​are all 0.

[0099] At the same time, the output linear layer n, where n is the number of linear output results (i.e., dimension), we also added a value output, which becomes n+1. At the index=1 position, the initial value is 0. The other values ​​are all the corresponding weight values ​​of the self-attention layer in the original model. The two linear layers of the feedforward neural network layer have the same value increase (i.e., n+1). The difference from the self-attention layer is that even if the self-attention layer has more layers, its weight value is still 0 during subsequent training, and only other rows and columns will change. However, the initial value 0 of the index=1 position added in the feedforward neural network layer will change during training. The final output linear layer, the position with index=1 added, also has an initial value of 0. Therefore, based on the above scheme, secondary fine-tuning training can be performed based on the existing language model to a certain extent, greatly reducing the overall cost while ensuring the effect.

[0100] Figure 7 Schematic diagram of an example electronic device 700 that can be used to implement an embodiment of the present specification is shown. Figure 1 The computing device 110 shown can be implemented by an electronic device 700. As shown, the electronic device 700 includes a central processing unit CPU 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory ROM 702 or computer program instructions loaded from a storage unit 708 into a random access memory RAM 703. In the random access memory RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The central processing unit CPU 701, the read-only memory ROM 702 and the random access memory RAM 703 are connected to each other via a bus 704. An input / output I / O interface 705 is also connected to the bus 704.

[0101] Multiple components in the electronic device 700 are connected to the input / output I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, a microphone, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0102] The various processes and processing described above, such as the method 200 for generating a blessing text to the method 300 for calculating the next character in the blessing text, can be executed by the central processing unit CPU701. For example, in some embodiments, the method 200 for generating a blessing text to the method 300 for calculating the next character in the blessing text can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via the read-only memory ROM702 and / or the communication unit 709. When the computer program is loaded into the random access memory RAM703 and executed by the central processing unit CPU701, one or more actions of the method 200 for generating a blessing text to the method 300 for calculating the next character in the blessing text described above can be performed.

[0103] The present invention relates to methods, apparatuses, systems, electronic devices, computer-readable storage media and / or computer program products. The computer program products may include computer-readable program instructions for executing various aspects of the present invention.

[0104] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0105] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge computing devices. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0106] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, the electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be executed by using the state information of the computer-readable program instructions to personalize and customize the electronic circuit, thereby implementing various aspects of the present invention.

[0107] Various aspects of the present invention are described herein with reference to flow charts and / or step diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each of the steps in the flow charts and / or step diagrams and the combination of the steps in the flow charts and / or step diagrams can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more steps in the flowchart and / or step diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more steps in the flowchart and / or step diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more steps in the flowchart and / or step diagram.

[0110] The flowchart and step diagram in the accompanying drawings show the possible architecture, functions and operations of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square step in the flowchart or step diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the square steps can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive square steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square step in the step diagram and / or the flowchart, and the combination of the square steps in the step diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a blessing text, characterized in that: include: Based on the blessing label and blessing prompt input by the user, generate a blessing label embedding vector and a blessing prompt embedding vector respectively; The blessing tag is related to the blessing scene, the blessing object and / or the blessing tone; the blessing tag is also used to indicate the reference range and / or initial weight of the blessing word segmentation stored in the training database associated with the language model during the current blessing text generation process; Concatenate the blessing tag embedding vector and the blessing prompt embedding vector to obtain a first embedding vector for generating a blessing text; Extracting, via the constructed language model, a relationship between each word segment in the first embedding vector so as to calculate an attention weight about the first embedding vector, wherein the language model is constructed based on a multi-head attention mechanism; as well as Based on the weight matrix, the blessing label embedding vector and the attention weight, the text content of the next character in the blessing text to be generated is calculated and output until the blessing text is generated.

2. The method according to claim 1, characterized in that Based on the weight matrix, the blessing label embedding vector and the attention weight, the text content of the next character in the blessing text to be generated is calculated and outputted, including: Calculate a predicted output value vector of each head based on the attention weight of each head's attention with respect to the first embedding vector; Adjust the weight matrix based on the blessing label embedding vector; splicing the predicted output value vectors of each head, and transforming the spliced ​​predicted output value vectors through the adjusted weight matrix to obtain the output value vector of the next character; and Based on the output value vector of the next character, the text content of the next character of the blessing text to be generated is output.

3. The method according to claim 2, characterized in that Based on the blessing label embedding vector, the weight matrix is ​​adjusted to include: Adjust the values ​​associated with the blessing label embedding vector in the weight matrix to 0.

4. The method according to claim 3, characterized in that Generating blessing label embedding vectors and blessing prompt embedding vectors includes: Generate a blessing label embedding vector based on one-hot encoding, determine the number of values ​​corresponding to each blessing label, and the value of each value corresponding to the blessing label; Make each value corresponding to the blessing label occupy one vector length bit; and Make the length of the generated blessing tag embedding vector the same as the length of the blessing prompt embedding vector.

5. The method according to claim 2, characterized in that Obtaining the first embedding vector for generating the blessing text includes: Determine the internal sequence of the first embedding vector as: a start vector column, a blessing label embedding vector column, a blessing prompt embedding vector column, and an end vector column; or Determine the internal sequence of the first embedding vector as: a starting vector column, a blessing prompt embedding vector column, a blessing label embedding vector column, and an ending vector column; Among them, the blessing prompt embedding vector column includes multiple sequentially connected vector columns.

6. The method according to claim 5, characterized in that The blessing tag is related to the blessing scene, blessing object and / or blessing tone and includes: The blessing scenarios include: festival type, festival customs and / or festival season; The blessing tag is related to the blessing object, including: the type of the object, the gender of the object, the personality of the object, the occupation of the object, the unit of the object, the position of the object, the personality of the object and / or the birthday of the object; The tones of the blessings include cheerful, solemn, passionate, plain, serious and / or internet-like.

7. A computing device, characterized in that include: at least one processing unit; At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a machine, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a machine, the method according to any one of claims 1 to 6 is performed.

Images