Email element setting system and email subject setting support system
The system uses a large-scale language model to optimize email subject lines by incorporating user attributes and reaction indices, addressing the inadequacy of conventional systems in generating suitable subject lines for articles or pages.
Patent Information
- Application Number
- JP2023084040
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2043-05-22
AI Technical Summary
Conventional email subject generation systems fail to provide suitable display information when introducing articles or pages via email, leading to inadequate subject line settings.
A system that utilizes a large-scale language model to convert requests into keyword information, which is then used to set email elements, including the subject line, based on user attributes and reaction indices, leveraging machine learning to optimize subject line candidates.
The system effectively sets email subjects with higher reaction indices, improving open rates and user engagement by considering character types, symbols, and user attributes.
Smart Images

Figure 2024167756000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a system for setting or supporting the setting of email subjects, etc., and in particular to an email element setting system and an email subject setting support system that are suitable for setting email subjects, etc. when introducing articles or pages by email. [Background technology]
[0002] 2. Description of the Related Art Conventionally, the technique described in Patent Document 1 is known as a technique for generating display information for electronic mail (hereinafter abbreviated as "mail").
[0003] The technology described in Patent Document 1 generates display information that is displayed as the sender name of the email based on the email information acquired by the acquisition unit 131, and that displays information other than the identification information that identifies the sender of the email (Patent Document 1
[0079] ). Specifically, the generation unit 132 refers to the project information storage unit 121 and acquires, as the project information, the phrase "Pet Special (First Edition)" that indicates the project name stored in the item of the project name. Next, the generation unit 132 breaks down the phrase included in the acquired project information into words by morphological analysis. The generation unit 132 breaks down the phrase "Pet Special (First Edition)" that indicates the project name acquired as the project information into "Pet Special" and "(First Edition)". Next, the generation unit 132 generates display information so that the number of characters that can be displayed as a sender name is within about 14 characters. The generating unit 132 extracts the word "Pet Special" by removing "(First Edition)" from "Pet Special (First Edition)", which is a phrase indicating the project name acquired as the project information. Next, the generating unit 132 generates display information in which the extracted word "Pet Special" is displayed as information other than the identification information that identifies the sender of the email. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2020-13410 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, in the technology described in Patent Document 1, display information is generated based on the project information (a phrase indicating the project name stored in the project name field) in the project information storage unit 121, so there was a problem that suitable display information could not be obtained when introducing an article or an internet site page (hereinafter abbreviated as "page") that does not match the project information via email.
[0006] Therefore, the present invention has been made with a focus on the unresolved issues inherent in such conventional technology, and aims to provide an email element setting system and an email subject setting support system that are suitable for setting email subjects, etc., when introducing articles or pages via email. [Means for solving the problem]
[0007] [Invention 1] In order to achieve the above object, an email element setting system of Invention 1 comprises a request input means for inputting a request including document information or link information relating to a link destination of the document information and a request to generate a keyword representative of the document information into a large-scale language model, a keyword information acquisition means for acquiring keyword information output from the large-scale language model in response to the request, and an email element setting means for setting the subject, body or other elements of an email based on the keyword information acquired by the keyword information acquisition means.
[0008] With this configuration, the request input means inputs a request including a request to generate document information or link information and keywords to the large-scale language model, the keyword information acquisition means acquires keyword information output from the large-scale language model in response to the request, and the email element setting means sets email elements based on the acquired keyword information.
[0009] Here, the keywords that represent the document information include, for example, words that express or represent the central content, main concept, or feature of the document information, and include any word units such as sentences, paragraphs, clauses, words, etc. The same applies below to the email subject setting support system of invention 2.
[0010] Furthermore, the keyword information can be, for example, configured as the keyword itself, or as information for identifying the keyword (for example, name, number, ID, code, link information such as URL, etc.). The keyword information can be, for example, configured as characters, numbers, figures, codes, symbols, images, sounds, and other information. The same applies below to the email subject setting support system of invention 2.
[0011] In addition, the email element setting means includes, for example, a configuration for setting the subject, body or other elements of an email based on the acquired keywords, or a configuration for recording some or all of the acquired keywords in a file or the like as the subject, body or other elements of an email.
[0012] In addition, this system may be realized as a single device, terminal, or other equipment, or may be realized as a network system in which multiple devices, terminals, or other equipment are communicatively connected. In the latter case, each component may belong to any of the multiple devices, etc., as long as they are communicatively connected to each other. The same applies below to the email subject setting support system of invention 2.
[0013] [Invention 2] On the other hand, in order to achieve the above object, an email subject setting support system of Invention 2 comprises: a request input means for inputting a request including a request to generate document information or link information related to a link destination of the document information and a plurality of keywords representative of the document information into a large-scale language model; a subject information acquisition means for acquiring a plurality of keyword information output from the large-scale language model in response to the request as subject information related to the subject which is a candidate for the email subject; an attribute information acquisition means for acquiring attribute information related to the attributes of a user to whom the email is to be sent; and an estimation means for estimating a response indicator of each candidate related to the subject information for a user to whom the email is to be sent from the subject information acquired by the subject information acquisition means and the attribute information acquired by the attribute information acquisition means, using a trained model that has been trained based on subject feature information including feature information that influences a response indicator indicating the degree of the user's response to the email, attribute information related to the attributes of the user who sent the email, and response indicator information related to the response indicator of the email.
[0014] With this configuration, the request input means inputs a request including a request to generate document information or link information and multiple keywords to the large-scale language model, and the subject information acquisition means acquires multiple keyword information output from the large-scale language model in response to the request as subject information. The attribute information acquisition means acquires attribute information related to the attributes of the user, and the estimation means estimates response indicators of each candidate related to the subject information for the user to whom the email is to be sent from the acquired subject information and attribute information using a trained model trained based on the subject feature information, attribute information, and response indicator information.
[0015] Here, the attribute information includes information such as age, sex, occupation, address, place of origin, income, and account balance.
[0016] In addition, the response indicators include an open rate indicating the probability that a user will open an email, an unopened rate indicating the probability that a user will not open an email, a selection rate indicating the probability that a user will select content such as link information contained in an email, a non-selection rate indicating the probability that a user will not select content such as link information contained in an email, a conversion rate which is the probability that a user will take the desired action on the linked site that is the outcome of the site, and a non-conversion rate which is the probability that a user will not take the desired action on the linked site that is the outcome of the site.
[0017] In addition, the manner of determining whether or not a symbol is present includes a manner in which if any symbol is included, it is determined to be "present" and if none is included, it is determined to be "absent"; a manner in which if a specific symbol (for example, !, ?, etc.) is included, it is determined to be "present" and if none is included, it is determined to be "absent"; a manner in which if a specific combination of symbols (for example, the combination of ☆ and !) is included, it is determined to be "present" and if none are included, it is determined to be "absent" (in this case, if only one of the combinations (for example, !) is included, it is also determined to be "absent");
[0018] [Invention 3] Furthermore, the email subject setting support system of Invention 3 is the email subject setting support system of Invention 2, wherein the feature information is feature information regarding the proportion of character types or the presence or absence of symbols in the subject of the email that affect a response index that indicates the degree of the user's response to the email.
[0019] With this configuration, the estimation means estimates the response index using a trained model that has been trained based on subject feature information, attribute information, and response index information, including feature information regarding the proportion of character types or the presence or absence of symbols in the subject of an email that affect the response index indicating the degree of the user's response to the email. Effect of the Invention
[0020] As described above, according to the email element setting system of the first aspect, it is expected that elements having high evaluations of reaction indices indicating the degree of a user's reaction to an email can be set.
[0021] On the other hand, according to the email subject setting support system of Invention 2, it is expected that a subject with a high evaluation of the response index can be set.
[0022] Furthermore, according to the email subject setting support system of Invention 3, it is expected that the subject candidates that include specific character types or symbols will be able to be set with a high evaluation of the response index. [Brief description of the drawings]
[0023] [Figure 1] FIG. 1 is a block diagram showing a configuration of a network system. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of a subject evaluation service providing server 100. [Diagram 3] 13A and 13B are diagrams showing the data structures of a user management table 400, a model creation information table 410, a subject information table 420, an AI-compatible information management table 430, and a subject evaluation AI model management table 440. [Figure 4] 13 is a sequence flow showing operations of the subject evaluation service providing server 100, the data control server 110, and the email subject evaluation server 120 when creating a subject evaluation AI model. [Diagram 5] 13 is a sequence flow showing the operations of the user terminal 200, the subject evaluation service providing server 100, the data control server 110, and the mail subject evaluation server 120 when evaluating subject candidates. [Figure 6] FIG. 13 is a diagram showing a case where subject candidates are generated using a large-scale language model server 130. [Figure 7] FIG. 13 is a diagram showing a case where subject candidates are evaluated. [Figure 8] 13 is a flowchart showing an evaluation subject file transfer process. [Figure 9] 13 is a flowchart showing an evaluation result file combining process. [Figure 10] FIG. 13 is a diagram showing a case where subject candidates are generated from the email body. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] An embodiment of the present invention will now be described with reference to Figures 1 to 9.
[0025] 〔composition〕 First, the configuration of this embodiment will be described. FIG. 1 is a block diagram showing the configuration of a network system.
[0026] As shown in FIG. 1, the Internet 199 is connected to a subject evaluation service providing server 100 that provides an email subject evaluation service, a data control server 110 that controls the transmission and reception of data related to a subject evaluation AI model that evaluates email subjects (hereinafter abbreviated as "subjects"), an email subject evaluation server 120 that generates a subject evaluation AI model and evaluates subjects using the generated subject evaluation AI model, and a large-scale language model server 130 that generates subjects using a large language model.
[0027] A user terminal 200 is connected to the Internet 199 for use by a user who uses the email subject evaluation service.
[0028] [Hardware configuration of the subject evaluation service providing server 100] Next, the hardware configuration of the subject evaluation service providing server 100 will be described.
[0029] FIG. 2 is a diagram showing the hardware configuration of the subject evaluation service providing server 100. As shown in FIG. As shown in Fig. 2, the subject evaluation service providing server 100 is composed of a CPU (Central Processing Unit) 30 that controls the calculations and the entire system based on a control program, a ROM (Read Only Memory) 32 in which the control program and the like for the CPU 30 are stored in advance in a specified area, a RAM (Random Access Memory) 34 for storing data read from the ROM 32 and the like and calculation results required in the calculation process of the CPU 30, and an I / F (InterFace) 38 that mediates the input and output of data to and from external devices, and these are connected to each other so that data can be sent and received by a bus 39, which is a signal line for transferring data. The I / F 38 also includes the function of a network adapter.
[0030] The I / F 38 is connected to external devices such as an input device 40 consisting of a keyboard, mouse, etc. that can input data as a human interface, a storage device 42 that stores data, tables, etc. as files, a display device 44 that displays a screen based on an image signal, and a signal line for connecting to the Internet 199.
[0031] The data control server 110 is configured to have the same hardware configuration and storage device 50 as the subject evaluation service providing server 100 .
[0032] The email subject evaluation server 120 is configured to have the same hardware configuration and storage device 60 as the subject evaluation service providing server 100 .
[0033] The large-scale language model server 130 is configured to have the same hardware configuration as the subject evaluation service providing server 100. The large-scale language model is a deep learning model that pre-learns from a huge amount of data what is called a language model that models human spoken words based on their occurrence probability. When the large-scale language model server 130 receives a request, it uses the large-scale language model to statistically estimate the generation probability of the next word from the sentence included in the received request, and transmits the estimation result to the request source. As the large-scale language model, for example, known technologies described in the Internet sites "https: / / chatgpt-lab.com / n / n418d3aa56f0b" and "https: / / agirobots.com / chatgpt-mechanism-and-problem / " can be adopted.
[0034] In addition, some or all of the subject evaluation service providing server 100, the data control server 110, the email subject evaluation server 120, and the large-scale language model server 130 may be configured as virtual servers on a server providing a cloud computing service.
[0035] [Hardware configuration of user terminal 200] The user terminal 200 is composed of a terminal assumed to be a stationary type such as the subject evaluation service providing server 100, or a mobile terminal such as a smartphone or tablet. In the latter case, the hardware configuration is composed of a CPU, a ROM that stores programs such as application software and data for executing the programs, a RAM for storing data read from the ROM or the results of calculations required in the CPU's calculation process, and an I / F that mediates the input and output of data to various attached devices. These are connected to each other and capable of sending and receiving data by signal lines for transferring data.
[0036] To the I / F, various accessory devices are connected, such as a touch panel display input device that allows data input and can display a screen based on an image signal, a storage device, a wireless communication device, a video camera, and the like.
[0037] [About the various tables] Next, the various tables stored in the storage device 42, the storage device 50, and the storage device 60 will be described.
[0038] FIG. 3 is a diagram showing the data structures of a user management table 400, a model creation information table 410, a subject information table 420, an AI-compatible information management table 430, and a subject evaluation AI model management table 440.
[0039] The storage device 42 stores a user management table 400 shown in FIG. 3(a), one record is registered for each user ID in the user management table 400. Each record includes the user ID, subject creation file ID, evaluation subject file ID, address, representative name, phone number, representative email address, login information, creation date, update date, and other information.
[0040] Here, the user ID is an ID for identifying a user who uses the email subject evaluation service, the subject creation file ID is an ID for identifying a data file such as a CSV file that includes email body data for creating subject candidates, and the evaluation subject file ID is an ID for identifying a data file such as a CSV file of evaluation data for subject candidates requested by a user corresponding to the user ID.
[0041] The storage device 42 also stores a model creation information table 410 shown in FIG.
[0042] As shown in Fig. 3(b), one record is registered for each user ID in the model creation information table 410. Each record includes an administrator ID, a subject evaluation AI model ID, a subject file ID for model creation, a subject file for model creation, a creation date, an update date, and other information.
[0043] Here, the administrator ID is an ID for identifying an administrator of a system that provides an email subject evaluation service, and the subject evaluation AI model ID is an ID for identifying a subject evaluation AI model that evaluates subjects. The subject file ID for model creation is an ID for identifying a data file such as a CSV file of data for creating a subject evaluation AI model, and the subject file for model creation is a data file of data for creating a subject evaluation AI model, which is a data file created from past performance data, etc. For example, it is a file that includes data that has been converted into feature quantities such as the number of characters of each subject delivered in the past, the ratio of kanji, hiragana, and katakana, the presence or absence of symbols, and words used, data on response indicators that indicate the degree of response of the recipient user to the email when delivered with each subject, and data on the attribute information of the recipient.
[0044] In addition, the response indicators include the email open rate, non-open rate, the selection rate of the content in the email, the non-selection rate, the conversion rate at the link destination clicked from the link information in the email, the non-conversion rate, etc. In the following, the response indicator will be explained using the "email open rate" as an example.
[0045] The storage device 42 also stores a subject information table 420 shown in FIG. As shown in Fig. 3(c), one record is registered for each user ID in the subject information table 420. Each record includes the user ID, subject creation file ID, evaluation subject file ID, evaluation result ID, subject creation file, evaluation subject file, evaluation result file, creation date, update date, and other information.
[0046] Here, the evaluation result ID is an ID for identifying a file of the evaluation result (also called a predicted result or estimated result) by the subject evaluation AI model, and the evaluation result file is a data file such as a CSV file of the evaluation result data. Here, the evaluation result data is set as a score (for example, out of 1000 points) of the open rate of emails when each candidate subject is used. A higher score indicates that it is evaluated as having a higher possibility of an open rate.
[0047] The subject creation file is a data file that includes multiple subject candidates for each email that the user of the email subject evaluation service is going to send, the email body of each email, keywords to be included in the subject, attribute information of the recipient (age, gender, etc.), etc. The evaluation subject file is a file that includes data on multiple subject candidates, feature data extracted from this data, attribute information of the recipient, delivery date and time, email body, etc.
[0048] The storage device 50 stores an AI correspondence information management table 430 shown in FIG. As shown in Fig. 3(d), one record is registered for each administrator ID in the AI-compatible information management table 430. Each record includes the administrator ID, the subject evaluation AI model ID, file transfer protocol information, the creation date, the update date, and other information.
[0049] Here, the AI-supported information management table 430 is a table for linking a subject evaluation AI model under the management of an administrator corresponding to an administrator ID with an evaluation subject file before evaluation. Specifically, it is a table for linking a subject evaluation AI model ID with a file transfer protocol used to transfer an evaluation subject file.
[0050] The file transfer protocol information is information about a file transfer protocol used for transmitting and receiving various files such as evaluation subject files, and is, for example, information such as a protocol name, a host name, a port number, user information, a password, etc. Examples of file transfer protocols include SFTP (SSH File Transfer Protocol) and FTPS (File Transfer Protocol over SSL / TLS).
[0051] The storage device 60 stores a subject line evaluation AI model management table 440 shown in FIG.
[0052] As shown in Fig. 3(e), one record is registered for each administrator ID in the subject evaluation AI model management table 440. Each record includes the administrator ID, subject evaluation AI model ID, subject evaluation AI model information, creation date, update date, and other information.
[0053] [Operation] [Sequence for creating a subject evaluation AI model] First, we will explain the sequence for creating a subject evaluation AI model.
[0054] FIG. 4 is a sequence flow showing the operations of the subject evaluation service providing server 100, the data control server 110, and the mail subject evaluation server 120 when creating a subject evaluation AI model.
[0055] The CPU 30 of the subject evaluation service providing server 100 is composed of an MPU (Micro-Processing Unit) or the like, and starts a program that performs data generation and the like for creating a subject evaluation AI model stored in a predetermined area of the ROM 32, and executes the following processes according to the program. The CPU of the data control server 110 starts a program that performs transfer control and the like for data used in the creation process of the subject evaluation AI model stored in a predetermined area of the ROM, and executes the following processes according to the program. The CPU of the email subject evaluation server 120 starts a program that performs creation processing of a subject evaluation AI model based on model creation data stored in a predetermined area of the ROM, and executes the following processes according to the program.
[0056] First, the subject line evaluation service providing server 100 generates data for creating a subject line evaluation AI model based on past performance data (S100).
[0057] Here, the past performance data is performance data of emails sent in the past, and includes information on the subject, email body, delivery date and time, recipient attribute information (age, gender, occupation (company executive, company employee, self-employed, etc.), annual income, number of times related web pages are viewed, account balance, etc.), and open rate. Specifically, based on the performance data, the subject evaluation service providing server 100 digitizes the number of characters in the subject, the percentage of kanji, hiragana, and katakana contained in the subject, the presence or absence of symbols such as "!" and "?", and the words used as features.
[0058] For example, if the subject is "Today's average stock price rebounds," the number of characters will be "10," the number of kanji characters will be "8," the number of hiragana characters will be "2," and the number of katakana characters will be "0," so the percentage of kanji characters will be "80%, the percentage of hiragana characters will be "20%, and the percentage of katakana characters will be "0%" calculated as features. In addition, the presence or absence of symbols will be "none," and the words used, "today," "average stock price," and "rebound," will be extracted as features.
[0059] Then, data for creating a subject line evaluation AI model is generated, including these feature data, subject lines, email bodies, delivery dates and times, recipient attribute information, and information on open rates.
[0060] Next, the subject evaluation service providing server 100 transmits the generated subject evaluation AI model creation data to the data control server 110 (S102).
[0061] When the data control server 110 receives the subject evaluation AI model creation data from the subject evaluation service providing server 100 (S104), it transmits the received subject evaluation AI model creation data to the email subject evaluation server 120 (S106).
[0062] When the email subject evaluation server 120 receives the subject evaluation AI model creation data from the data control server 110 (S108), it creates a subject evaluation AI model based on the received subject evaluation AI model creation data (S110).
[0063] Here, in creating the subject line evaluation AI model, machine learning is performed based on data for creating the subject line evaluation AI model to model the relationship between each feature amount or attribute information and the open rate. At this time, learning is performed using past performance information (open rate, etc.) as training data. In this embodiment, this machine learning is used to construct a model that inputs the subject line and attribute information of the delivery destination and outputs a score related to the open rate. The score is a larger value (for example, the upper limit is 1000) the more likely it is to be opened.
[0064] Examples of learning algorithms that can be used for the subject evaluation AI model include linear regression analysis, multivariate adaptive regression splines (MARS), support vector regression (SVR), regression trees, model trees, genetic programs, binary classification, logistic regression, k-nearest neighbors, support vector machines, decision trees, random forests, and neural networks.
[0065] When the creation of the subject evaluation AI model is completed, the email subject evaluation server 120 transmits a creation completion notification to the data control server 110 (S112).
[0066] When the data control server 110 receives a creation completion notification from the email subject evaluation server 120 (S114), it links the created subject evaluation AI model with the evaluation subject file (S116). Then, it transmits the creation completion notification to the subject evaluation service providing server 100 (S118). The subject evaluation service providing server 100 receives the creation completion notification from the data control server 110 (S120). This completes the creation of the subject evaluation AI model.
[0067] Here, data is exchanged between the servers using, for example, an encrypted communication technique, and in this embodiment, data is exchanged using SFTP.
[0068] [Sequence for evaluating subject candidates] Next, we will explain the sequence when evaluating subject candidates using the subject evaluation AI model.
[0069] FIG. 5 is a sequence flow showing the operations of the user terminal 200, the subject evaluation service providing server 100, the data control server 110, and the mail subject evaluation server 120 when evaluating subject candidates.
[0070] The CPU 30 starts a predetermined program stored in a predetermined area of the ROM 32, and the subject evaluation service providing server 100 executes the following processing according to the program. In addition, the user terminal 200 has installed therein a program for requesting evaluation of subject candidates by communicating with the subject evaluation service providing server 100. The user terminal 200 starts a predetermined program stored in a predetermined area of the ROM, and executes the following processing according to the program. In addition, the data control server 110 has installed therein a program for performing transfer control of data used in evaluation processing of subject candidates by communicating with the subject evaluation service providing server 100 and the email subject evaluation server 120. The data control server 110 starts a predetermined program stored in a predetermined area of the ROM, and executes the following processing according to the program. In addition, the email subject evaluation server 120 has installed therein a program for performing evaluation processing using a subject evaluation AI model based on an evaluation subject file from the data control server 110. The email subject evaluation server 120 starts a predetermined program stored in a predetermined area of the ROM, and executes the following processing according to the program.
[0071] First, the user accesses (logs in to) a web page for using the service provided by the subject evaluation service providing server 100 using the user terminal 200. That is, the user is authenticated in this login process.
[0072] After logging in, the user terminal 200 transmits a subject creation file including a request for evaluating a subject candidate to the subject evaluation service providing server 100 (S200).
[0073] Here, the user corresponds to, for example, a service provider that provides services such as Internet mail order services, stock trading services, FX trading services, etc. Such users frequently send e-mails to registered users informing them of, for example, sales, stock price trends, and exchange rate conditions.
[0074] When the subject evaluation service providing server 100 receives a subject creation file from the user terminal 200 (S202), it creates an evaluation subject file based on the received subject creation file (S204).
[0075] Here, the subject creation file corresponds to, for example, a file containing data such as multiple subject candidates for one email body, the email body to be delivered, attribute information of the delivery destination (recipient user), and the scheduled delivery date and time, or a file from this file excluding the data of multiple subject candidates.
[0076] FIG. 6 is a diagram showing a case where subject candidates are generated using the large-scale language model server 130.
[0077] The subject evaluation service providing server 100 has a function of automatically generating subject candidates from the body of the email to be delivered, for example, when no subject candidates are included in the subject creation file.
[0078] When the URL of an article or page is included in the email body to introduce the article or page, the subject evaluation service providing server 100 acquires the URL from the email body, generates a request including the acquired URL and a request to generate keywords representative of the article or page, and sends the request to the large-scale language model server 130.
[0079] For example, if you plan to send an email introducing a page called "Easy Foreign Currency Deposit Class," the request will be generated as shown in the first column of Figure 6, containing the URL "https: / / www.aaa.com / news.html" and the generation request "Please think of three subject lines of about 20 characters each to use in introducing the above site in an email. Please use bullet points. Please put $ at the beginning of each subject line. Please include an important keyword in brackets at the beginning."
[0080] As another example, if you plan to send an email introducing a page called "Foreign Currency Fixed Term Deposit Interest Rate Special Feature," the request will be generated as shown in the second column of Figure 6, containing the URL "https: / / www.bbb.com / news.html" and the generation request "Please think of three subject lines of about 20 characters each to use in introducing the above site in an email. Please use bullet points. Please put $ at the beginning of each subject line. Please include an important keyword in brackets at the beginning."
[0081] As another example, if you plan to send an email introducing a page called "Foreign Currency Deposit Profit and Loss Simulation," the request will be generated as shown in the third column of Figure 6, containing the URL "https: / / www.ccc.com / news.html" and the generation request "Please think of three subject lines of about 20 characters each to use in introducing the above site in an email. Please use bullet points. Please put $ at the beginning of each subject line. Please include an important keyword in brackets at the beginning."
[0082] The subject evaluation service providing server 100 acquires, as subject candidates, the estimation results output from the large-scale language model server 130 in response to the transmitted request.
[0083] In the example of “Foreign currency fixed term deposit interest rate special feature,” “$[Beginners must see] How to start a foreign currency deposit!$[For foreign currency beginners] Starting foreign currency deposits with foreign exchange$[For foreign currency beginners must see] Recommendations for starting foreign currency deposits with foreign exchange” is output as the inferred result, and therefore the three “Beginners must see how to start a foreign currency deposit!”, “For foreign currency beginners starting foreign currency deposits with foreign exchange”, and “Recommendations for starting foreign currency deposits with foreign exchange” are respectively obtained as subject candidates.
[0084] In the example of "Special feature on interest rates on foreign currency fixed term deposits," "Getting closer to the appeal of $[foreign currency deposits]! A chance to learn about $[the world's money situation]! $A must-see for those interested in $[global investments]!" is output as the inferred result, so the three items "Getting closer to the appeal of foreign currency deposits!", "A chance to learn about the world's money situation!", and "A must-see for those interested in global investments!" are obtained as subject candidates.
[0085] In the example of "Foreign currency deposit profit and loss simulation," "$The [foreign currency deposit simulation] tool is useful!$Consider foreign currency deposits with [exchange rate forecasts]$Find advantageous accounts with [foreign currency deposit interest rate comparisons]" is output as the inferred result, so the three items "Foreign currency deposit simulation tools are useful!", "Consider foreign currency deposits with exchange rate forecasts," and "Find advantageous accounts by comparing foreign currency deposit interest rates" are obtained as subject candidates.
[0086] Then, the subject evaluation service providing server 100 generates an evaluation subject file including data such as subject candidates, the email body, delivery date and time, and attribute information of the delivery destination.
[0087] The subject evaluation service providing server 100 transmits the generated evaluation subject file to the data control server 110 (S206).
[0088] When the data control server 110 receives the evaluation subject file from the subject evaluation service providing server 100 (S208), it executes a data transfer process according to the data amount of the received evaluation subject file (S210). Note that, by the data transfer process, the evaluation subject file is sent to the email subject evaluation server 120 as one or more files of a predetermined data amount (for example, 1 GB (gigabyte)) or less.
[0089] Here, in the email subject evaluation server 120, in order to reduce the processing load of the evaluation process in the subject evaluation AI model, the evaluation process is performed for each evaluation subject file having a predetermined data amount or less.
[0090] When the mail subject evaluation server 120 receives one or more evaluation subject files from the data control server 110, it stores the received files in the storage device 60 (S212).
[0091] On the other hand, when the transfer of the evaluation subject file is completed, the data control server 110 transmits an evaluation start instruction to the mail subject evaluation server 120 (S214).
[0092] When the email subject evaluation server 120 receives an evaluation start instruction from the data control server 110, it executes evaluation processing of subject candidates using the corresponding subject evaluation AI model based on the received evaluation subject file (S216). Here, since the amount of data that can be processed at one time is limited, the evaluation processing is performed for each evaluation subject file in turn.
[0093] FIG. 7 is a diagram showing a case where subject candidates are evaluated. The subject candidate evaluation is performed, for example, by inputting the subject candidate, delivery date and time, email body, and delivery destination attribute information contained in the evaluation subject file into the subject evaluation AI model in sequence, as shown in Figure 7. As a result, the trained subject evaluation AI model outputs a score related to the open rate for the input subject candidate and attribute information. This score is a score indicating the possibility that a sent email will be opened for a combination of subject candidate and attribute information, with a higher value indicating a higher possibility.
[0094] In the example of Figure 7, an email with the subject "Today's average stock price rebounds" sent to a male office worker in his 20s has a score of "800" indicating the evaluation of the open rate, while an email with the subject "Poor direction even toward the closing" has a score of "723" indicating the evaluation of the open rate. In other words, for male office workers in their 20s, "Today's average stock price rebounds" is evaluated as being more likely to be opened than "Poor direction even toward the closing."
[0095] Also, when an email was sent to a female office worker in her 20s with the subject "Today's average stock price rebounds," the score indicating the open rate was 621, while an email with the subject "Poor direction even as markets close" received a score of 631. In other words, for female office workers in their 20s, "Poor direction even as markets close" was evaluated as being more likely to be opened than "Today's average stock price rebounds."
[0096] In addition, when an email was sent to a self-employed woman in her 40s with the subject "Today's average stock price rebounds," it received a score of 800, which indicates the evaluation of the open rate, while an email with the subject "Poor direction even toward the closing" received a score of 723, which indicates the evaluation of the open rate. In other words, for self-employed women in their 40s, "Today's average stock price rebounds" is evaluated as being more likely to be opened than "Poor direction even toward the closing."
[0097] In this way, in addition to the change in score due to differences in subject lines, the score of the open rate for each subject line candidate also changes depending on attributes such as gender, age, etc. This is not limited to the open rate, but is similar for other response indicators as well.
[0098] The data control server 110 periodically checks with the email subject evaluation server 120 whether the evaluation process has been completed, and when it confirms that the evaluation process has been completed (S220), it sends an evaluation result download (DL) request to the email subject evaluation server 120 (S222).
[0099] When the email subject evaluation server 120 receives an evaluation result download request from the data control server 110 (S224), it transmits to the data control server 110 one or more evaluation result files, which are data files of the evaluation results generated for each evaluation subject file of the received one or more evaluation subject files (S226).
[0100] When a plurality of evaluation result files are to be transmitted, identification information that identifies them as evaluation results of a common evaluation subject file is added to each file.
[0101] When the data control server 110 receives one or more evaluation result files from the mail subject evaluation server 120 (S228), it executes a file combination process (S230). After that, it transmits the evaluation result file that has undergone the combination process to the subject evaluation service providing server 100 (S232).
[0102] When the subject evaluation service providing server 100 receives the evaluation result file from the data control server 110 (S234), it transmits the received evaluation result file to the user terminal 200 (S236).
[0103] The user terminal 200 receives the evaluation result file from the subject line evaluation service providing server 100 (S238). This completes the evaluation process of the subject line candidates.
[0104] As a result, the user side who receives the evaluation result file can check the score of the open rate of each subject candidate, and set, for example, the subject candidate with the highest score as the subject to be delivered. Alternatively, an information processing terminal such as the user terminal 200 may read the evaluation result file, select the subject candidate with the highest score corresponding to each delivery destination user, and execute a program to automatically deliver an email with the selected subject candidate as the subject.
[0105] [Evaluation subject file transfer process] FIG. 8 is a flowchart showing the evaluation subject file transfer process.
[0106] The evaluation subject file transfer process is executed by the data control server 110. When the evaluation subject file transfer process is started in step S210, the process first proceeds to step S300 as shown in FIG.
[0107] In step S300, it is determined whether the data volume of the received evaluation subject file exceeds a predetermined data volume (e.g., 1 GB), and if it is determined that it does (YES), the process proceeds to step S302, and if it is determined that it does not (NO), the process proceeds to step S308.
[0108] If the process has proceeded to step S302, a process is executed to divide the received evaluation subject file into a plurality of files each having a predetermined data amount or less, and the process proceeds to step S304.
[0109] Here, the splitting process may, for example, split the received file into a file with a predetermined amount of data, which is the quotient of the amount of data obtained by dividing the amount of data in the received file by a predetermined amount of data, and if the amount of data is not divisible, into a file with the remainder of the amount of data, or may split the file into multiple files of equal or approximately equal amounts of data.
[0110] In addition, identification information is added to each divided file to identify that the divided files were originally a common file.
[0111] In step S304, the multiple files divided in step S302 are sequentially transmitted to the mail subject evaluation server 120, and the process proceeds to step S306.
[0112] In step S306, it is determined whether or not the transmission of the file has been completed. If it is determined that the transmission of the file has been completed (YES), the series of processes is terminated and the original process is resumed. If it is determined that the transmission of the file has not been completed (NO), the process proceeds to step S304.
[0113] On the other hand, if the amount of data in the evaluation subject file received in step S300 is equal to or less than the predetermined amount of data and the process proceeds to step S308, the received evaluation subject file is sent as is to the email subject evaluation server 120 and the process proceeds to step S306.
[0114] [Evaluation result file merging process] FIG. 9 is a flowchart showing the evaluation result file combining process.
[0115] The evaluation result file combining process is executed by the data control server 110. When the evaluation result file combining process is started in step S230, the process first proceeds to step S400 as shown in FIG.
[0116] In step S400, it is determined whether or not multiple evaluation result files have been received. If it is determined that multiple evaluation result files have been received (YES), the process proceeds to step S402. If it is determined that multiple evaluation result files have been received (NO), the process proceeds to step S406.
[0117] If the process proceeds to step S402, the received multiple evaluation result files are combined into one file, and the process proceeds to step S404.
[0118] For example, if the evaluation result file is a CSV file, multiple files are vertically linked to generate one evaluation result file.
[0119] In step S404, the single combined evaluation result file is sent to the subject evaluation service providing server 100, the series of processes is terminated, and the process returns to the original process.
[0120] On the other hand, if there are not multiple evaluation result files received in step S400 and the process proceeds to step S406, the received evaluation result file is sent directly to the subject evaluation service providing server 100, the series of processes is terminated, and the process returns to the original process.
[0121] 〔effect〕 Next, the effects of this embodiment will be described. In this embodiment, the subject evaluation service providing server 100 transmits a request including a request for generating the URL of an article or page and a keyword representative of the article or page to the large-scale language model server 130, obtains an estimation result output from the large-scale language model server 130 in response to the request, and sets a subject based on the obtained estimation result. As an example of the effect, when subject candidates were generated using the large-scale language model server 130, the email open rate was significantly improved from 50.2% to 59.6%, compared to when they were not generated.
[0122] This will enable you to set subject lines that have a high response index rating. Furthermore, in this embodiment, the subject evaluation service providing server 100 generates an evaluation subject file including multiple subject candidates and attribute information of the delivery destination user, and transmits the generated evaluation subject file to the data control server, and the data control server 110 transmits the evaluation subject file to the email subject evaluation server 120, receives an evaluation result file from the email subject evaluation server 120, and transmits the evaluation result file to the subject evaluation service providing server 100, and the email subject evaluation server 120 receives the evaluation subject file from the data control server 110, evaluates the subject candidates included in the evaluation subject file using the subject evaluation AI model, and transmits an evaluation result file including the evaluation result to the data control server 110. In addition, as the subject evaluation AI model, a trained AI model created by machine learning based on data for creating a subject evaluation AI model including data on features including the proportion of character types and the presence or absence of symbols that affect response indicators such as the open rate of emails included in the subject, the subject, the email body, the delivery date and time, attribute information of the delivery destination, and information on the open rate, etc., was used.
[0123] This will enable us to set a subject line candidate that has a high response index rating among candidates that contain specific characters or symbols. In addition, we can perform evaluation according to the attributes of the recipient.
[0124] Furthermore, in this embodiment, when the data control server 110 receives an evaluation subject file that exceeds a predetermined data amount, the data control server 110 executes a file division process to divide the evaluation subject file into a plurality of files each having a predetermined data amount or less, and transmits the divided files to the email subject evaluation server 120. Also, the email subject evaluation server 120 is configured to execute the evaluation process with a predetermined data amount as an upper limit in consideration of the processing load, etc.
[0125] This allows the subject evaluation service providing server 100 to create a subject file for evaluation without considering the amount of data and send it as a single file, thereby reducing the effort required for the subject evaluation service providing server 100 to create multiple files of less than a specified data amount.
[0126] Furthermore, in this embodiment, when the data control server 110 receives multiple evaluation result files, it executes a process of combining the received multiple evaluation result files into a single file, and transmits the combined single evaluation result file to the subject evaluation service providing server 100.
[0127] As a result, the subject evaluation service providing server 100 does not need to combine a plurality of evaluation result files into one file, and the processing load on the subject evaluation service providing server 100 side can be reduced.
[0128] In this embodiment, step S202 corresponds to the attribute information acquisition means of invention 2, and step S204 corresponds to the request input means of invention 1 or 2, the keyword information acquisition means of invention 1, the subject information acquisition means of invention 2, or the email element setting means of invention 1. In addition, step S216 corresponds to the estimation means of invention 2, and the subject evaluation AI model corresponds to the trained model of invention 2.
[0129] [Modifications] In the above embodiment, a request including the URL of an article or page is generated, but this is not limited to the above. It is also possible to generate a request including the document information of the article or page itself instead of or in addition to the URL.
[0130] In addition, in the above embodiment and its variations, a trained AI model was used that was created by performing machine learning based on feature data including the proportion of character types and the presence or absence of symbols that affect response indicators such as the open rate of emails included in the subject line. However, without being limited to this, a trained AI model created by performing machine learning based on feature data including either the proportion of character types and the presence or absence of symbols that affect the response indicators, or feature data including keywords and other elements that affect the response indicators, can also be used.
[0131] In the above embodiment and its modified example, some or all of the keywords included in the estimation result output from the large-scale language model server 130 are acquired as subject candidates, but the present invention is not limited to this. Alternatively, some or all of the keywords generated based on the estimation result output from the large-scale language model server 130 may be acquired as subject candidates, or some or all of the keywords generated based on the estimation result and additional information output from the large-scale language model server 130 may be acquired as subject candidates. Examples of the configuration for generating keywords based on the estimation result include a configuration for providing the estimation result output from the large-scale language model server 130 as an input to a specific AI and acquiring the estimation result output from the specific AI as a keyword, and a configuration for adding or replacing an additional keyword to a keyword included in the estimation result output from the large-scale language model server 130.
[0132] In the above embodiment and its modified example, multiple subject candidates are obtained from the estimation results output from the large-scale language model server 130, each subject candidate is evaluated, and the subject candidate with the highest evaluation is set as the subject, but this is not limiting, and a configuration in which subject candidates are not evaluated can also be adopted. That is, one subject candidate is obtained from the estimation results output from the large-scale language model server 130, and a subject that includes part or all of the obtained subject candidate is set.
[0133] In addition, in the above embodiment and its modified examples, the subject is generated or set based on the estimation result output from the large-scale language model server 130. However, this is not limited to this, and the body of the email or other elements can also be generated or set.
[0134] In the above embodiment and its modified examples, subject candidates are generated using the large-scale language model server 130. However, this is not limiting, and subject candidates can also be generated from the body of an email. A modified example of step S204 is shown below.
[0135] FIG. 10 is a diagram showing a case where subject candidates are generated from the email body. The subject evaluation service providing server 100 has a function of automatically generating subject candidates from the body of the email to be delivered, for example, when no subject candidates are included in the subject creation file.
[0136] Specifically, the subject evaluation service providing server 100 extracts key sentences from the email body as subject candidates, such as, for example, as shown in FIG. 10, “Today's average stock price rebounded,” “Poor direction even toward the closing bell,” and “Attention on manufacturing industries that make effective use of assets.”
[0137] In addition, since response indicators such as the open rate of emails change depending on whether or not symbols are used, it is also possible to generate subject line candidates by adding symbols to the extracted subject line candidates, such as "Today's average stock price rebounds!", "Poor direction even as the market closes?", and "Attention on the manufacturing industry for effective use of assets."
[0138] Furthermore, in the above embodiment and its modified examples, various processes such as generating and transferring evaluation subject files, receiving evaluation result files, dividing and transferring evaluation subject files, combining and transferring evaluation result files, evaluating subject candidates using a subject evaluation AI model and transferring evaluation results may be configured to be automatically executed during times of low processing load such as overnight. In this case, the subject evaluation service providing server 100 uses a scheduler or the like to set an execution schedule for generating evaluation subject files, sending prediction subject files, receiving evaluation result files, etc.
[0139] In the above embodiment and its modified example, Japanese hiragana, katakana, and kanji are used as examples of character types, but the character types handled are not limited to these. For example, for katakana, full-width and half-width may be distinguished, and in a subject line that is not limited to Japanese but is a mixture of Japanese and other languages such as English, the proportion of character types of the other language may be extracted as characteristic information. In a subject line that is not limited to a mixture of Japanese and a native language other than Japanese and a foreign language other than Japanese, the proportion of character types of the foreign language may be extracted as characteristic information.
[0140] In the above embodiment and its modified example, when the subject evaluation service providing server 100 transmits an evaluation subject file exceeding a predetermined data amount to the data control server 110, the data control server 110 divides the received evaluation subject file into a plurality of files of a predetermined data amount or less, and transmits the divided plurality of files to the email subject evaluation server 120. When the email subject evaluation server 120 receives a plurality of subject evaluation files, it evaluates the subject candidate for each file, and transmits a plurality of evaluation result files generated for each subject evaluation file to the data control server 110. When the data control server 110 receives a plurality of evaluation result files, it combines the plurality of evaluation result files into one file, and transmits the combined single evaluation result file to the subject evaluation service providing server 100. This configuration is not limited to the above, and for example, the email subject evaluation server 120 may combine a plurality of evaluation result files into one file and transmit this single evaluation result file to the data control server 110. In this configuration, the data control server 110 does not need to perform the combining process. Also, for example, even if the evaluation subject file from the subject evaluation service providing server 100 exceeds a predetermined data amount, the data control server 110 does not perform splitting processing and sends it as is to the email subject evaluation server 120, and the email subject evaluation server 120 performs evaluation processing by splitting the file into multiple files with a data amount equal to or less than a predetermined data amount according to the processing capacity of the subject evaluation AI model, and sends multiple evaluation result files, which are evaluation result files for each file, to the data control server 110. Furthermore, the data control server 110 may be configured to combine the received multiple evaluation result files into one file and send the combined single evaluation result file to the subject evaluation service providing server 100. In this configuration, the data control server 110 does not need to perform splitting processing.
[0141] Furthermore, in the above embodiment and its modified examples, the system is realized as a network system, but the present invention is not limited to this and can also be realized as a single device or application.
[0142] In addition, in the above embodiment and its modified example, the case where the present invention is applied to a network system consisting of the Internet 199 has been described, but the present invention is not limited to this, and may be applied, for example, to a so-called intranet that performs communication in the same manner as the Internet 199. Of course, the present invention is not limited to a network that performs communication in the same manner as the Internet 199, and may be applied to a network of any communication method.
[0143] In addition, in the above embodiment and its modified examples, when executing the processes shown in the sequence flows of Figures 4 and 5 and the flowcharts of Figures 8 and 9, a case has been described in which a program pre-stored in ROM is executed. However, this is not limited to the above, and the program showing these procedures may be read into RAM from a storage medium on which the program is stored and executed.
[0144] Here, storage media refers to semiconductor storage media such as RAM and ROM, magnetic storage type storage media such as FD and HD, optically readable storage media such as CD, CDV, LD and DVD, and magnetic storage type / optically readable storage media such as MO, and includes any storage medium that can be read by a computer, regardless of the reading method (electronic, magnetic, optical, etc.).
[0145] Furthermore, in the above embodiment and its modified examples, the storage devices included in each server are used, but the present invention is not limited to this, and the system can also be configured to use an external storage device such as a database server. [Explanation of symbols]
[0146] 100...Subject evaluation service providing server, 110...Data control server, 120...Email subject evaluation server, 130...Large-scale language model server, 200...User terminal, 30...CPU, 32...ROM, 34...RAM, 38...I / F, 40...Input device, 42,50,60...Storage device, 44...Display device, 400...User management table, 410...Model creation information table, 420...Subject information table, 430...AI compatible information management table, 440...Subject evaluation AI model management table
Claims
1. a request input means for inputting a request including a request for generating document information or link information relating to a link destination of the document information and a keyword representative of the document information into the large-scale language model; a keyword information acquisition means for acquiring keyword information output from the large-scale language model in response to the request; a mail element setting means for setting a subject, a body or other elements of a mail based on the keyword information acquired by said keyword information acquisition means.
2. a request input means for inputting a request including a request for generating document information or link information relating to a link destination of the document information and a plurality of keywords representative of the document information into the large-scale language model; a subject information acquisition means for acquiring a plurality of pieces of keyword information output from the large-scale language model in response to the request as subject information related to the subject, the subject information being a candidate for the subject of the email; An attribute information acquisition means for acquiring attribute information relating to attributes of a user to whom an email is to be sent; an estimation means for estimating a response index of each candidate related to the subject information for a user to whom the email is to be sent, from the subject information acquired by the subject information acquisition means and the attribute information acquired by the attribute information acquisition means, using a trained model that has been trained based on subject feature information including feature information in the subject of the email that affects a response index indicating the degree of the user's response to the email, attribute information regarding the attributes of the user who sent the email, and response index information regarding the response index of the email.
3. In claim 2, An email subject setting support system characterized in that the characteristic information is characteristic information regarding the proportion of characters or the presence or absence of symbols in the subject of the email that affect a response index that indicates the degree of the user's response to the email.
Citation Information
Patent Citations
Email title generation model training method and email title generation method
CN113824624A
Apparatus, method and program for creating of link collection
JP2003016082A
Information processing device
JP2019020916A
Mail proposition device, mail proposition system, mail proposition program, and method for proposing mail
JP2022014529A
Generating message effectiveness predictions and insights
US20210004437A1
