AI content summary generation method and device

Abstract

The embodiment of the present invention provides an AI content summary generation method, device and electronic device, which belongs to the field of data processing technology. The method includes: performing data preprocessing operations on raw data to obtain standard data DB; performing data screening operations on the standard data DB based on the constructed content screening vector Sn=[Ft,Hc,Rc,Fl] and the evaluation function Fp to obtain screening data DS stored in the form of a string; tokenizing the screening data DS through the token library provided by the AI ​​processing engine to obtain a token list TL; generating a model matching parameter PC corresponding to the verification result Vy; after the API processing interface receives the text to be processed Tdc, the text to be processed Tdc is sent to the summary processing engine ZP, so as to generate a content summary Tnc corresponding to the text to be processed Tdc in the summary processing engine ZP. The present invention can improve the accuracy and adaptability of the content summary.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an AI content summary generation method, device and electronic device. Background Art

[0002] AI content summarization is the process of using artificial intelligence technology to extract key information from longer texts and generate a concise summary. This technology can automatically identify the main points and important data in a document, and then present this information in a coherent and easy-to-understand way. AI content summarization is often used in news articles, research papers, reports and other types of long texts. The purpose is to help readers quickly understand the core content of the document and save time reading and processing large amounts of information.

[0003] The existing technology has the following problems in the process of AI content summary generation:

[0004] 1. Poor model generalization ability: The AI ​​model does not perform as expected when processing unseen data, resulting in insufficient accuracy of the AI ​​summary content.

[0005] 2. High consumption of computing resources: Deep learning models, especially large neural network models, often require a lot of computing resources for training, which is a challenge for small companies or individual developers with limited resources.

[0006] 3. Insufficient multilingual and cross-cultural support: Currently, many AI systems are developed mainly for specific languages ​​(such as English), and lack support for other languages, especially niche languages.

[0007] The above problems become technical problems that need to be solved. Summary of the invention

[0008] In view of this, an embodiment of the present invention provides an AI content summary generation method, device and electronic device, which at least partially solve the problems existing in the prior art.

[0009] In a first aspect, an embodiment of the present invention provides an AI content summary generation method, comprising:

[0010] Obtain the original data DY of n users from the social media platform, perform data preprocessing operations on the original data, and obtain standard data DB={DB1, DB2, ...DBn};

[0011] Based on the constructed content screening vector Sn=[Ft,Hc,Rc,Fl] and the evaluation function Fp, a data screening operation is performed on the standard data DB to obtain screening data DS={DS1,DS2,…DSn} stored in the form of a string, where Ft represents the user's creation time, Hc represents the content interaction amount, Rc represents the content relevance value, and Fl represents the content attention amount;

[0012] Through the token library provided by the AI ​​processing engine, the screening data DS is tokenized to obtain a token list TL, the length of the token list TL is calculated to obtain the number of tokens Nt=|TL|, when the number of tokens Nt is less than the first threshold N0, the compression rate Ctk=Nt / N0 of the token is calculated, when the compression rate Ctk is less than the second threshold C0, the token list is token verified to obtain a verification result Vy;

[0013] Generate a model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy, where Tx represents the temperature coefficient T, Ls represents the maximum output length L, and Cc represents the sampling strategy P;

[0014] The model matching parameters PC are used to construct a summary processing engine ZP based on the AI ​​processing engine, and an API processing interface is configured for the summary processing engine ZP. After the API processing interface receives the text to be processed Tdc, the text to be processed Tdc is sent to the summary processing engine ZP, so as to generate a content summary Tnc corresponding to the text to be processed Tdc in the summary processing engine ZP.

[0015] According to a specific implementation of an embodiment of the present invention, the method of obtaining the original data DY of n users from the social media platform and performing data preprocessing operations on the original data to obtain standard data DB={DB1, DB2, ...DBn} includes:

[0016] After determining the data source and data capture API interface, formulate the time range and frequency of data collection to obtain the original data DY of target users from social media platforms;

[0017] Remove empty records contained in the original data, filter out non-Chinese characters, and deduplicate the data to obtain cleaned data;

[0018] Perform spelling correction and synonym replacement operations on the cleaned data in a unified text format to obtain unified data;

[0019] Classifying the unified data into different categories according to the subject and sentiment tendency of the text in the unified data to form classified data;

[0020] According to a preset data format, the classified data is structured to obtain standard data DB={DB1, DB2, ...DBn}.

[0021] According to a specific implementation of an embodiment of the present invention, the data screening operation is performed on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain the screening data DS={DS1, DS2, ...DSn} stored in the form of a string, including:

[0022] Based on the evaluation function Fp, the comprehensive score of each standard data DBi is evaluated, and the content screening vector corresponding to DBi is , for each standard data DBi, calculate its comprehensive score Si:

[0023]

[0024] is the weight coefficient.

[0025] According to a specific implementation of an embodiment of the present invention, the data screening operation is performed on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain the screening data DS={DS1, DS2, ...DSn} stored in the form of a string, and further includes:

[0026] Sort all standard data according to the comprehensive score Si, and select data with a score higher than the threshold T0 as the screening data DS:

[0027]

[0028] The filtered data DS is converted into a string and stored.

[0029] According to a specific implementation of the embodiment of the present invention, the token library provided by the AI ​​processing engine performs a tokenization operation on the screening data DS to obtain a token list TL, including:

[0030] By calling the loading function, the selected token library is loaded into the memory, and the search and matching operations in the tokenization process are performed in the memory;

[0031] Splitting each text DSi contained in the screening data into independent phrases, mapping each phrase after word segmentation to a unique identifier in a token library, and forming each independent phrase into an independent token;

[0032] All tokens in the screening data DS are combined in order into a token list TL={TL1, TL2,…, TLn}, and each element TLi in the token list corresponds to a record DSi in the screening data DS.

[0033] According to a specific implementation of the embodiment of the present invention, when the compression rate Ctk is less than the second threshold C0, token verification is performed on the token list to obtain a verification result Vy, including:

[0034] Check each token in the token list TL and remove invalid or incorrect tokens;

[0035] Analyze the keywords in the token list to confirm whether the token list and the DS topic of the filtered data meet the consistency requirements;

[0036] According to the verification process, a verification result Vy is generated, and the verification result Vy is represented by a Boolean value.

[0037] According to a specific implementation of the embodiment of the present invention, the generating of the model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy includes:

[0038] If the verification result Vy is True, a low temperature coefficient is used, and if the verification result Vy is False, a high temperature coefficient is used;

[0039] If the verification result Vy is True, use the default value as the maximum output length. If the verification result Vy is False, increase the maximum output length.

[0040] If the verification result Vy is True, the text is collected using the beam search method. If the verification result Vy is False, the Top-K sampling method is used for sampling.

[0041] According to a specific implementation of an embodiment of the present invention, the use of the model matching parameter PC to construct a summary processing engine ZP based on an AI processing engine includes:

[0042] During model generation, the temperature coefficients are adjusted to control the diversity and certainty of the output;

[0043] Set the maximum output length Ls to limit the maximum length of the generated summary;

[0044] Based on the Cc value, set the corresponding sampling strategy.

[0045] In a second aspect, an embodiment of the present invention provides an AI content summary generation device, including:

[0046] The standard module obtains the original data DY of n users from the social media platform, performs data preprocessing operations on the original data, and obtains standard data DB={DB1, DB2, ...DBn};

[0047] A screening module performs a data screening operation on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain screening data DS={DS1, DS2, ...DSn} stored in a string form, where Ft represents the user's creation time, Hc represents the content interaction amount, Rc represents the content relevance value, and Fl represents the content attention amount;

[0048] The calculation module tokenizes the screening data DS through the token library provided by the AI ​​processing engine to obtain a token list TL, calculates the length of the token list TL, obtains the number of tokens Nt=|TL|, and when the number of tokens Nt is less than a first threshold value N0, calculates the compression rate Ctk=Nt / N0 of the token, and when the compression rate Ctk is less than a second threshold value C0, performs token verification on the token list to obtain a verification result Vy;

[0049] A matching module generates a model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy, where Tx represents the temperature coefficient T, Ls represents the maximum output length L, and Cc represents the sampling strategy P;

[0050] A generation module uses the model matching parameters PC to build a summary processing engine ZP based on the AI ​​processing engine, and configures an API processing interface for the summary processing engine ZP. After the API processing interface receives the text to be processed Tdc, the text to be processed Tdc is sent to the summary processing engine ZP, so as to generate a content summary Tnc corresponding to the text to be processed Tdc in the summary processing engine ZP.

[0051] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0052] at least one processor; and,

[0053] a memory communicatively connected to the at least one processor; wherein,

[0054] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the AI ​​content summary generation method in any of the aforementioned first aspects or any of the implementations of the first aspect.

[0055] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the AI ​​content summary generation method in the aforementioned first aspect or any implementation of the first aspect.

[0056] In the fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the AI ​​content summary generation method in the aforementioned first aspect or any implementation of the first aspect.

[0057] The AI ​​content summary generation scheme in the embodiment of the present invention includes: obtaining the original data DY of n users from the social media platform, performing data preprocessing operations on the original data to obtain standard data DB={DB1, DB2, ... DBn}; performing data screening operations on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain screening data DS={DS1, DS2, ... DSn} stored in the form of a string, where Ft represents the user's creation time, Hc represents the content interaction amount, Rc represents the content relevance value, and Fl represents the content attention amount; tokenizing the screening data DS through the token library provided by the AI ​​processing engine to obtain a token list TL, calculating the length of the token list TL, and obtaining the token number Nt=|TL| , when the token number Nt is less than the first threshold value N0, the compression rate Ctk=Nt / N0 of the token is calculated, and when the compression rate Ctk is less than the second threshold value C0, the token list is subjected to token verification to obtain a verification result Vy; a model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy is generated, where Tx represents the temperature coefficient T, Ls represents the maximum output length L, and Cc represents the sampling strategy P; using the model matching parameter PC, a summary processing engine ZP based on an AI processing engine is constructed, and an API processing interface is configured for the summary processing engine ZP. After the API processing interface receives the text to be processed Tdc, the text to be processed Tdc is sent to the summary processing engine ZP, so as to generate a content summary Tnc corresponding to the text to be processed Tdc in the summary processing engine ZP. The present invention can generate high-quality content summaries through multi-dimensional analysis of social media data, thereby helping users to efficiently extract key information. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A flowchart of an AI content summary generation method provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of another AI content summary generation method provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of the structure of an AI content summary generation device provided in an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0064] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0065] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0066] It should also be noted that the illustrations provided in the following embodiments only schematically illustrate the basic concept of the present disclosure. The components shown in the drawings only include those related to the present disclosure, rather than being drawn according to the number, shape, and size of the components in actual implementation. The types, quantities, and proportions of the components in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0067] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0068] An embodiment of the present disclosure provides an AI content summary generation method. The AI content summary generation method provided in this embodiment can be executed by a computing device, which can be implemented as software, or as a combination of software and hardware. The computing device can be integrally provided in a server, a terminal device, etc.

[0069] See Figure 1 and Figure 2 , an embodiment of the present disclosure provides an AI content summary generation method, including:

[0070] S101, obtain the original data DY of n users from a social media platform, and perform data preprocessing operations on the original data to obtain standard data DB = {DB1, DB2, …, DBn}.

[0071] The process of obtaining the original data DY of n users from a social media platform and performing data preprocessing operations on the original data to obtain standard data DB = {DB1, DB2, …, DBn} can be divided into the following steps

[0072] Select a social media platform to crawl data, such as Weibo, WeChat, etc. API interfaces, crawler tools, or other data scraping tools can be used to obtain data. Determine the time range, frequency, and user group for data collection. For example, posts and comments of active users in the past month can be collected.

[0073] For the collected data, records without content can be removed, records containing advertisements and promotional information can be removed, duplicate records can be removed to ensure the uniqueness of each piece of data, and special characters in the text, such as HTML tags, emojis, etc., can be removed.

[0074] Convert all text to a unified format, such as converting all to lowercase. Remove common stop words, such as "de", "shi", "zai", etc. Stem extraction and lemmatization are performed to restore words to their basic forms, such as restoring "running" to "run". Replace synonyms with a unified form to reduce the diversity of vocabulary.

[0075] Classify the text into different categories based on the topic, sentiment, and other factors. Use natural language processing techniques (such as TF-IDF and LDA) to classify the text into different topic categories. Use sentiment analysis tools to classify the text and mark it as positive, negative, or neutral. Extract keywords from the text for subsequent analysis and processing.

[0076] Organize the processed data into a structured format for easy management and query. Generate a unique identifier for each record, such as user ID, publishing time, etc. Organize the processed data into a structured format, such as JSON, CSV, etc. Store the structured data in a database or file for subsequent use.

[0077] Through these steps, users' raw data can be obtained and preprocessed from social media platforms to generate a high-quality standard dataset DB, providing a solid foundation for subsequent content screening, analysis, and application.

[0078] S102, based on the constructed content filtering vector Sn=[Ft,Hc,Rc,Fl] and the evaluation function Fp, a data filtering operation is performed on the standard data DB to obtain the filtering data DS={DS1,DS2,…DSn} stored in the form of a string, where Ft represents the user's creation time, Hc represents the content interaction amount, Rc represents the content relevance value, and Fl represents the content attention amount.

[0079] The process of performing a data screening operation on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain the screening data DS stored in the form of a string can be described in detail as follows:

[0080] Define the content filtering vector Sn, which includes four dimensions:

[0081] Ft: User creation time

[0082] Hc: Content interaction (such as likes, comments, and shares)

[0083] Rc: content relevance value (such as relevance to a specific topic)

[0084] Fl: Content attention (such as the number of followers of the user)

[0085] The evaluation function Fp is used to evaluate the quality of each standard data DBi. The specific formula is as follows:

[0086]

[0087] in: is the weight of each dimension.

[0088] : Take the square root of the user creation time to smooth out the effect of time.

[0089] : Take the logarithm of content interaction to reduce the impact of extreme values.

[0090] : Normalize the content-related values ​​to between 0 and 1, and max() is the maximum value function.

[0091] : Normalize the content attention to between 0 and 1.

[0092] For each standard data DBi, calculate its evaluation score Fp(DBi):

[0093]

[0094] The standard data DBi is screened according to the evaluation score Fp(DBi). A threshold can be set so that only data with an evaluation score greater than or equal to the threshold will be retained. All retained data DS_i are combined into a screened data set DS:

[0095]

[0096] in, It is the filter data in string form.

[0097] Through the above steps, the data screening operation can be performed on the standard data DB based on the content screening vector Sn and the evaluation function Fp, and finally a screening data set DS stored in the form of a character string is generated.

[0098] S103, tokenize the screening data DS through the token library provided by the AI ​​processing engine to obtain a token list TL, calculate the length of the token list TL, and obtain the token number Nt=|TL|. When the token number Nt is less than the first threshold N0, calculate the token compression rate Ctk=Nt / N0. When the compression rate Ctk is less than the second threshold C0, perform token verification on the token list to obtain a verification result Vy.

[0099] Through the token library provided by the AI ​​processing engine, the screening data DS is tokenized to obtain a token list TL, the length of the token list TL is calculated, and the number of tokens Nt=|TL| is obtained. When the number of tokens Nt is less than the first threshold N0, the compression rate Ctk=Nt / N0 of the token is calculated. When the compression rate Ctk is less than the second threshold C0, the token list is token verified. The process of obtaining the verification result Vy can be divided into the following steps:

[0100] Select a token library that is suitable for the current application scenario, such as the vocabulary of pre-trained models such as BERT, T5, etc. Load the token library into memory for fast lookup and matching during the tokenization process.

[0101] Use the tokenizer provided by the AI ​​processing engine to initialize a token initializer. Perform a tokenization operation on each piece of filtered data DSi (i=1,2,...,n) to split the text into individual tokens. Map each word or phrase after tokenization to a unique identifier (usually an integer ID) in the token library.

[0102] All tokens in the filtered data DS are combined in order into a token list TL. The total length of the generated token list TL is counted to obtain the token number Nt.

[0103] Set the first threshold N0: Determine an expected token quantity threshold N0 and calculate the token compression rate Ctk. The formula is:

[0104]

[0105] A compression rate threshold C0 is determined. If the compression rate Ctk is less than the second threshold C0, the token verification step is performed; otherwise, the verification step is skipped.

[0106] Perform token verification on the token list to ensure the quality and validity of the token list, including:

[0107] Check the validity of the token: Check each token in the token list TL one by one and remove invalid or erroneous tokens.

[0108] Evaluate the representativeness of tokens: Analyze the keywords in the token list to ensure that they can represent the main content of the original text.

[0109] Check the number of tokens: Make sure the length of the token list is within a reasonable range, neither too long nor too short.

[0110] Generate verification result Vy: According to the above verification process, generate verification result Vy. Verification result Vy can be a Boolean value (True or False), indicating whether the token list has passed the verification.

[0111] If the token list passes the verification, set the verification result Vy to True; if the token list fails the verification, set the verification result Vy to False.

[0112] Through the above steps, the screening data DS can be tokenized, a token list TL can be generated, the number of tokens Nt and the compression rate Ctk can be calculated, and token verification can be performed when the compression rate Ctk is less than the second threshold C0, and finally a verification result Vy can be obtained. These steps ensure that the generated token list is both effective and representative, providing high-quality input data for the subsequent summary generation task.

[0113] S104, generating a model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy, where Tx represents the temperature coefficient T, Ls represents the maximum output length L, and Cc represents the sampling strategy P.

[0114] In order to generate the model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy, it is necessary to adjust the temperature coefficient Tx, the maximum output length Ls and the sampling strategy Cc according to the verification result Vy.

[0115] Model matching parameters can be defined. Model matching parameters PC include three parts:

[0116] Tx: Temperature coefficient T

[0117] Ls: Maximum output length L

[0118] Cc: Sampling strategy P

[0119] The temperature coefficient Tx controls the diversity and randomness of the generated summary. The temperature coefficient Tx is adjusted according to the verification result Vy. The maximum output length Ls defines the maximum length of the generated summary. The maximum output length Ls is adjusted according to the verification result Vy. The sampling strategy Cc specifies the sampling method of the model when generating the summary. The sampling strategy Cc is adjusted according to the verification result Vy.

[0120] Based on the above adjustments, the final model matching parameters PC=[Tx,Ls,Cc] are generated.

[0121] Among them, the temperature coefficient adjustment formula is:

[0122]

[0123] For example, is the indicator function.

[0124] Maximum output length adjustment formula:

[0125]

[0126] For example, .

[0127] Sampling strategy selection formula:

[0128]

[0129] For example, , representing different sampling strategies respectively.

[0130] Through the above steps and mathematical formulas, the model matching parameters PC=[Tx,Ls,Cc] corresponding to the verification result Vy can be generated to ensure that the generated summary is both accurate and representative.

[0131] S105: construct a summary processing engine ZP based on the AI ​​processing engine by using the model matching parameters PC, configure an API processing interface for the summary processing engine ZP, and after receiving the text to be processed Tdc, the API processing interface sends the text to be processed Tdc to the summary processing engine ZP, so as to generate a content summary Tnc corresponding to the text to be processed Tdc in the summary processing engine ZP.

[0132] Select a pre-trained model suitable for the task of generating summaries, such as BERT, T5, GPT, etc., load the pre-trained model into memory, and prepare for inference.

[0133] Configure the model according to the model matching parameters PC generated by the verification result Vy. Use the configured model matching parameters PC to initialize the summary processing engine ZP. Design the input processing logic to convert the text to be processed Tdc into a format acceptable to the model; design the output processing logic to convert the token sequence generated by the model into a readable text summary Tnc.

[0134] Configure the API processing interface for the summary processing engine ZP so that it can receive external requests and return the generated summary. Define the URL path, request method (such as POST), request parameters (such as the text to be processed Tdc), and response format (such as JSON) of the API interface. Pass the received text to be processed Tdc to the summary processing engine ZP and return the generated summary Tnc.

[0135] Through the above steps, the model matching parameters PC can be used to build a summary processing engine ZP based on the AI ​​processing engine, and an API processing interface can be configured for it to ensure that the generated summary is both accurate and representative.

[0136] According to a specific implementation of an embodiment of the present invention, the method of obtaining the original data DY of n users from the social media platform and performing data preprocessing operations on the original data to obtain standard data DB={DB1, DB2, ...DBn} includes:

[0137] After determining the data source and data capture API interface, formulate the time range and frequency of data collection to obtain the original data DY of target users from social media platforms;

[0138] Remove empty records contained in the original data, filter out non-Chinese characters, and deduplicate the data to obtain cleaned data;

[0139] Perform spelling correction and synonym replacement operations on the cleaned data in a unified text format to obtain unified data;

[0140] Classifying the unified data into different categories according to the subject and sentiment tendency of the text in the unified data to form classified data;

[0141] According to a preset data format, the classified data is structured to obtain standard data DB={DB1, DB2, ...DBn}.

[0142] According to a specific implementation of an embodiment of the present invention, the data screening operation is performed on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain the screening data DS={DS1, DS2, ...DSn} stored in the form of a string, including:

[0143] Based on the evaluation function Fp, the comprehensive score of each standard data DBi is evaluated, and the content screening vector corresponding to DBi is , for each standard data DBi, calculate its comprehensive score Si:

[0144]

[0145] is the weight coefficient.

[0146] According to a specific implementation of an embodiment of the present invention, the data screening operation is performed on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain the screening data DS={DS1, DS2, ...DSn} stored in the form of a string, and further includes:

[0147] Sort all standard data according to the comprehensive score Si, and select data with a score higher than the threshold T0 as the screening data DS:

[0148]

[0149] The filtered data DS is converted into a string and stored.

[0150] According to a specific implementation of the embodiment of the present invention, the token library provided by the AI ​​processing engine performs a tokenization operation on the screening data DS to obtain a token list TL, including:

[0151] By calling the loading function, the selected token library is loaded into the memory, and the search and matching operations in the tokenization process are performed in the memory;

[0152] Splitting each text DSi contained in the screening data into independent phrases, mapping each phrase after word segmentation to a unique identifier in a token library, and forming each independent phrase into an independent token;

[0153] All tokens in the screening data DS are combined in order into a token list TL={TL1, TL2,…, TLn}, and each element TLi in the token list corresponds to a record DSi in the screening data DS.

[0154] According to a specific implementation of the embodiment of the present invention, when the compression rate Ctk is less than the second threshold C0, token verification is performed on the token list to obtain a verification result Vy, including:

[0155] Check each token in the token list TL and remove invalid or incorrect tokens;

[0156] Analyze the keywords in the token list to confirm whether the token list and the DS topic of the filtered data meet the consistency requirements;

[0157] According to the verification process, a verification result Vy is generated, and the verification result Vy is represented by a Boolean value.

[0158] According to a specific implementation of the embodiment of the present invention, the generating of the model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy includes:

[0159] If the verification result Vy is True, a low temperature coefficient is used, and if the verification result Vy is False, a high temperature coefficient is used;

[0160] If the verification result Vy is True, use the default value as the maximum output length. If the verification result Vy is False, increase the maximum output length.

[0161] If the verification result Vy is True, the text is collected using the beam search method. If the verification result Vy is False, the Top-K sampling method is used for sampling.

[0162] According to a specific implementation of an embodiment of the present invention, the use of the model matching parameter PC to construct a summary processing engine ZP based on an AI processing engine includes:

[0163] During model generation, the temperature coefficients are adjusted to control the diversity and certainty of the output;

[0164] Set the maximum output length Ls to limit the maximum length of the generated summary;

[0165] Based on the Cc value, set the corresponding sampling strategy.

[0166] Corresponding to the above method embodiment, see Figure 3 The embodiment of the present invention further discloses an AI content summary generating device 30, comprising:

[0167] The standard module 301 obtains the original data DY of n users from the social media platform, performs data preprocessing operations on the original data, and obtains standard data DB={DB1, DB2, ...DBn};

[0168] The screening module 302 performs a data screening operation on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain the screening data DS={DS1, DS2, ...DSn} stored in the form of a string, where Ft represents the user's creation time, Hc represents the content interaction amount, Rc represents the content relevance value, and Fl represents the content attention amount;

[0169] The calculation module 303 performs a tokenization operation on the screening data DS through the token library provided by the AI ​​processing engine to obtain a token list TL, calculates the length of the token list TL, obtains the number of tokens Nt=|TL|, and when the number of tokens Nt is less than the first threshold N0, calculates the compression rate Ctk=Nt / N0 of the token, and when the compression rate Ctk is less than the second threshold C0, performs token verification on the token list to obtain a verification result Vy;

[0170] A matching module 304 generates a model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy, where Tx represents the temperature coefficient T, Ls represents the maximum output length L, and Cc represents the sampling strategy P;

[0171] The generation module 305 uses the model matching parameters PC to construct a summary processing engine ZP based on the AI ​​processing engine, configures an API processing interface for the summary processing engine ZP, and after the API processing interface receives the text to be processed Tdc, sends the text to be processed Tdc to the summary processing engine ZP, so as to generate a content summary Tnc corresponding to the text to be processed Tdc in the summary processing engine ZP.

[0172] See also Figure 4 The embodiment of the present invention further provides an electronic device 60, the electronic device comprising:

[0173] at least one processor; and,

[0174] a memory communicatively connected to the at least one processor; wherein,

[0175] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the AI ​​content summary generation method in the aforementioned method embodiment.

[0176] An embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the aforementioned method embodiment.

[0177] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the AI ​​content summary generation method in the aforementioned method embodiment.

[0178] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device 60 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0179] like Figure 4 As shown, the electronic device 60 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0180] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, a gage, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 60 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0181] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0182] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An AI content summary generation method, characterized in that: include: Obtain the original data DY of n users from the social media platform, perform data preprocessing operations on the original data, and obtain standard data DB={DB1, DB2, ...DBn}; Based on the constructed content screening vector Sn=[Ft,Hc,Rc,Fl] and the evaluation function Fp, a data screening operation is performed on the standard data DB to obtain screening data DS={DS1,DS2,…DSn} stored in the form of a string, where Ft represents the user's creation time, Hc represents the content interaction amount, Rc represents the content relevance value, and Fl represents the content attention amount; Through the token library provided by the AI ​​processing engine, the screening data DS is tokenized to obtain a token list TL, the length of the token list TL is calculated to obtain the number of tokens Nt=|TL|, when the number of tokens Nt is less than the first threshold N0, the compression rate Ctk=Nt / N0 of the token is calculated, when the compression rate Ctk is less than the second threshold C0, the token list is token verified to obtain a verification result Vy; Generate a model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy, where Tx represents the temperature coefficient T, Ls represents the maximum output length L, and Cc represents the sampling strategy P; Using the model matching parameters PC, a summary processing engine ZP based on the AI ​​processing engine is constructed, and an API processing interface is configured for the summary processing engine ZP. After the API processing interface receives the text to be processed Tdc, the text to be processed Tdc is sent to the summary processing engine ZP, so as to generate a content summary Tnc corresponding to the text to be processed Tdc in the summary processing engine ZP; wherein The data screening operation is performed on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain the screening data DS={DS1, DS2, ...DSn} stored in the form of a string, including: Based on the evaluation function Fp, the comprehensive score of each standard data DBi is evaluated, and the content screening vector corresponding to DBi is , for each standard data DBi, calculate its comprehensive score Si: is the weight coefficient; Sort all standard data according to the comprehensive score Si, and select data with a score higher than the threshold T0 as the screening data DS: The filtered data DS is converted into a string and stored.

2. The method according to claim 1, characterized in that: The method of obtaining the original data DY of n users from the social media platform and performing data preprocessing operations on the original data to obtain standard data DB={DB1, DB2, ...DBn} includes: After determining the data source and data capture API interface, formulate the time range and frequency of data collection to obtain the original data DY of target users from social media platforms; Remove empty records contained in the original data, filter out non-Chinese characters, and deduplicate the data to obtain cleaned data; Perform spelling correction and synonym replacement operations on the cleaned data in a unified text format to obtain unified data; Classifying the unified data into different categories according to the subject and sentiment tendency of the text in the unified data to form classified data; According to a preset data format, the classified data is structured to obtain standard data DB={DB1, DB2, ...DBn}.

3. The method according to claim 1, characterized in that The token library provided by the AI ​​processing engine performs a tokenization operation on the screening data DS to obtain a token list TL, including: By calling the loading function, the selected token library is loaded into the memory, and the search and matching operations in the tokenization process are performed in the memory; Splitting each text DSi contained in the screening data into independent phrases, mapping each phrase after word segmentation to a unique identifier in a token library, and forming each independent phrase into an independent token; All tokens in the screening data DS are combined in order into a token list TL={TL1, TL2,…, TLn}, and each element TLi in the token list corresponds to a record DSi in the screening data DS.

4. The method according to claim 3, characterized in that When the compression rate Ctk is less than the second threshold C0, token verification is performed on the token list to obtain a verification result Vy, including: Check each token in the token list TL and remove invalid or incorrect tokens; Analyze the keywords in the token list to confirm whether the token list and the DS topic of the filtered data meet the consistency requirements; According to the verification process, a verification result Vy is generated, and the verification result Vy is represented by a Boolean value.

5. The method according to claim 4, characterized in that The generating of the model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy comprises: If the verification result Vy is True, a low temperature coefficient is used, and if the verification result Vy is False, a high temperature coefficient is used; If the verification result Vy is True, use the default value as the maximum output length. If the verification result Vy is False, increase the maximum output length. If the verification result Vy is True, the text is collected using the beam search method. If the verification result Vy is False, the Top-K sampling method is used for sampling.

6. The method according to claim 5, characterized in that The method of using the model matching parameters PC to construct a summary processing engine ZP based on an AI processing engine includes: During model generation, the temperature coefficients are adjusted to control the diversity and certainty of the output; Set the maximum output length Ls to limit the maximum length of the generated summary; Based on the Cc value, set the corresponding sampling strategy.

7. An AI content summary generation device, characterized in that: include: The standard module obtains the original data DY of n users from the social media platform, performs data preprocessing operations on the original data, and obtains standard data DB={DB1, DB2, ...DBn}; A screening module performs a data screening operation on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain screening data DS={DS1, DS2, ...DSn} stored in a string form, where Ft represents the user's creation time, Hc represents the content interaction amount, Rc represents the content relevance value, and Fl represents the content attention amount; The calculation module tokenizes the screening data DS through the token library provided by the AI ​​processing engine to obtain a token list TL, calculates the length of the token list TL, obtains the number of tokens Nt=|TL|, and when the number of tokens Nt is less than a first threshold value N0, calculates the compression rate Ctk=Nt / N0 of the token, and when the compression rate Ctk is less than a second threshold value C0, performs token verification on the token list to obtain a verification result Vy; A matching module generates a model matching parameter PC=[Tx, Ls, Cc] corresponding to the verification result Vy, where Tx represents the temperature coefficient T, Ls represents the maximum output length L, and Cc represents the sampling strategy P; A generation module is provided, which uses the model matching parameter PC to construct a summary processing engine ZP based on the AI ​​processing engine, and configures an API processing interface for the summary processing engine ZP. After the API processing interface receives the text to be processed Tdc, the text to be processed Tdc is sent to the summary processing engine ZP, so as to generate a content summary Tnc corresponding to the text to be processed Tdc in the summary processing engine ZP; wherein The data screening operation is performed on the standard data DB based on the constructed content screening vector Sn=[Ft, Hc, Rc, Fl] and the evaluation function Fp to obtain the screening data DS={DS1, DS2, ...DSn} stored in the form of a string, including: Based on the evaluation function Fp, the comprehensive score of each standard data DBi is evaluated, and the content screening vector corresponding to DBi is , for each standard data DBi, calculate its comprehensive score Si: is the weight coefficient; Sort all standard data according to the comprehensive score Si, and select data with a score higher than the threshold T0 as the screening data DS: The filtered data DS is converted into a string and stored.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the AI ​​content summary generation method described in any one of claims 1 to 6.

Images