Industrial research report generation method and device, computer equipment and storage medium
By performing block and loop summary of the industry report text library, the accuracy and logic of large language models when generating industry research reports is solved, and the quality and accuracy of the report are improved.
Patent Information
- Application Number
- CN202311204645.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-15
- Publication Date
- 2025-03-21
AI Technical Summary
When generating industry research reports, existing large language models have random accuracy, output length and expression logic, which cannot meet the requirements of market reports.
By obtaining the industry report text library and industry topics of the target industry, calling preset large language models to summarize the text library in blocks and loops, and outputting industry research reports on industry topics. This method breaks through the token limits of large language models, making reports more accurate in length, accuracy and logic.
The quality and accuracy of industry research reports are improved, making the reports more in line with industry topics and facilitate subsequent analysis, solving the problem of random output content of large language models.
Smart Images
Figure CN119669457A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of natural language processing technology, and specifically to a method, device, computer equipment and storage medium for generating an industry research report. Background Art
[0002] Industry research reports enable relevant departments, enterprises and individual investors to more clearly understand the development of a certain industry and the development prospects of the industry's products. Market research reports are generally produced by market researchers. Industry research reports require a large amount of data collection, filtering, analysis, and verification to produce an accurate industry research report, which takes a lot of time and manpower.
[0003] With the development of generative AI technology, industry research reports can also be automatically generated through generative AI technology, such as LLM (Large Language Mode). However, the existing large language models are AIGC models (Artificial Intelligence Generated Content) designed with underlying logic. Their output content is random in terms of accuracy, output length, expression logic, etc., and cannot meet the requirements of market reports. Summary of the invention
[0004] The embodiments of the present application provide a method, apparatus, computer device and storage medium for generating an industry research report to solve the technical problem that the research report output by a large language model has randomness in terms of accuracy, output length, expression logic, etc.
[0005] On the one hand, the present application provides an industry research report generation method, which includes: obtaining an industry report text library of a target industry, and obtaining the industry theme of the target industry; according to the industry theme, calling a preset large language model to perform block-by-block cyclic aggregation processing on the industry report text library, and outputting an industry research report on the industry theme; wherein, the industry research report is obtained by calling a preset large language model to perform block processing on the industry report text library, and aggregating the industry report text blocks obtained after the block processing.
[0006] The embodiment of the present application breaks through the token limitation of large language models by summarizing and aggregating the contents of industry report text libraries for industry themes in a continuous cycle, so that a large amount of ultra-long content can be understood by the large language model, and the large language model can realize the integration and output of long content, so that industry research reports on industry themes are more accurate in length, accuracy and logic, thereby improving the quality and accuracy of industry research reports.
[0007] In some implementation schemes of the present application, the industry report text library includes multiple industry report texts; the preset large language model includes a preset character number threshold, a block model and a summary model, wherein the block model includes a block character number threshold, and the block character number threshold is 1 / N of the preset character number threshold, and N is a natural number greater than 1; according to the industry theme, calling the preset large language model to perform block loop summary processing on the industry report text library, and outputting an industry research report on the industry theme, including: using the block model to block process the multiple industry report texts to obtain multiple industry report text blocks corresponding to each of the industry report texts, and the number of characters in each of the industry report text blocks is less than or equal to the block character number threshold; according to the industry theme, using the summary model to splice the industry report text blocks to generate the industry research report.
[0008] The embodiment of the present application divides multiple industry report texts into blocks according to a threshold value for the number of characters in the blocks, and concatenates and summarizes the processed industry report text blocks according to industry themes, thereby making the industry research report more in line with the industry theme and facilitating subsequent analysis of the industry report text blocks, thereby breaking through token restrictions.
[0009] In some embodiments of the present application, the summary model includes a vectorization sub-model, a vector calculation sub-model and a splicing sub-model; the industry report text blocks are spliced using the summary model according to the industry theme to generate the industry research report, including: traversing each of the industry report text blocks, extracting the text block vectors of the industry report text blocks corresponding to each industry report text using the vectorization sub-model, and extracting the theme vector of the industry theme using the vectorization sub-model; calculating the text Euclidean distance between the text block vectors of each industry report text and the theme vector using the vector calculation sub-model to determine the target report text; calculating the text block Euclidean distance between each of the text block vectors in the target report text and the theme vector using the vector calculation sub-model, sorting the calculated multiple text block Euclidean distances in ascending order, and extracting the industry report text blocks corresponding to the first N text block Euclidean distances; splicing the industry report text blocks corresponding to the first N text block Euclidean distances using the splicing sub-model to generate the industry research report.
[0010] The embodiment of the present application determines the target report text by extracting the text block vectors of the industry report text blocks corresponding to each industry report text and the theme vectors of the industry themes, and continues to calculate the text block Euclidean distances between each of the text block vectors and the theme vector in the target report text, extracts the industry report text blocks corresponding to the Euclidean distances of the first N text blocks for splicing, thereby realizing a cyclic summary analysis of the report text blocks, and extracts the industry report text blocks corresponding to the Euclidean distances of the first N text blocks for splicing, thereby improving the accuracy of the extracted report text blocks under the premise that the length of the spliced industry research report is less than or equal to the predetermined character number threshold.
[0011] In some implementation schemes of the present application, the use of the splicing sub-model to splice the industry report text blocks corresponding to the Euclidean distances of the first N text blocks to generate the industry research report includes: determining the order of appearance of the industry report text blocks corresponding to the Euclidean distances of the first N text blocks in the target industry report text; splicing the industry report text blocks corresponding to the Euclidean distances of the first N text blocks according to the order of appearance to generate the industry research report.
[0012] The embodiment of the present application splices the corresponding industry report text blocks according to the order of appearance of the industry report text blocks corresponding to the Euclidean distances of the first N text blocks in the target industry report text, thereby ensuring that the order of the industry report text blocks in the industry research report is consistent with the industry report text, ensuring the logical accuracy of the industry research report, and further improving the quality of the industry research report.
[0013] In some embodiments of the present application, the industry report text library of the target industry is obtained, including: determining the mainstream industry information websites of the target industry, each of the mainstream consulting websites in the industry is marked with credibility; obtaining industry keywords of the target industry; crawling the mainstream information websites in the industry according to the industry keywords to obtain an initial report text library, the initial report text library including multiple initial report texts and the credibility of the mainstream consulting websites in the industry where each initial report text is located; based on the industry keywords and the credibility, multiple initial report texts in the initial report text library are screened to obtain the industry report text library.
[0014] The embodiment of the present application captures the initial report text library through a crawler, thereby improving the timeliness of the initial report text library, and comprehensively considers the credibility of the source of the initial report text and the association between industry-related words and industry research reports, which is beneficial to improving the timeliness, quality and accuracy of the industry report text library.
[0015] In some embodiments of the present application, the multiple initial report texts in the initial report text library are screened according to the industry keywords and the credibility to obtain the industry report text library, including: determining the semantic similarity between the industry keywords and each of the initial report texts; obtaining a first weight for the semantic similarity and a second weight for the credibility; determining a quality score for each of the initial report texts according to the semantic similarity and the first weight, the credibility and the second weight; and determining the initial report text having a quality score greater than a preset score threshold as the industry report text library.
[0016] The embodiment of the present application ensures the quality and accuracy of the screened initial report text by performing a comprehensive analysis on the similarity between the initial report text and industry keywords and the credibility of the source of the initial report text, thereby improving the quality and accuracy of the industry report text library.
[0017] In some embodiments of the present application, determining the semantic similarity between the industry keywords and each of the initial report texts includes: preprocessing each of the initial report texts to obtain a preprocessed report text; performing inverted index processing on the preprocessed report text according to the industry keywords to obtain a preliminary screening report text; and calculating the semantic similarity between the industry keywords and each of the initial report texts based on the industry keywords and the preliminary screening report text.
[0018] The embodiment of the present application calculates the semantic similarity between industry keywords and each preliminary screening report text after determining the preliminary screening report text, thereby achieving secondary screening of the preliminary screening report text, thereby further improving the quality and accuracy of the industry report text library.
[0019] On the other hand, the present application provides an industry research report generating device, the industry research report generating device comprising:
[0020] An acquisition module is used to acquire an industry report text library of a target industry and acquire industry themes of the target industry;
[0021] A summarizing module is used to call a preset large language model to perform block-by-block cyclic summarization processing on the industry report text library, and output an industry research report on the industry theme; wherein the industry research report is obtained by calling a preset large language model to perform block-by-block processing on the industry report text library, and summarizing the industry report text blocks obtained after the block processing.
[0022] In some embodiments of the present application, the industry report text library includes multiple industry report texts; the preset large language model includes a preset character number threshold, a block model and a summary model, wherein the block model includes a block character number threshold, and the block character number threshold is 1 / N of the preset character number threshold, N is a natural number greater than 1; the summary module is specifically used to: use the block model to block the multiple industry report texts to obtain multiple industry report text blocks corresponding to each of the industry report texts, and the number of characters in each of the industry report text blocks is less than or equal to the block character number threshold; according to the industry theme, use the summary model to splice the industry report text blocks to generate the industry research report.
[0023] In some embodiments of the present application, the summary model includes a vectorization sub-model, a vector calculation sub-model and a splicing sub-model; the summary module is specifically used to: traverse each of the industry report text blocks, use the vectorization sub-model to extract the text block vector of the industry report text block corresponding to each industry report text, and use the vectorization sub-model to extract the theme vector of the industry theme; use the vector calculation sub-model to calculate the text Euclidean distance between the text block vector of each industry report text and the theme vector to determine the target report text; use the vector calculation sub-model to calculate the text block Euclidean distance between each of the text block vectors in the target report text and the theme vector, sort the calculated multiple text block Euclidean distances in ascending order, and extract the industry report text blocks corresponding to the first N text block Euclidean distances; use the splicing sub-model to splice the industry report text blocks corresponding to the first N text block Euclidean distances to generate the industry research report.
[0024] In some implementation schemes of the present application, the summary module is also specifically used to: determine the order of appearance of the industry report text blocks corresponding to the Euclidean distances of the first N text blocks in the target industry report text; splice the industry report text blocks corresponding to the Euclidean distances of the first N text blocks according to the order of appearance to generate the industry research report.
[0025] In some implementation schemes of the present application, the acquisition module is specifically used to: determine the mainstream industry information websites of the target industry, each of the mainstream consulting websites in the industry is marked with credibility; obtain industry keywords of the target industry; crawl the mainstream information websites in the industry according to the industry keywords to obtain an initial report text library, the initial report text library includes multiple initial report texts and the credibility of the mainstream consulting websites in the industry where each initial report text is located; according to the industry keywords and the credibility, multiple initial report texts in the initial report text library are screened to obtain the industry report text library.
[0026] In some embodiments of the present application, the acquisition module is specifically used to: determine the semantic similarity between the industry keywords and each of the initial report texts; obtain a first weight for the semantic similarity and a second weight for the credibility; determine a quality score for each of the initial report texts based on the semantic similarity and the first weight, the credibility and the second weight; and determine the initial report text whose quality score is greater than a preset score threshold as the industry report text library.
[0027] In some implementation schemes of the present application, the acquisition module is specifically used to: preprocess each of the initial report texts to obtain a preprocessed report text; perform inverted index processing on the preprocessed report text according to the industry keywords to obtain a preliminary screening report text; and calculate the semantic similarity between the industry keywords and each of the initial report texts based on the industry keywords and the preliminary screening report text.
[0028] On the other hand, the present application also provides a computer device, the computer device comprising:
[0029] one or more processors;
[0030] Memory; and
[0031] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps in the industry research report generation method described in any one of the first aspects.
[0032] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps in the method for generating an industry research report as described in any one of the first aspects.
[0033] The present application obtains an industry report text library of a target industry and obtains an industry theme of the target industry; based on the industry theme, a preset large language model is called to perform block-by-block cyclic aggregation processing on the industry report text library, and an industry research report on the industry theme is output. The embodiment of the present application breaks through the token limitation of the large language model. At the same time, based on the industry theme, the content of the industry report text library is summarized and aggregated in a continuous cycle, breaking through the token limitation of the large language model, so that a large amount of super-long content can be understood by the large language model, and the large language model can realize the integration and output of long content, so that the industry research report on the industry theme is more accurate in length, accuracy and logic, thereby improving the quality and accuracy of the industry research report. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 It is a schematic diagram of an application scenario of the industry research report generating device provided in an embodiment of the present application;
[0036] Figure 2 This is a flow chart of an embodiment of a method for generating an industry research report provided in an embodiment of the present application;
[0037] Figure 3 It is a schematic diagram of the process of calling the chatGPT4.0 model to perform block loop aggregation processing provided in an embodiment of the present application;
[0038] Figure 4 It is a schematic diagram of the structure of an embodiment of an industry research report generating device provided in an embodiment of the present application;
[0039] Figure 5 It is a schematic diagram of the structure of an embodiment of a computer device provided in the embodiments of the present application. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0041] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present application, "plurality" means two or more, unless otherwise clearly and specifically defined.
[0042] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.
[0043] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, the time described in this article is actually time information, which is used to characterize the information of time. It can be understood that if the size, quantity, position, etc. are mentioned in the subsequent embodiments, they all exist in the corresponding data form to facilitate the computer device to process, and the details will not be repeated here.
[0044] The embodiments of the present application provide a method, apparatus, computer device, and storage medium for generating an industry research report, which are described in detail below.
[0045] See also Figure 1 , Figure 1 Schematic diagram of a scenario of an industry research report generating device provided in an embodiment of the present application. The industry research report generating device may include a computer device 100, in which the industry research report generating device is integrated, such as Figure 1 Computer equipment in.
[0046] In the embodiment of the present application, the computer device 100 is mainly used to obtain an industry report text library of a target industry and to obtain an industry theme of the target industry; according to the industry theme, a preset large language model is called to perform block-by-block cyclic aggregation processing on the industry report text library, and an industry research report on the industry theme is output. By using the embodiment of the present application, the token limitation of the large language model is broken through. At the same time, according to the industry theme, the content of the industry report text library is summarized and aggregated, and the process is continuously cyclical, thus breaking through the token limitation of the large language model, so that a large amount of super-long content can be understood by the large language model, and the large language model can realize the integration and output of long content, so that the industry research report on the industry theme is more accurate in length, accuracy and logic, thereby improving the quality and accuracy of the industry research report.
[0047] In the embodiment of the present application, the computer device 100 may be an independent server, or a server network or server cluster composed of servers. For example, the computer device 100 described in the embodiment of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.
[0048] It is understandable that the computer device 100 used in the embodiments of the present application may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such a device may include: a cellular or other communication device having a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific computer device 100 may be a desktop terminal or a mobile terminal, and the computer device 100 may also be one of a mobile phone, a tablet computer, a laptop computer, etc.
[0049] Those skilled in the art will understand that Figure 1 The application scenario shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application scenarios may also include Figure 1 More or less computer equipment as shown in Figure 1 Only one computer device is shown. It can be understood that the industry research report generating device can also include one or more other services, which are not specifically limited here.
[0050] In addition, if Figure 1 As shown, the industry research report generating device may also include a storage module 200 for storing data, which may include report text, large language models, product keywords, etc.
[0051] It should be noted that Figure 1 The scenario diagram of the industry research report generating device shown is merely an example. The industry research report generating device and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can know that with the evolution of the industry research report generating device and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.
[0052] First, an industry research report generation method is provided in an embodiment of the present application. The executor of the industry research report generation method is an industry research report generation device, which is applied to a computer device. The industry research report generation method includes: obtaining an industry report text library of a target industry, and obtaining an industry theme of the target industry; according to the industry theme, calling a preset large language model to perform block-by-block cyclic aggregation processing on the industry report text library, and outputting an industry research report on the industry theme. By using the embodiment of the present application, the token limitation of the large language model is broken through. At the same time, according to the industry theme, the content of the industry report text library is summarized and aggregated, and the process is continuously repeated, thus breaking through the token limitation of the large language model, so that a large amount of super-long content can be understood by the large language model, and the large language model can realize the integration and output of long content, so that the industry research report on the industry theme is more accurate in length, accuracy and logic, thereby improving the quality and accuracy of the industry research report.
[0053] like Figure 2 As shown, it is a flow chart of an embodiment of the method for generating an industry research report in the embodiment of the present application. The method for generating an industry research report may include the following steps 301 to 302, which are as follows:
[0054] 301. Obtain an industry report text library of a target industry, and obtain an industry theme of the target industry.
[0055] The target industry refers to the industry where the industry research report needs to be generated, such as the logistics industry, the home appliance industry, the new energy industry, etc. The industry report text library refers to a text library composed of multiple report texts of the target industry, which is used to provide a knowledge base of relevant industry reports of the target industry. For example, the industry report text library can be a text library of multiple articles of relevant research reports of the target industry. The industry theme refers to the content that reflects the relevant themes of the target industry. For example, the industry theme can be a keyword or a combination of multiple keywords.
[0056] Specifically, multiple industry report texts can be captured from the relevant website crawlers of the target industry to form an industry report text library, or multiple industry report texts can be pre-entered on the server to form an industry report text library, and the industry report text library can be obtained from the server. The industry theme can be obtained through a pre-designed prompt instruction, or it can be obtained through user input. In this embodiment, the industry theme and the industry report text library of the target industry are obtained, so that the industry report text in the industry report text library can be analyzed and processed according to the industry theme.
[0057] In a specific implementation, the step 301 of obtaining the industry report text library of the target industry may include the following steps 301A to 301D, which are as follows:
[0058] 301A. Determine the mainstream information websites of the target industry, and mark each mainstream consulting website of the target industry as having credibility.
[0059] 301B, obtaining industry keywords of the target industry;
[0060] 301C. Crawling the mainstream information website of the industry according to the industry keywords to obtain an initial report text library, wherein the initial report text library includes a plurality of initial report texts and the credibility of the mainstream consulting website of the industry where each initial report text is located;
[0061] 301D. Screen multiple initial report texts in the initial report text library according to the industry keywords and the credibility to obtain the industry report text library.
[0062] Among them, the mainstream industry information website refers to the mainstream World Wide Web of the target industry. The credibility is used to characterize the credibility of the mainstream industry information website in the target industry. The higher the credibility, the more accurate the information in the mainstream industry information website. The credibility can be set according to manual experience.
[0063] Industry keywords refer to a combination of one or more keywords that can reflect the key information of the target industry. For example, industry keywords can be a combination of enterprise and policy keywords of the target industry. They can be determined based on the research direction or the keyword information of the target industry.
[0064] The initial report text library refers to a database formed by multiple initial report texts captured by crawlers on mainstream industry information websites.
[0065] Specifically, after obtaining industry keywords, a crawler is used to capture multiple initial report texts on mainstream industry information websites. Based on industry keywords and credibility, an industry report text library is screened out from the multiple initial report texts in the initial report text library. In this embodiment, the initial report text library is captured by crawling, which improves the timeliness of the initial report text library, and comprehensively considers the credibility of the source of the initial report text and the correlation between industry-related words and industry research reports, which is conducive to improving the timeliness, quality and accuracy of the industry report text library.
[0066] In a specific implementation, the step 301D of screening multiple initial report texts in the initial report text library according to the industry keywords and the credibility to obtain the industry report text library may include the following steps 301D1 to 301D4, which are as follows:
[0067] 301D1. Determine the semantic similarity between the industry keyword and each of the initial report texts;
[0068] 301D2. Obtain a first weight of the semantic similarity and a second weight of the credibility;
[0069] 301D3, determining a quality score of each of the initial report texts according to the semantic similarity and the first weight, the credibility, and the second weight;
[0070] 301D4. Determine the initial report texts whose quality scores are greater than a preset score threshold as the industry report text library.
[0071] Specifically, the semantic vectors of industry keywords can be calculated respectively by the word2vce algorithm or the neural network model in natural language technology to obtain the keyword vector, and the semantic vectors of each initial report text can be calculated to obtain the respective text vectors, and the similarity between the keyword vector and each text vector, such as cosine similarity, is calculated to obtain multiple semantic associations, and then the weight of the semantic similarity, that is, the first weight, and the weight of the credibility, that is, the second weight, are obtained, and the semantic similarity and the first weight, the credibility and the second weight are weighted and summed, and the calculation result is the quality score of the initial report text. In an example, the quality score S of the i-th initial report text is i =λ1*ai+λ2*bi, λ1 is the first weight, λ2 is the second weight, ai is the semantic similarity between the i-th initial report text and the industry keyword, bi is the credibility of the i-th initial report text, then filter out the quality score S i The initial report texts with a score greater than a preset threshold are used as the industry report text library. It can be understood that in this embodiment, by comprehensively analyzing the similarity between the initial report text and the industry keywords and the credibility of the source of the initial report text, the quality and accuracy of the selected initial report texts are ensured, and the quality and accuracy of the industry report text library are improved.
[0072] In a specific implementation, the step 301D1 of determining the semantic similarity between the industry keyword and each of the initial report texts may include the following steps 301D11 to 301D13, which are as follows:
[0073] 301D11. Preprocessing each of the initial report texts to obtain a preprocessed report text;
[0074] 301D12. Perform inverted index processing on the pre-processing report text according to the industry keywords to obtain a preliminary screening report text;
[0075] 301D13. Based on the industry keywords and the initial screening report texts, the semantic similarity between the industry keywords and each of the initial report texts is calculated.
[0076] Among them, the preprocessing can be at least one of the following processing methods for the initial report text: deduplication of text, removal of illegal characters, or removal of line breaks. In order to improve the screening efficiency, the preprocessing in this embodiment is preferably a combination of deduplication of text, removal of illegal characters, and removal of line breaks, that is, deduplication of text, removal of illegal characters, and removal of line breaks are performed on the initial report text, which reduces the interference of invalid information and improves the quality of the preprocessed report text.
[0077] An inverted index is a mapping between "content" (e.g., words, numbers) and the "location" where the content is stored (e.g., a database, a file, a group of files), which is used for fast full-text retrieval to quickly find files containing it based on the "content". In this embodiment, pre-processed report texts containing industry keywords are quickly screened out based on industry keywords, and inverted index processing can be performed through a search server. The process is: the pre-processed report text is segmented using a python word segmentation toolkit to obtain multiple report text segmentations, and the report text segmentation is performed, and the frequency of report text segmentations consistent with industry keywords in each pre-processed report text is counted. According to the frequency and the report text segmentations consistent with the industry keywords, a keyword frequency-pre-processed report text identifier is constructed to implement an inverted index, and a preliminary screening of the pre-processed report text is achieved, that is, the pre-processed report text with a higher frequency is used as the initial screening report text.
[0078] Specifically, after determining the initial screening report text, the Sentence-BERT model can be used to calculate the semantic similarity between industry keywords and each initial screening report text, to achieve secondary screening of the initial screening report text, and further improve the quality and accuracy of the industry report text library.
[0079] 302. According to the industry theme, call a preset large language model to perform block-based cyclic aggregation processing on the industry report text library, and output an industry research report on the industry theme; wherein, the industry research report is obtained by calling a preset large language model to perform block-based processing on the industry report text library, and aggregating the industry report text blocks obtained after the block processing.
[0080] Among them, the preset large language model is a pre-trained large prediction model for text processing, and the large language model can be a chatGPT4.0 model or a chatGPT3.0 model.
[0081] Block-by-block cyclic summarization is a text summarization method that divides the text into blocks, and then splices and combines the texts in each block before analyzing it. Figure 3As shown, it is a schematic diagram of the process of calling the chatGPT4.0 model for block loop aggregation processing. As can be seen from the figure, the process of calling the chatGPT4.0 model for block loop aggregation processing is: use the spacy Chinese model to divide the full text of the industry report text library into sentences, traverse each sentence, splice to form less than 3200 tokens (number of characters), and summarize; traverse each summary, splice to form less than 3200 tokens, and summarize, and continue to loop until all content is summarized.
[0082] Specifically, since large language models, such as the chatGPT model, have a token upper limit for a single instruction, and the output of industry research reports is generally long, most of them exceed the token limit of GPT. Therefore, in this embodiment, the token limit of large language models is broken through through block loop aggregation processing. At the same time, the content of the industry report text library is summarized and aggregated according to industry themes, and the cycle is continuously repeated, breaking through the token limit of large language models, so that a large amount of ultra-long content can be understood by the large language model, and the large language model can realize the integration and output of long content, so that industry research reports on industry themes are more accurate in length, accuracy and logic, thereby improving the quality and accuracy of industry research reports.
[0083] In the above embodiment, according to the industry theme, a preset large language model is called to perform block-by-block cyclic aggregation processing on the industry report text library, breaking through the token limitation of the large language model. At the same time, according to the industry theme, the content of the industry report text library is summarized and aggregated, and the cycle is continuously broken through the token limitation of the large language model, so that a large amount of ultra-long content can be understood by the large language model, and the large language model can realize the integration and output of long content, so that the industry research report on the industry theme is more accurate in length, accuracy and logic, thereby improving the quality and accuracy of the industry research report.
[0084] In a specific implementation, the industry report text library includes a plurality of industry report texts; the preset large language model includes a preset character quantity threshold, a block model and a summary model, wherein the block model includes a block character quantity threshold, and the block character quantity threshold is 1 / N of the preset character quantity threshold, where N is a natural number greater than 1; in step 302, according to the industry theme, calling the preset large language model to perform block cyclic summary processing on the industry report text library, and outputting the industry research report on the industry theme, may include the following steps 302A to 302B, which are specifically as follows:
[0085] 302A. Using the block model to perform block processing on the multiple industry report texts to obtain multiple industry report text blocks corresponding to each of the industry report texts, wherein the number of characters in each of the industry report text blocks is less than or equal to the block character number threshold;
[0086] 302B. Based on the industry theme, the industry report text blocks are spliced using the summary model to generate the industry research report.
[0087] The preset character number threshold is a preset upper limit of the number of characters, that is, the character number threshold, which is set according to the preset large language model. For example, for the chatGPT4.0 model, the preset character number threshold is 3200. The block character number threshold is 1 / N of the preset character number threshold. For example, N=4, that is, the block character number threshold is 800.
[0088] The block model can be a Chinese model of spacy, which is used to block the industry report text, that is, to divide the industry report text into multiple sentences. The summary model is used to analyze, integrate and summarize the report text blocks. For example, the summary model can be a model in which a text vectorization model and a text vector calculation model are cascaded. The text vectorization model can be an Embedding model, a Bert model, etc., and the text vector calculation model can be a cosine similarity calculation model. The block model and the summary model are cascaded.
[0089] Specifically, the multiple industry report texts can be processed in blocks according to the block character number threshold using a block model to obtain multiple industry report text blocks corresponding to each industry report text. According to the industry theme, the multiple industry report text blocks of an industry report text in each industry report text can be spliced using a summary model to generate an industry research report. In this embodiment, the multiple industry report texts are processed in blocks according to the block character number threshold, and according to the industry theme, the processed industry report text blocks are spliced and summarized, so that the industry research report is more in line with the industry theme and facilitates the subsequent analysis of the industry report text blocks, thereby breaking through the token restriction.
[0090] In a specific implementation, the summary model includes a vectorization sub-model, a vector calculation sub-model and a splicing sub-model; step 302B, according to the industry theme, splicing the industry report text blocks using the summary model to generate the industry research report, may include the following steps 302B1 to 302B4, which are as follows:
[0091] 302B1, traversing each of the industry report text blocks, using the vectorization sub-model to extract the text block vector of the industry report text block corresponding to each industry report text, and using the vectorization sub-model to extract the theme vector of the industry theme;
[0092] 302B3. Calculate the Euclidean distance between the text block vector and the topic vector of each industry report text using the vector calculation sub-model to determine the target report text;
[0093] 302B3. Calculate the text block Euclidean distance between each text block vector and the topic vector in the target report text using the vector calculation submodel, sort the calculated multiple text block Euclidean distances in ascending order, and extract the industry report text blocks corresponding to the first N text block Euclidean distances;
[0094] 302B4. Use the splicing sub-model to splice the industry report text blocks corresponding to the Euclidean distances of the first N text blocks to generate the industry research report.
[0095] Among them, the vectorization sub-model can be an Embedding model, a Bag of Words model, a TF-IDF, an N-gram model, a Word2vec, a Document2vec, etc., which are used to extract the vector of the text. As a preferred embodiment of this embodiment, the Embedding model is selected, because the Embedding sub-model can represent the semantics of a word or a sentence, which can improve the accuracy of text vectorization.
[0096] Specifically, each report text block is traversed, and the text block vectors of the industry report text blocks corresponding to each industry report text are extracted by using the vectorization sub-model, and the theme vectors of the industry themes are extracted by using the vectorization sub-model, and the target report text is determined by using the vector calculation sub-model, and the text block Euclidean distances between each text block vector and the theme vector in the target report text are further calculated by using the vector calculation sub-model, and the industry report text blocks corresponding to the Euclidean distances of the first N text blocks are extracted, and the industry report text blocks corresponding to the Euclidean distances of the first N text blocks are spliced by using the splicing sub-model, thereby realizing a cyclic summary analysis of the report text blocks, and extracting the industry report text blocks corresponding to the Euclidean distances of the first N text blocks for splicing, which improves the accuracy of the extracted report text blocks under the premise that the length of the spliced industry research report is less than or equal to the preset character number threshold.
[0097] In a specific implementation, the step 302B3 of splicing the industry report text blocks corresponding to the Euclidean distances of the first N text blocks using the splicing sub-model to generate the industry research report may include the following steps 302B1-302B2, which are as follows:
[0098] 302B1. Determine the order of appearance of the industry report text blocks corresponding to the Euclidean distances of the first N text blocks in the target industry report text;
[0099] 302B2. Concatenate the industry report text blocks corresponding to the Euclidean distances of the first N text blocks according to the order of appearance to generate the industry research report.
[0100] Specifically, the industry report text blocks corresponding to the Euclidean distances of the first N text blocks are spliced according to their appearance order in the target industry report text, so that the order of the industry research report is consistent with the industry report text. For example, the industry report text blocks corresponding to the Euclidean distances of the first N text blocks are C1, C2, ... C N , the order of appearance in the target industry report text is 1, 2, ... N, then the industry research report is C1C2...C N In this embodiment, the order of the industry report text blocks in the industry research report is ensured to be consistent with the industry report text, thereby ensuring the logical accuracy of the industry research report and further improving the quality of the industry research report.
[0101] In order to better implement the industry research report generation method in the embodiment of the present application, based on the industry research report generation method, the embodiment of the present application also provides an industry research report generation device, such as Figure 4 As shown, the industry research report generating device 400 includes:
[0102] An acquisition module 401 is used to acquire an industry report text library of a target industry and acquire an industry theme of the target industry;
[0103] The summarizing module 402 is used to call a preset large language model to perform block-by-block cyclic summarizing processing on the industry report text library, and output an industry research report on the industry theme.
[0104] In the embodiment of the present application, the industry report text library includes multiple industry report texts; the preset large language model includes a preset character quantity threshold, a block model and a summary model, wherein the block model includes a block character quantity threshold, and the block character quantity threshold is 1 / N of the preset character quantity threshold, where N is a natural number greater than 1; the summary module 402 is specifically used for:
[0105] The plurality of industry report texts are processed into blocks using the block model to obtain a plurality of industry report text blocks corresponding to each of the industry report texts, wherein the number of characters in each of the industry report text blocks is less than or equal to the block character number threshold;
[0106] According to the industry theme, the industry report text blocks are spliced using the summary model to generate the industry research report.
[0107] In the embodiment of the present application, the summary model includes a vectorization sub-model, a vector calculation sub-model and a splicing sub-model; the summary module 402 is further specifically used for:
[0108] Traversing each of the industry report text blocks, using the vectorized sub-model to extract the text block vector of the industry report text block corresponding to each industry report text, and using the vectorized sub-model to extract the subject vector of the industry subject;
[0109] The vector calculation sub-model is used to calculate the Euclidean distance between the text block vector and the topic vector of each industry report text to determine the target report text;
[0110] Calculate the text block Euclidean distance between each text block vector and the topic vector in the target report text by using the vector calculation submodel, sort the calculated multiple text block Euclidean distances in ascending order, and extract the industry report text blocks corresponding to the first N text block Euclidean distances;
[0111] The industry report text blocks corresponding to the Euclidean distances of the first N text blocks are spliced using the splicing sub-model to generate the industry research report.
[0112] In the embodiment of the present application, the aggregation module 402 is further used to:
[0113] Determine the order of appearance of the industry report text blocks corresponding to the Euclidean distances of the first N text blocks in the target industry report text;
[0114] The industry report text blocks corresponding to the Euclidean distances of the first N text blocks are spliced according to the order of appearance to generate the industry research report.
[0115] In the embodiment of the present application, the acquisition module 401 is further used to:
[0116] Determine the mainstream information websites of the target industry, and mark each mainstream consulting website of the target industry as credible;
[0117] Obtain industry keywords of the target industry;
[0118] Crawling the mainstream information websites of the industry according to the industry keywords to obtain an initial report text library, wherein the initial report text library includes a plurality of initial report texts and the credibility of the mainstream consulting websites of the industry where the initial report texts are located;
[0119] According to the industry keywords and the credibility, multiple initial report texts in the initial report text library are screened to obtain the industry report text library.
[0120] In the embodiment of the present application, the acquisition module 401 is further used to:
[0121] Determining the semantic similarity between the industry keywords and each of the initial report texts;
[0122] Obtaining a first weight of the semantic similarity and a second weight of the credibility;
[0123] Determine a quality score of each of the initial report texts according to the semantic similarity and the first weight, the credibility and the second weight;
[0124] The initial report texts whose quality scores are greater than a preset score threshold are determined as the industry report text library.
[0125] In the embodiment of the present application, the acquisition module is further used to:
[0126] Preprocessing each of the initial report texts to obtain a preprocessed report text;
[0127] Perform inverted index processing on the pre-processing report text according to the industry keywords to obtain a preliminary screening report text;
[0128] Based on the industry keywords and the initial screening report texts, the semantic similarity between the industry keywords and each of the initial report texts is calculated.
[0129] The embodiment of the present application further provides a computer device, which integrates any one of the industry research report generating devices provided in the embodiment of the present application, and the computer device includes:
[0130] one or more processors;
[0131] Memory; and
[0132] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the processor to execute the steps of the industry research report generation method described in any of the above-mentioned industry research report generation method embodiments.
[0133] The present application also provides a computer device that integrates any of the industry research report generation devices provided in the present application. Figure 5 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:
[0134] The computer device may include one or more processing core processors 701, one or more computer readable storage media memories 702, a power supply 703, an input unit 704 and other components. Those skilled in the art will appreciate that Figure 5 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:
[0135] The processor 701 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 702 and calling data stored in the memory 702, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 701.
[0136] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 702 may also include a memory controller to provide the processor 701 with access to the memory 702.
[0137] The computer device also includes a power supply 703 for supplying power to various components. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 703 can also include any combination of one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, etc.
[0138] The computer device may further include an input unit 704, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0139] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 701 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 702 according to the following instructions, and the processor 701 will run the application programs stored in the memory 702, thereby realizing various functions, as follows:
[0140] Obtain an industry report text library of a target industry and obtain industry themes of the target industry;
[0141] According to the industry theme, a preset large language model is called to perform block-based cyclic aggregation processing on the industry report text library, and an industry research report on the industry theme is output, wherein the industry research report is obtained by calling a preset large language model to perform block-based processing on the industry report text library, and aggregating the industry report text blocks obtained after the block processing.
[0142] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0143] To this end, an embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the industry research report generation methods provided in the embodiments of the present application. For example, the computer program can be loaded by a processor to execute the following steps:
[0144] Obtain an industry report text library of a target industry and obtain industry themes of the target industry;
[0145] According to the industry theme, a preset large language model is called to perform block-based cyclic aggregation processing on the industry report text library, and an industry research report on the industry theme is output, wherein the industry research report is obtained by calling a preset large language model to perform block-based processing on the industry report text library, and aggregating the industry report text blocks obtained after the block processing.
[0146] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.
[0147] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.
[0148] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0149] The above is a detailed introduction to an industry research report generation method, device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for generating an industry research report, characterized in that: The method for generating the industry research report comprises: Obtain an industry report text library of a target industry and obtain industry themes of the target industry; According to the industry theme, calling a preset large language model to perform block-by-block cyclic aggregation processing on the industry report text library, and outputting an industry research report on the industry theme; The industry research report is obtained by calling a preset large language model to perform block processing on the industry report text library, and summarizing the industry report text blocks obtained after the block processing.
2. The method for generating an industry research report according to claim 1, characterized in that: The industry report text library includes a plurality of industry report texts; the preset large language model includes a preset character quantity threshold, a block model and a summary model, wherein the block model includes a block character quantity threshold, and the block character quantity threshold is 1 / N of the preset character quantity threshold, where N is a natural number greater than 1; According to the industry theme, calling a preset large language model to perform block-by-block cyclic aggregation processing on the industry report text library, and outputting an industry research report on the industry theme, including: The plurality of industry report texts are processed into blocks using the block model to obtain a plurality of industry report text blocks corresponding to each of the industry report texts, wherein the number of characters in each of the industry report text blocks is less than or equal to the block character number threshold; According to the industry theme, the industry report text blocks are spliced using the summary model to generate the industry research report.
3. The method for generating an industry research report according to claim 2, characterized in that: The summary model includes a vectorization sub-model, a vector calculation sub-model and a splicing sub-model; The industry report text blocks are spliced using the summary model according to the industry theme to generate the industry research report, including: Traversing each of the industry report text blocks, using the vectorized sub-model to extract the text block vector of the industry report text block corresponding to each industry report text, and using the vectorized model to extract the subject vector of the industry subject; Calculate the text distance between the text block vector and the topic vector of each industry report text by using the vector calculation sub-model to determine the target report text; Calculate the text block distance between each text block vector and the topic vector in the target report text by using the vector calculation submodel, sort the calculated multiple text block distances in ascending order, and extract the industry report text blocks corresponding to the first N text block distances; The industry report text blocks corresponding to the first N text block distances are spliced using the splicing sub-model to generate the industry research report.
4. The method for generating an industry research report according to claim 3, characterized in that: The step of using the splicing sub-model to splice the industry report text blocks corresponding to the first N text block distances to generate the industry research report includes: Determine the order of appearance of the industry report text blocks corresponding to the first N text block distances in the target industry report text; The industry report text blocks corresponding to the first N text blocks are spliced according to the order of appearance to generate the industry research report.
5. The method for generating an industry research report according to claim 1, characterized in that: The industry report text library of the target industry is obtained, including: Determine the mainstream information websites of the target industry, and mark each mainstream consulting website of the target industry as credible; Obtain industry keywords of the target industry; Crawling the mainstream information websites of the industry according to the industry keywords to obtain an initial report text library, wherein the initial report text library includes a plurality of initial report texts and the credibility of the mainstream consulting websites of the industry where the initial report texts are located; According to the industry keywords and the credibility, multiple initial report texts in the initial report text library are screened to obtain the industry report text library.
6. The method for generating an industry research report according to claim 5, characterized in that: The step of screening multiple initial report texts in the initial report text library according to the industry keywords and the credibility to obtain the industry report text library includes: Determining the semantic similarity between the industry keywords and each of the initial report texts; Obtaining a first weight of the semantic similarity and a second weight of the credibility; Determine a quality score of each of the initial report texts according to the semantic similarity and the first weight, the credibility and the second weight; The initial report texts whose quality scores are greater than a preset score threshold are determined as the industry report text library.
7. The method for generating an industry research report according to claim 6, characterized in that: The determining of the semantic similarity between the industry keyword and each of the initial report texts includes: Preprocessing each of the initial report texts to obtain a preprocessed report text; Perform inverted index processing on the pre-processing report text according to the industry keywords to obtain a preliminary screening report text; Based on the industry keywords and the initial screening report texts, the semantic similarity between the industry keywords and each of the initial report texts is calculated.
8. An industry research report generating device, characterized in that: The industry research report generating device comprises: An acquisition module is used to acquire an industry report text library of a target industry and acquire industry themes of the target industry; A summary module is used to call a preset large language model to perform block-by-block cyclic summary processing on the industry report text library, and output an industry research report on the industry theme; The industry research report is obtained by calling a preset large language model to perform block processing on the industry report text library, and summarizing the industry report text blocks obtained after the block processing.
9. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the industry research report generating method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the method for generating an industry research report as described in any one of claims 1 to 7.