Method and system for analyzing domain trends based on scientific documents

Abstract

The application relates to the technical field of data information processing, and discloses a field trend analysis method based on scientific and technological documents, which comprises the following steps: collecting and arranging paper and patent data to form scientific and technological documents and storing the scientific and technological documents; generating scientific and technological document screening rules by self-defining query attribute values; dividing the screening rules into three categories and operating the scientific and technological documents in sequence; and analyzing and querying the attribute development trend of the scientific and technological documents in years according to the finally obtained scientific and technological document data. Two kinds of scientific and technological documents can be obtained through a mature crawler scheme, and metadata of the documents can be recognized and stored, so that attribute query and trend analysis in the future are facilitated; the scientific and technological documents are analyzed from multiple angles and multiple attributes, and finally, field trend analysis is obtained, the field trend analysis function can be realized, and the result display effect is rich and the result is complete.

Description

Technical Field

[0001] The present invention relates to the technical field of data information processing, and in particular to a field trend analysis method and system based on scientific and technological documents. Background Art

[0002] With the advancement of science and technology, every industry is experiencing continuous technological advancements. With this rapid technological evolution, it becomes increasingly difficult to accurately grasp development trends. Accurately predicting technological trends can be crucial for companies to formulate strategic policies and adjust their technological direction. Traditional methods typically require manual research on extensive industry technical data to conduct complex data analysis and forecasting. This is a significant waste of manpower and resources, resulting in significant inefficiencies. Furthermore, manual analysis is difficult and makes it difficult to accurately predict technological trends across every industry.

[0003] For example, the technology trend prediction method, system and storage medium with patent publication number CN111008874A discloses the following steps: confirming the technical field to be predicted; selecting leading enterprises in the technical field and product information of leading enterprises at different time stages; selecting keywords in product information to establish a vocabulary; selecting online communities in the technical field and comment information of all users in the online community; processing the comment information and the vocabulary to obtain the degree of correlation between the keywords in the vocabulary and the user's comment information and the early or late appearance time; obtaining the corresponding leading degree of the user based on the correlation degree and the early or late appearance time, and determining the leading user from the users based on the leading degree; obtaining the comment information of the leading user, and parsing the comment information to predict the technology development trend. In this way, the correlation between professional field comment information, patent information and time information can be used to build a model to find the leading user, and through the leading user comment prediction technology,

[0004] However, the keyword acquisition operations in comment information and patent information are different. Considering the two texts with different writing styles, different keyword acquisition models are required. Incorrect keywords will directly lead to errors in the correlation between the two. There is no screening of user comments. Invalid comments or water army comments may interfere with the model, and it is impossible to analyze the technical trends of the paper from multiple angles. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for analyzing field trends based on scientific and technological documents to solve the following technical problems:

[0006] How to provide a field trend analysis method and system that can perform multi-attribute analysis on scientific and technological documents and facilitate user interaction.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A field trend analysis method based on scientific documents comprises:

[0009] In step S10, after collecting the paper and patent data, the scientific documents are formed and stored after being sorted.

[0010] In step S20, a custom query attribute value is used to generate a scientific document screening rule.

[0011] In step S30, the screening rule is divided into three categories and the scientific documents are operated in sequence.

[0012] In step S40, the development trend of the query attribute of the scientific documents in the years is analyzed.

[0013] Through the above technical solution, two kinds of scientific documents can be obtained through a mature crawler scheme, and the metadata of the documents is recognized and stored, which facilitates subsequent attribute query and trend analysis. The scientific documents can be analyzed from multiple angles and multiple attributes, and finally the field trend analysis is obtained. In addition, the user can have more interaction and freedom, and can set attributes according to own needs, design multiple trend analysis parameters, and give more diversified analysis results according to the user's customization. At the same time, a multi-unit and multi-module system is proposed, which can completely realize the field trend analysis function, and the result display effect is rich and the result is complete.

[0014] As a further scheme of the present application, the sorting method comprises:

[0015] In step S101, it is determined whether the local scientific documents exist; if yes, step S103 is entered, otherwise step S102 is entered.

[0016] In step S102, the paper and patent data are crawled.

[0017] In step S103, the metadata of the paper and patent data is divided.

[0018] In step S104, the metadata is stored in the database in units of documents.

[0019] As a further scheme of the present application, the step S20 comprises:

[0020] In step S201, the query attribute is determined.

[0021] In step S202, the screening rule is constructed.

[0022] In step S203, the final query request is submitted.

[0023] As a further scheme of the present application, the step S30 comprises:

[0024] Step S301, the screening rules are divided into statistical rules, filtering rules and sorting rules;

[0025] Step S302, the database is queried according to the statistical rules;

[0026] Step S303, the filtered results are selected according to the filtering rules;

[0027] Step S303, the filtered results are selected according to the filtering rules;

[0028] As a further scheme of the application, the step S40 comprises:

[0029] Step S401, the page shows the attribute trend result;

[0030] Step S402, the single attribute result is clicked to show the scientific document statistical result;

[0031] Step S403, the single scientific document field information is clicked to show the scientific document field information;

[0032] Step S404, the single scientific document field information is clicked to show the scientific document detailed information.

[0033] The field trend analysis system based on scientific documents comprises:

[0034] The collection and arrangement module is used for collecting and arranging papers and patent data, forming scientific documents and storing the scientific documents;

[0035] The rule setting module is used for customizing query attribute values to generate scientific document screening rules;

[0036] The screening execution module is used for dividing the screening rules into three categories and operating the scientific documents according to the order;

[0037] The trend query display module is used for analyzing and querying the development trend of the attributes in the years according to the finally obtained scientific document data.

[0038] The application has the advantages that two kinds of scientific documents can be obtained through a mature crawler scheme, and the documents are subjected to metadata recognition and storage, so that subsequent attribute query and trend analysis are facilitated; numpy syntax is used to construct a high-speed filtering matrix by customizing rules, and a dynamic programming idea is added to construct an algorithm framework to realize high-speed and efficient scientific document search and classification; in addition, users can have more interaction and freedom, can set attributes according to their own needs, design various trend analysis parameters, and give more diversified analysis results according to user customization; meanwhile, a multi-unit and multi-module system is proposed, which can completely realize the field trend analysis function, and the result display effect is rich and the result is complete. BRIEF DESCRIPTION OF DRAWINGS

[0039] The application will be further described below with reference to the drawings.

[0040] Figure 1 The overall flowchart of the field trend analysis method in the application is shown in the figure;

[0041] Figure 2 The specific flowchart of step S10 of the field trend analysis method in the application is shown in the figure;

[0042] Figure 3 The specific flowchart of step S20 of the field trend analysis method in the application is shown in the figure;

[0043] Figure 4 The specific flowchart of step S30 of the field trend analysis method in the application is shown in the figure;

[0044] Figure 5 The specific flowchart of step S40 of the field trend analysis method in the application is shown in the figure;

[0045] Figure 6 The detailed flowchart of the paper document in step S10 of the application is shown in the figure;

[0046] Figure 7 The interface layout of the front-end page screening rule customization in step S20 of the application is shown in the figure;

[0047] Figure 8 The detailed flowchart of the back-end processing in step S30 of the application is shown in the figure;

[0048] Figure 9 The trend display result of the query attribute page in step S40 of the application is shown in the figure. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0050] Please refer to Figure 1 and Figure 2 The application is a field trend analysis method based on scientific and technical documents, which comprises:

[0051] Step S10: collecting papers and patent data, organizing and storing scientific and technical documents after the organization;

[0052] Step S20: customizing query attribute values to generate scientific and technical document screening rules;

[0053] Step S30, the screening rules are divided into three categories and the scientific documents are operated in sequence;

[0054] Step S40, the development trend of the scientific documents in the years is analyzed according to the data analysis query attribute.

[0055] The method for organizing includes:

[0056] Step S101, whether the scientific documents exist in the local is inquired; if yes, step S103 is entered, otherwise step S102 is entered;

[0057] Step S102, the paper and patent data are crawled;

[0058] Step S103, the paper and patent data are divided into metadata;

[0059] Step S104, the metadata is stored to the database in the unit of the document.

[0060] In the database construction of the application, the scientific documents are obtained by crawling and processing the local documents, and the scientific documents include the paper information and the domestic and foreign patent data. The paper data includes the code address in addition to the paper itself.

[0061] Taking the paper data as an example, referring to Figure 6 As shown in the figure, the paper data is crawled from the related paper website by using the scheme of the crawler. Specifically, the title, author and other information of the paper under the journal and conference are accessed by constructing the dblpapi link of the journal and conference to access the dblp website, the paper pdf is downloaded by searching the related paper website by using the information, and the text data is obtained by text extraction of the text in the pdf.

[0062] The metadata of all the scientific document data (paper and patent data) is divided. The author, code, abstract, data set, evaluation index and the like of each paper in the above paper text data are identified and extracted, and are separately archived. The abstract, declaration, background, inventor information and the like of each patent are also identified from the patent document. If the original text is English, the above metadata also needs to be translated and saved. Finally, all the data are stored in the mongodb.

[0063] As a further scheme of the application, as shown in Figure 3 The step S20 includes:

[0064] Step S201, a query attribute is determined;

[0065] Step S202, a screening rule is constructed;

[0066] Step S203, submit the final query request.

[0067] Through the above technical solution, the user can input a standard mongodb query statement for query according to his own needs through the text box "Mongo data filtering statement" displayed on the front-end web page. Also, the user can input query conditions according to the pre-defined query statement format, such as square brackets, parentheses, etc. in another text box "manual query group", as shown in the figure. Figure 6 The user can also customize the input or select the value of each attribute through a drop-down table according to the pre-defined query attribute in the group query area displayed on the front end. The three ways can be used alone or in combination.

[0068] As shown in the figure, Figure 7 After completing the text filtering rule input, the user clicks the query statement submission button on the front-end web page, and waits for the back-end system to process the filtering rule transmitted by the front-end system. Finally, the result is displayed on the front-end page in the form of text and table.

[0069] As a further scheme of the application, as shown in the figure, Figure 4 The step S30 includes:

[0070] Step S301, divide the filtering rule into a statistical rule, a filtering rule and a sorting rule;

[0071] Step S302, query the database according to the statistical rule;

[0072] Step S303, select the query result according to the filtering rule;

[0073] Step S303, process the filtered result according to the sorting rule.

[0074] Through the above technical solution, as shown in the figure, Figure 8 According to the filtering rule transmitted by the front-end web page, first, the statistical rule is obtained. Here, the statistical rule mainly includes the statistical year, whether the object of statistics is paper data or patent data, the query object (for example, the query object of paper includes: phrase / author / task / dataset, etc.), and other large-granularity filtering rules.

[0075] According to the filtering rules transmitted by the front-end webpage, the filtering rules are acquired. The filtering objects in the filtering rules are defined by the back-end system, and the user needs to manually input or select the value of each attribute on the front-end webpage. If the user does not select the back-end system, the default value is used. These attributes include the number of each batch, the number of papers or patents, the standard deviation of each two consecutive batches, the coefficient of variation of each two consecutive batches, and other self-defined attributes.

[0076] According to the filtering rules transmitted by the front-end webpage, the sorting rules are acquired. The sorting rules are also self-defined attributes of the back-end system, and the user selects or customizes the value of each attribute through the front-end webpage. If the user does not select, the system default value is used. The specific attributes are the same as those in the filtering rules, and the user only needs to select the attribute or attributes for sorting.

[0077] Through the above three rules, a high-speed filter is constructed using a numpy matrix to quickly retrieve data that meets the rules from the database. At the same time, a dynamic programming algorithm is used to enhance the efficiency of database data retrieval. The dynamic programming and matrix acceleration algorithm is described as follows:

[0078] The number of n-keyword appearing scientific texts is equal to the number of n-1-keyword appearing scientific texts plus the number of single-keyword appearing scientific texts. That is, when the number of 1-keyword appearing scientific texts is calculated, the number of 2-keyword appearing scientific texts can be calculated using the number of 1-keyword appearing scientific texts. When the number of 2-keyword appearing scientific texts is calculated, the number of 3-keyword appearing scientific texts can be calculated using the number of 2-keyword appearing scientific texts plus the number of 1-keyword appearing scientific texts. Similarly, when the number of n-1-keyword appearing scientific texts is calculated, the number of n-keyword appearing scientific texts can be easily calculated. Matrix filtering changes the maximum and minimum limits into two matrices, and the number of keyword appearing scientific texts is also a matrix. The calculation between these three matrices can be parallel to increase the calculation speed, and there is no need to use loop judgment.

[0079] The scientific documents that meet the conditions are retrieved from the database, and the trend is displayed according to the change trend of the past two years (including standard deviation, coefficient of variation, etc.).

[0080] As a further scheme of the present application, as shown in Figure 5 the step S40 comprises:

[0081] Step S401, displaying attribute trend results on a page;

[0082] Step S402, clicking on a single attribute result to display scientific document statistical results;

[0083] Step S403, clicking on a single said scientific document to display the field information of the scientific document;

[0084] Step S404, clicking on a single said scientific document field information to display the detailed information of the scientific document.

[0085] Through the above technical solution, referring to Figure 9 The front-end data display is mainly in the form of table output, and first is the batch result display. Taking the query of paper trends of word groups as an example (the following steps are all this example): the user needs to arrange and combine the word groups; assuming that there are n word groups, the final result is divided into n batches; the i-th batch displays the trends of papers in which i word groups appear in each year. The table header is defined by us (i.e. the filtering in step 3, the attributes in the sorting rule).

[0086] For the hyperlink in each result of each batch in step 4.1, the user can click on it, and the page will jump after clicking. The new page displays all the paper information with the word group combination, which is also composed of a table, and the table header is composed of the paper meta array obtained in step 1.

[0087] For the hyperlink in each paper information in the table of all paper information meeting the word group combination condition in step 4.3, the user can click on it, and the page will jump after clicking. The new page will display more specific information of the paper, and the page is displayed in the form of table plus text.

[0088] A field trend analysis system based on scientific documents, comprising:

[0089] A collection and arrangement module for collecting and arranging papers and patent data, forming scientific documents and storing them;

[0090] A rule setting module for customizing query attribute values to generate scientific document filtering rules;

[0091] A filtering execution module for dividing the filtering rules into three categories and operating on the scientific documents in order;

[0092] A trend query display module for analyzing and querying the development trend of the data analysis query attribute of the scientific documents in each year.

[0093] The above describes one embodiment of the present application in detail, but the content is only the preferred embodiment of the present application, and cannot be considered as limiting the scope of the present application. Any equivalent changes and improvements made according to the scope of the present application should still belong to the patent coverage of the present application.

Claims

1. A method for field trend analysis based on scientific documents, characterized in that, include: Step S10, collecting and organizing the paper and patent data, forming scientific and technological documents and storing them; Step S20, generating scientific and technological document screening rules by customizing query attribute values; Step S30, dividing the screening rules into three categories and performing operations on the scientific documents in order; Step S40, analyzing the development trend of query attributes over the years based on the finally obtained data of the scientific and technological documents; The step S30 includes: Step S301, dividing the screening rules into statistical rules, filtering rules and sorting rules; Step S302, querying the database according to the statistical rules; Step S303, selecting the query results according to the filtering rules; Step S303, processing the filtered results according to the sorting rules; The statistical rules include statistical year, statistical object and query object; The filtering objects in the filtering rules are defined by the backend system, and the user needs to manually enter or select the value of each attribute; the attributes include the number of papers or patents in each batch, the standard deviation between two consecutive batches, and the coefficient of variation between two consecutive batches; The sorting rule is sorted according to the attribute selected by the user; According to the statistical rules, scientific and technological documents that meet the conditions are retrieved from the database, and the trend of the selected attributes is displayed according to the change trend of two consecutive years; The step S40 includes: Step S401: The page displays the attribute trend results; Step S402: Click on a single attribute result to display the scientific document statistics result; Step S403: Click on a single scientific document to display the field information of the scientific document; Step S404: Click on a single field of the scientific document to display detailed information of the scientific document; In step S401, a hyperlink for clicking and jumping is set in each result of each batch, and the jump page displays all paper information with the specified phrase combination in a table format; In the table of all paper information that meets the specified phrase combination conditions in step S403, a hyperlink for clicking and jumping is set in each paper information, and the jump page displays more specific information about the paper. 2.The sci-tech document based domain trend analysis method according to claim 1, characterized in that, The method for arranging comprises: Step S101, check whether there is a scientific document locally; if so, go to step S103, otherwise go to step S102; Step S102, crawling the paper and patent data; Step S103, performing metadata division on the paper and patent data; In step S104 , metadata is stored in a database in units of documents. 3.The sci-tech document based domain trend analysis method according to claim 1, characterized in that, The step S20 includes: Step S201, determining query attributes; Step S202, constructing screening rules; Step S203: Submit the final query request.

4. The system for analyzing trends in a field based on scientific documents according to any one of claims 1 to 3, wherein, include: The collection and organization module is used to collect and organize papers and patent data, and then store them as scientific and technological documents; Rule setting module, used to customize query attribute values ​​to generate scientific and technological document screening rules; A screening execution module, used to divide the screening rules into three categories and operate on the scientific documents in order; The trend query display module is used to analyze the development trend of query attributes over the years based on the data of the scientific and technological documents finally obtained.

Images