Automatic inquiry and analysis method and system for machine account information in bidding and tendering industry

Abstract

The invention discloses a bidding industry machine account information automatic query and analysis method and system, and relates to the technical field of information query and analysis, and the method comprises the steps: collecting machine account information containing a target competitor and various certificate information, and carrying out the preprocessing; converting key field data in the machine account information and text data in the certificate information into low-dimensional dense vectors by utilizing an Embedding model, storing the low-dimensional dense vectors into Milvus, establishing indexes on vector fields, and meanwhile, storing text contents in the machine account information and the certificate information into Elasticsearch; constructing a training set based on machine account information and certificate information, and training a JointBERT model; text content needing to be inquired is input into the JointBERT model, firstly, intention recognition is conducted on the input text through the JointBERT model, then corresponding search and analysis processes are executed according to the recognition result, and finally a detailed analysis report is generated. According to the invention, efficient and accurate management and query of machine account information and certificate information can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of information query and analysis, and in particular to a method and system for automatic query and analysis of ledger information in the bidding industry. Background Art

[0002] In the traditional bidding industry, the management of ledger information and the query and analysis of the certificate information base is a complex and time-consuming task. With the development of the industry, the types and number of certificates continue to increase, and the certificate information base has become increasingly large and complex. At the same time, the ledger information table has also been continuously enriched, containing a large amount of corporate information, certificate information, etc. The query and analysis of this information requires a lot of manpower and time, and is inefficient, and it is difficult to guarantee accuracy and timeliness. Summary of the invention

[0003] With the development of the bidding industry, the increase in the types and number of certificates has made the information database large and complex. Traditional query and analysis methods can no longer meet the growing business needs. Based on this, the present invention provides an automated query and analysis method and system for the bidding industry ledger information to quickly query the detailed information of a specified certificate and analyze the information ledger, thereby improving the efficiency of querying various certificates and information in the ledger.

[0004] In the first aspect, the present invention provides a method for automatic query and analysis of information in the bidding industry ledger. The technical solution adopted to solve the above technical problems is as follows:

[0005] A method for automatic query and analysis of bidding industry ledger information, comprising the following steps:

[0006] S1. Collect the ledger information and various certificate information of target competitors and pre-process the collected information;

[0007] S2. Using the Embedding model, the key field data in the ledger information and the text data in the certificate information are converted into low-dimensional dense vectors, stored in the Milvus vector database, and indexed on the vector fields. At the same time, the text content in the ledger information and certificate information is stored in the Elasticsearch database.

[0008] S3. Build a training set based on ledger information and certificate information to train a JointBERT model that can perform intent recognition and entity content extraction.

[0009] S4. Input the text content to be queried into the trained JointBERT model. First, the JointBERT model is used to identify the intent of the input text. Then, the corresponding search and analysis process is performed based on the recognition results, and finally a detailed analysis report is generated.

[0010] Optionally, the step S1 specifically includes the following operations:

[0011] Through automated data crawling technology, we collect target competitors’ ledger information and various certificate information from the Internet;

[0012] Implement data governance processes for collected information to ensure the legitimacy, accuracy and consistency of data sources;

[0013] Data cleaning is performed on the collected information to remove invalid or erroneous information, fill in missing values, and convert the data into a structured format.

[0014] Further optionally, step S1 is executed, and the collected ledger information includes the target competitor, the certificates contained by the target competitor, and the category to which the certificates belong; the collected certificate information includes the certificate query website, the certificate introduction, the certificate field introduction, and the certificate level introduction.

[0015] Optionally, execute step S4, first perform intent recognition on the input text through the JointBERT model to identify whether the input text is to query ledger information or certificate information;

[0016] (a) For ledger information query: ① The JointBERT model uses natural language processing technology to extract company names and certificate names from the input text; ② The JointBERT model calculates the similarity between the extracted company names and certificate names and the content stored in the Milvus vector database, and selects the result with the highest similarity; ③ Based on the selected results, the ledger information is further retrieved in the Elasticsearch database, and the data processing tool Pandas is used to organize and format the retrieved data; ④ The organized data is combined with the input text to form a prompt, which is input into the large language model LLM to generate a detailed analysis report;

[0017] (b) For certificate information query: ① The JointBERT model calculates semantic similarity and keyword matching between the input text and the certificate content in the Milvus vector database and the Elasticsearch database to obtain relevant entries. ② The reranker model is used to re-rank the matched entries and select the top ten most relevant results. ③ The selected results are combined with the input text to construct a prompt and generate a detailed analysis report through the large language model LLM.

[0018] Preferably, when executing (a), if the JointBERT model does not extract the company name and certificate name from the input text, the extraction result is empty and the subsequent process cannot be executed; after using the data processing tool Pandas to organize and format the retrieved data, the data format of "XXX company holds XXX certificate" and "The company holding XXX certificate has XXX" is formed.

[0019] In the second aspect, the present invention provides an automated query and analysis system for bidding industry ledger information. The technical solution adopted to solve the above technical problems is as follows:

[0020] An automatic query and analysis system for bidding industry ledger information, comprising:

[0021] The data collection and preprocessing module is used to collect ledger information and various certificate information containing target competitors and preprocess the collected information;

[0022] The vector representation and storage module is used to use the Embedding model to convert the key field data in the ledger information and the text data in the certificate information into low-dimensional dense vectors, store them in the Milvus vector database, and create indexes on the vector fields. It is also used to store the text content in the ledger information and certificate information in the Elasticsearch database;

[0023] The JointBERT model training module is used to build a training set based on ledger information and certificate information to train a JointBERT model capable of intent recognition and entity content extraction;

[0024] The intent recognition and report generation module is used to input the text content to be queried into the trained JointBERT model. The JointBERT model first recognizes the intent of the input text, then executes the corresponding search and analysis process based on the recognition results, and finally generates a detailed analysis report.

[0025] Optionally, the data collection and preprocessing module involved first collects the target competitor's ledger information and various certificate information from the Internet through automated data crawling technology, and then implements a data governance process for the collected information to ensure the legitimacy, accuracy and consistency of the data source. Finally, the collected information is cleaned to remove invalid or erroneous information, fill in missing values, and convert the data into a structured format.

[0026] Further optionally, the ledger information collected by the data collection and preprocessing module includes target competitors, certificates contained by the target competitors, and categories to which the certificates belong;

[0027] The certificate information collected by the data collection and preprocessing module includes the certificate query website, the certificate introduction, the certificate field introduction and the certificate level introduction.

[0028] Optionally, the intent recognition and report generation module first uses the JointBERT model to recognize the intent of the input text to identify whether the input text is to query the ledger information or the certificate information, and then:

[0029] (a) For ledger information query: ① The JointBERT model uses natural language processing technology to extract company names and certificate names from the input text; ② The JointBERT model calculates the similarity between the extracted company names and certificate names and the content stored in the Milvus vector database, and selects the result with the highest similarity; ③ Based on the selected results, the ledger information is further retrieved in the Elasticsearch database, and the data processing tool Pandas is used to organize and format the retrieved data; ④ The organized data is combined with the input text to form a prompt, which is input into the large language model LLM to generate a detailed analysis report;

[0030] (b) For certificate information query: ① The JointBERT model calculates semantic similarity and keyword matching between the input text and the certificate content in the Milvus vector database and the Elasticsearch database to obtain relevant entries. ② The reranker model is used to re-rank the matched entries and select the top ten most relevant results. ③ The selected results are combined with the input text to construct a prompt and generate a detailed analysis report through the large language model LLM.

[0031] Preferably, when executing (a), if the JointBERT model does not extract the company name and certificate name from the input text, the extraction result is empty and the subsequent process cannot be executed; after using the data processing tool Pandas to organize and format the retrieved data, the data format of "XXX company holds XXX certificate" and "The company holding XXX certificate has XXX" is formed.

[0032] Compared with the prior art, the method and system for automatic query and analysis of bidding industry ledger information of the present invention has the following beneficial effects:

[0033] 1. The present invention integrates large-scale language models, JointBERT models and automatic query and analysis of vector databases. By combining natural language processing technology with data storage and retrieval technology, it can achieve efficient and accurate management and query of ledger information and certificate information, and solve the problems of low efficiency, insufficient accuracy and time-consuming and labor-intensive ledger information management and certificate information database query and analysis in the traditional bidding industry;

[0034] 2. The present invention can greatly reduce the time and workload of manual query and analysis of ledger information and certificate information through automated data collection and preprocessing; by utilizing the Embedding model, combined with the Milvus vector database and the Elasticsearch full-text search engine, it can achieve fast semantic search and keyword matching, speed up information retrieval, and ensure the reliability of query results; it can handle complex query requirements, and can provide detailed and relevant analysis reports for both certificate information and ledger information, which is convenient for users to refer to and use, thereby providing strong technical support for the digital transformation of the bidding industry and promoting technological progress in the bidding industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Attached Figure 1 is a flowchart of the method implementation of the first embodiment of the present invention;

[0036] Attached Figure 2 is a flowchart for implementing step S1 in the first embodiment of the present invention;

[0037] Attached Figure 3 is a flowchart for implementing step S2 in the first embodiment of the present invention;

[0038] Attached Figure 4 is a flowchart for implementing step S3 in the first embodiment of the present invention;

[0039] Attached Figure 5 is a flowchart for implementing step S4 in the first embodiment of the present invention;

[0040] Attached Figure 6 This is a module connection block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0042] Embodiment 1:

[0043] Combined with Figure 1-5 This embodiment proposes a method for automatic query and analysis of bidding industry ledger information, which includes the following steps:

[0044] S1. Collect ledger information and various certificate information of target competitors from the Internet through automated data crawling technology, and perform the following pre-processing operations on the collected information: implement data governance processes to ensure the legitimacy, accuracy and consistency of data sources; perform data cleaning, remove invalid or erroneous information, fill in missing values, and convert data into a structured format.

[0045] The ledger information collected in this step includes the target competitors, the certificates contained by the target competitors, and the categories to which the certificates belong; the collected certificate information includes but is not limited to the certificate query website, the certificate introduction, the certificate field introduction, and the certificate level introduction.

[0046] S2. Use the Embedding model to convert the key field data in the ledger information and the text data in the certificate information into low-dimensional dense vectors, store them in the Milvus vector database, and create indexes on the vector fields to facilitate fast retrieval. At the same time, store the text content in the ledger information and certificate information in the Elasticsearch database. Elasticsearch has a distributed full-text search engine that can achieve a retrieval speed of milliseconds for billions of data.

[0047] The Milvus vector database can perform semantic search and matching by calculating vector similarity, and Elasticsearch can perform content retrieval based on keywords. By combining the Milvus vector database and the Elasticsearch database, fast semantic and keyword matching can be achieved, significantly improving the accuracy and efficiency of retrieval.

[0048] S3. Build a training set based on ledger information and certificate information to train a JointBERT model that can recognize intent and extract entity content. This capability of the JointBERT model enables it to provide more accurate and efficient analysis results when processing complex ledger information and certificate data.

[0049] S4. Input the text content to be queried into the trained JointBERT model. First, the JointBERT model is used to perform intent recognition on the input text to identify whether the input text is to query ledger information or certificate information.

[0050] (a) For ledger information query: ① The JointBERT model uses natural language processing technology to extract the company name and certificate name from the input text. If the JointBERT model does not extract the company name and certificate name from the input text, the extraction result is empty and the subsequent process cannot be executed. ② The JointBERT model calculates the similarity between the extracted company name and certificate name and the content stored in the Milvus vector database, and selects the result with the highest similarity. ③ Based on the selection result, the ledger information is further retrieved in the Elasticsearch database, and the data processing tool Pandas is used to organize and format the retrieved data to form the data format of "XXX company holds XXX certificate" and "Companies holding XXX certificate have XXX". ④ The organized data is combined with the input text to form a prompt, which is input into the Large Language Model (LLM) to generate a detailed analysis report;

[0051] (b) For certificate information query: ① The JointBERT model calculates semantic similarity and keyword matching between the input text and the certificate content in the Milvus vector database and the Elasticsearch database, and obtains the top 20 matching items respectively, and a total of 40 relevant items. ② The reranker model is used to re-arrange the matched items and select the top ten most relevant results. ③ The selected results are combined with the input text to construct a prompt, and a detailed analysis report is generated through a large language model (LLM).

[0052] Embodiment 2:

[0053] Combined with Figure 6 This embodiment proposes an automatic query and analysis system for bidding industry ledger information, which includes:

[0054] The data collection and preprocessing module is used to collect ledger information and various certificate information containing target competitors and preprocess the collected information;

[0055] The vector representation and storage module is used to use the Embedding model to convert the key field data in the ledger information and the text data in the certificate information into low-dimensional dense vectors, store them in the Milvus vector database, and create indexes on the vector fields. It is also used to store the text content in the ledger information and certificate information in the Elasticsearch database;

[0056] The JointBERT model training module is used to build a training set based on ledger information and certificate information to train a JointBERT model capable of intent recognition and entity content extraction. This capability of the JointBERT model enables it to provide more accurate and efficient analysis results when processing complex ledger information and certificate data.

[0057] The intent recognition and report generation module is used to input the text content to be queried into the trained JointBERT model. The JointBERT model first recognizes the intent of the input text, then executes the corresponding search and analysis process based on the recognition results, and finally generates a detailed analysis report.

[0058] In this embodiment, the data collection and preprocessing module involved first collects the target competitor's ledger information and various certificate information from the Internet through automated data crawling technology, and then implements a data governance process for the collected information to ensure the legitimacy, accuracy and consistency of the data source. Finally, the collected information is cleaned to remove invalid or erroneous information, fill in missing values, and convert the data into a structured format.

[0059] The ledger information collected by the data collection and preprocessing module includes target competitors, certificates contained by target competitors, and the categories to which the certificates belong; the certificate information collected by the data collection and preprocessing module includes the certificate query website, certificate introduction, certificate field introduction, and certificate level introduction.

[0060] In this embodiment, an index is established on the vector field of the Milvus vector database to facilitate fast retrieval; Elasticsearch has a distributed full-text search engine that can achieve a retrieval speed of 100 million data in milliseconds. When performing subsequent queries, the Milvus vector database can perform semantic search and matching by calculating vector similarity, and Elasticsearch can perform content retrieval based on keywords. By combining the Milvus vector database and the Elasticsearch database, fast semantic and keyword matching can be achieved, significantly improving the accuracy and efficiency of retrieval.

[0061] In this embodiment, the intent recognition and report generation module first uses the JointBERT model to recognize the intent of the input text, identifying whether the input text is to query the ledger information or the certificate information, and then:

[0062] (a) For ledger information query: ① The JointBERT model uses natural language processing technology to extract the company name and certificate name from the input text. If the JointBERT model does not extract the company name and certificate name from the input text, the extraction result is empty and the subsequent process cannot be executed. ② The JointBERT model calculates the similarity between the extracted company name and certificate name and the content stored in the Milvus vector database, and selects the result with the highest similarity. ③ Based on the selection result, the ledger information is further retrieved in the Elasticsearch database, and the data processing tool Pandas is used to organize and format the retrieved data to form the data format of "XXX company holds XXX certificate" and "Companies holding XXX certificate have XXX". ④ The organized data is combined with the input text to form a prompt, which is input into the Large Language Model (LLM) to generate a detailed analysis report;

[0063] (b) For certificate information query: ① The JointBERT model calculates semantic similarity and keyword matching between the input text and the certificate content in the Milvus vector database and the Elasticsearch database, and obtains the top 20 matching items respectively, and a total of 40 relevant items. ② The reranker model is used to re-arrange the matched items and select the top ten most relevant results. ③ The selected results are combined with the input text to construct a prompt, and a detailed analysis report is generated through a large language model (LLM).

[0064] In summary, the use of an automated query and analysis method and system for ledger information in the bidding industry of the present invention can achieve efficient and accurate management and query of ledger information and certificate information, and solve the problems of low efficiency, lack of accuracy, and time and labor consumption in ledger information management and certificate information library query and analysis in the traditional bidding industry.

[0065] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method for automatic query and analysis of bidding industry ledger information, characterized in that: The steps include: S1. Collect the ledger information and various certificate information of target competitors and pre-process the collected information; S2. Using the Embedding model, the key field data in the ledger information and the text data in the certificate information are converted into low-dimensional dense vectors, stored in the Milvus vector database, and indexed on the vector fields. At the same time, the text content in the ledger information and certificate information is stored in the Elasticsearch database. S3. Build a training set based on ledger information and certificate information to train a JointBERT model that can perform intent recognition and entity content extraction. S4. Input the text content to be queried into the trained JointBERT model. First, the JointBERT model is used to identify the intent of the input text. Then, the corresponding search and analysis process is performed based on the recognition results, and finally a detailed analysis report is generated.

2. According to claim 1, a bidding industry account information automatic query and analysis method is characterized in that: The step S1 specifically includes the following operations: Through automated data crawling technology, we collect target competitors’ ledger information and various certificate information from the Internet; Implement data governance processes for collected information to ensure the legitimacy, accuracy and consistency of data sources; Perform data cleaning on the collected information, remove invalid or erroneous information, fill in missing values, and convert the data into a structured format.

3. According to claim 2, a method for automatic query and analysis of bidding industry ledger information is characterized in that: Execute step S1, and collect the ledger information including the target competitor, the certificates contained by the target competitor, and the category to which the certificates belong; The collected certificate information includes the certificate query website, certificate introduction, certificate field introduction and certificate level introduction.

4. According to claim 1, a bidding industry account information automatic query and analysis method is characterized in that: Execute step S4, first use the JointBERT model to perform intent recognition on the input text to identify whether the input text is to query ledger information or certificate information; (a) For ledger information query: ① The JointBERT model uses natural language processing technology to extract company names and certificate names from the input text; ② The JointBERT model calculates the similarity between the extracted company names and certificate names and the content stored in the Milvus vector database, and selects the result with the highest similarity; ③ Based on the selected results, further search for ledger information in the Elasticsearch database, and use the data processing tool Pandas to organize and format the retrieved data; ④ The organized data is combined with the input text to form a prompt, which is input into the large language model to generate a detailed analysis report; (b) For certificate information query: ① The JointBERT model calculates semantic similarity and keyword matching between the input text and the certificate content in the Milvus vector database and the Elasticsearch database to obtain relevant entries. ② The reranker model is used to re-rank the matched entries and select the top ten most relevant results. ③ The selected results are combined with the input text to construct a prompt and generate a detailed analysis report through a large language model.

5. According to claim 4, a method for automatic query and analysis of bidding industry ledger information is characterized in that: When executing (a), if the JointBERT model does not extract the company name and certificate name from the input text, the extraction result is empty and the subsequent process cannot be executed; after using the data processing tool Pandas to organize and format the retrieved data, the data format of "XXX company holds XXX certificate" and "Companies holding XXX certificate have XXX" is formed.

6. An automated query and analysis system for bidding industry ledger information, characterized in that: It includes: The data collection and preprocessing module is used to collect ledger information and various certificate information containing target competitors and preprocess the collected information; The vector representation and storage module is used to use the Embedding model to convert the key field data in the ledger information and the text data in the certificate information into low-dimensional dense vectors, store them in the Milvus vector database, and create indexes on the vector fields. It is also used to store the text content in the ledger information and certificate information in the Elasticsearch database; The JointBERT model training module is used to build a training set based on ledger information and certificate information to train a JointBERT model capable of intent recognition and entity content extraction; The intent recognition and report generation module is used to input the text content to be queried into the trained JointBERT model. The JointBERT model first recognizes the intent of the input text, then executes the corresponding search and analysis process based on the recognition results, and finally generates a detailed analysis report.

7. According to claim 6, the automatic query and analysis system for bidding industry ledger information is characterized in that: The data collection and preprocessing module first collects the target competitor's ledger information and various certificate information from the Internet through automated data crawling technology, and then implements a data governance process for the collected information to ensure the legality, accuracy and consistency of the data source. The collected information is then cleaned to remove invalid or erroneous information, fill in missing values, and convert the data into a structured format.

8. The automatic query and analysis system for bidding industry account information according to claim 7 is characterized in that: The ledger information collected by the data collection and preprocessing module includes target competitors, certificates contained by the target competitors, and the categories to which the certificates belong; The certificate information collected by the data collection and preprocessing module includes the certificate query website, the certificate introduction, the certificate field introduction and the certificate level introduction.

9. The automatic query and analysis system for bidding industry account information according to claim 6 is characterized in that: The intent recognition and report generation module first uses the JointBERT model to recognize the intent of the input text to identify whether the input text is to query the ledger information or the certificate information, and then: (a) For ledger information query: ① The JointBERT model uses natural language processing technology to extract company names and certificate names from the input text; ② The JointBERT model calculates the similarity between the extracted company names and certificate names and the content stored in the Milvus vector database, and selects the result with the highest similarity; ③ Based on the selected results, further search for ledger information in the Elasticsearch database, and use the data processing tool Pandas to organize and format the retrieved data; ④ The organized data is combined with the input text to form a prompt, which is input into the large language model to generate a detailed analysis report; (b) For certificate information query: ① The JointBERT model calculates semantic similarity and keyword matching between the input text and the certificate content in the Milvus vector database and the Elasticsearch database to obtain relevant entries. ② The reranker model is used to re-rank the matched entries and select the top ten most relevant results. ③ The selected results are combined with the input text to construct a prompt and generate a detailed analysis report through a large language model.

10. The automatic query and analysis system for bidding industry ledger information according to claim 9 is characterized in that: When executing (a), if the JointBERT model does not extract the company name and certificate name from the input text, the extraction result is empty and the subsequent process cannot be executed; after using the data processing tool Pandas to organize and format the retrieved data, the data format of "XXX company holds XXX certificate" and "Companies holding XXX certificate have XXX" is formed.

Images