An artificial intelligence-based massive government affair appeal data induction method

Abstract

The application discloses a kind of based on artificial intelligence's mass government affair appeal data induction method, belong to artificial intelligence and big data processing technical field, the implementation of this method includes preprocessing stage, induction processing stage and statistical analysis stage, preprocessing stage, including split, classification and polishing step, by collating classification rule table, dynamic construction big model prompt word is to realize business rule injection, by classification realizes the parallelization of serial induction operation;Induction processing stage, obtain classification inside appeal, each classification corresponds to one and only one induction thread, realize the serial processing of appeal induction;Statistical analysis stage, utilize the library achievement of induction processing stage, output the appeal list most anticipated by the masses.This application can effectively improve the speed and quality of mass appeal data induction using large language model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and big data processing technology, specifically a method for summarizing massive government affairs request data based on artificial intelligence. Background Technology

[0002] Large-scale models far surpass traditional artificial intelligence in natural language understanding and text summarization, making them highly suitable for organizing and summarizing natural language information such as requests and suggestions in the government sector. However, when organizing and summarizing massive amounts of requests using large-scale models, issues such as token limitations, attention dilution, illusions, and processing performance problems often arise. Here is a specific example: For example, when compiling public demands, a massive amount of natural language descriptions are received. Each new demand must be compared with an existing list of summarized demands. If similar demands exist, they are integrated; otherwise, the new demand is added to the existing list. However, as the list of demands expands, the size of the prompts sent to the large model also increases. At this point, the large model encounters various problems more and more frequently, such as the token size exceeding the model context length limit, inability to find similar existing entries (attention dilution), exponential growth in interaction time, discrepancies between the returned result's entry number and the actual existing entry number (illusion), disordered summarized content (illusion), and incompatibility due to dependence on upstream content. Because each new demand must be summarized based on the list of summarized demands from previous demands, parallel processing is not possible, resulting in underutilization of computing resources and extremely long processing cycles. Summary of the Invention

[0003] The technical objective of this invention is to address the above-mentioned shortcomings by providing a method for summarizing massive government service requests based on artificial intelligence, which can effectively improve the speed and quality of summarizing massive requests using large language models.

[0004] The technical solution adopted by this invention to solve its technical problem is: A method for summarizing massive government service requests based on artificial intelligence is proposed. The implementation of this method includes a preprocessing stage, a summarizing stage, and a statistical analysis stage. The preprocessing stage includes splitting, classifying, and refining steps. By organizing the classification rule table, large model prompt words are dynamically constructed to realize the injection of business rules. The classification enables the parallelization of serial inductive operations. In the inductive processing stage, requests within each category are obtained. Each category corresponds to one and only one inductive thread, realizing the serial processing of request induction. In the statistical analysis phase, the database writing results from the inductive processing phase are used to output a list of the most anticipated demands from the public. The data supports flexible statistics based on dimensions such as region and personnel category. It guides the work direction of government departments at all levels through data. By designing foreign key relationships between the three key tables—the original demand table, the split demand table, and the summary demand table—massive amounts of unstructured text are transformed into structured multidimensional index tags. This significantly reduces the memory paging overhead (I / O overhead) when performing multi-attribute cross-aggregation queries on massive amounts of text, and significantly improves the concurrent retrieval response speed in the big data analysis phase. In the preprocessing stage, the large model breaks down the original request into one or more sub-requests and refines them. Similar requests are grouped into the same group according to classification rules. In the induction processing stage, corresponding induction threads are launched concurrently for different groups to achieve parallel processing. However, within each specific group, the induction thread dedicated to that group performs serial aggregation and induction of the requests in the group according to their order.

[0005] The following example illustrates the reason for the serial processing mechanism of request summarization: If there are three independent, split request items, then... 1. Free bus rides are provided for seniors aged 60 and above; 2. Seniors aged 60 and above can ride the subway for free; 3. Disabled veterans can ride public transportation (buses and subways) for free.

[0006] (1) After summarizing Article 1 and Article 2, we can obtain the following summary request: Seniors aged 60 and above can ride buses and subways for free; (2) By summarizing the demands in Article 3 and Step (1) again, we can obtain a new summary demand: elderly people over 60 years old and veterans can ride buses and subways for free; Therefore, the third requirement needs to be based on the list of inductive results from the first two requirements. If processed in parallel, multiple similar inductive results may be generated, rendering the induction meaningless. Therefore, each category can only have one processing thread, maintaining a list of inductive requests and processing each split request serially.

[0007] In addition to collecting the information about the original requests, the key to data collection is to include the key business dimensions required for statistical analysis, such as: district, personnel identity category, age, gender, etc.

[0008] Furthermore, the preprocessing first involves a splitting step, followed by a classification or polishing step; it should be ensured that the splitting operation comes first, while the order of the other two operations can be arranged arbitrarily. The three steps can be completed separately by calling the large model, or they can be merged and then completed by the large model in one go.

[0009] Furthermore, the preprocessing stage specifically includes the following steps: Based on semantics, the original request containing multiple requests is split, and each split request contains only one independent and complete request. Based on the rules, the split demands are categorized; Based on the example, refine the split requirements; In the message queue, topics are created according to categories, and the split requests after completing the above categorization steps are written to the message queue. The message queue is used to smooth out peak and off-peak traffic, decouple processes, and improve parallel processing efficiency. The inductive processing stage specifically includes the following steps: Retrieve requests within each category of the message queue; Within a category, based on semantics, the similarity of each request is compared. If there is high similarity, similar requests are merged into a single integrated request and added to the request list. If there is no high similarity, the request is added to the request list as a new independent request.

[0010] Furthermore, the inductive processing stage also includes a dynamic hotspot splitting and virtual sub-queue routing mechanism based on semantic entities: when the backlog of the message queue corresponding to a certain secondary category exceeds a set threshold Y, the hotspot splitting procedure is triggered; using the fine-grained semantic entity tags in the multi-level and multi-dimensional tags output in the preprocessing stage as hash keys, a unique physical queue index suffix is ​​calculated through a deterministic hash algorithm, and the request texts with the same or similar semantic entity tags are accurately routed to the corresponding virtual sub-queues; each virtual sub-queue is allocated an exclusive inductive thread and a corresponding memory vector array segment lock, ensuring that requests with the same semantic entity are serially aggregated without omission in their respective virtual sub-queues, while realizing cross-thread concurrent consumption of ultra-large hotspot categories.

[0011] Furthermore, a lightweight multilingual text representation model is adopted to add a vectorized field to the summary list. Each time a new requirement is received, it is first vectorized and then compared with the vectors from the existing summary list. Multiple entries (up to N) with similarity exceeding the set threshold X are selected and sent to the large model, thereby fixing the size of the prompt words and limiting the infinite expansion of the prompt word context.

[0012] The reason for setting it to multiple instead of one is mainly because the best matching item obtained by filtering through the lightweight multilingual text representation model may not be the item with the best actual semantic match. Providing N items allows the large model to select the closest one and then summarize it to obtain the one with the closest semantics.

[0013] Therefore, the threshold X should not be set too high. In this system, it is preferred to set it to 0.7, and N can be set to 5 to 10.

[0014] Vector comparison is implemented using a lightweight multilingual text representation model. If the value is below a set threshold, a large model illusion avoidance mechanism is activated. By initiating multiple rounds of re-induction and taking the highest value, the negative risk of filtering out reasonable results in vector comparison is avoided.

[0015] This involves employing a lightweight multilingual text representation model and evaluating the similarity between the split and inductive claims through vectorized comparison. If the similarity difference is below a threshold (preferably set to 0.8 to 0.85), hallucinations may occur. For inductive results that may indicate hallucinations, an evaluation process is used to rule out the possibility that correct inductive results actually have low similarity. The same request summarization task can be resent to the large model up to 3 times. If the similarity summarization result is higher than the threshold, it will be directly adopted. If none of the 3 times exceed the threshold, it means that the actual summarization result in this scenario may be lower than the threshold. The result with the highest similarity will be taken, and this summarization will be specially marked when it is entered into the database for easy manual review later.

[0016] Furthermore, the large model illusion avoidance mechanism also includes dynamic parameter intervention and input reverse feedback control during the retry inference stage: when the similarity difference is lower than a set threshold to trigger a re-induction task, the system dynamically lowers the underlying generation temperature parameter of the large model before the next call to the large model, in order to suppress the randomness and divergence of the large model generation; at the same time, the system injects the previously judged erroneous induction text as a negative prompt into the new round of input prompts, explicitly restricting the large model from outputting the erroneous text again in this inference, thereby reducing the model illusion rate through the bidirectional closed-loop feedback control of 'parameter frequency variation-input correction'.

[0017] Furthermore, a memory vector array is constructed based on NumPy. Data in a category is processed by one and only one thread. Therefore, one thread maintains a vector array, and the size of the vector array is only the list of requests that have been summarized in this category. The memory vector array has a content structure including index subscripts, the original text of the summary appeal, and the vector value of the summary appeal. The index subscript does not change after the initial insertion. After each summary, if an existing summary result is matched, only the fields of the original text of the summary appeal and the vector value of the summary appeal change, and the index subscript is updated by the matching index returned by the large model. If no existing summary result is matched, a new auto-incrementing subscript is inserted. Its update mechanism is automatic updating after summarization; The memory usage calculation method is as follows: estimated number of summaries × (memory size occupied by vectors + average memory size occupied by summarized content + average memory size occupied by subscripts); Monitor the load rate of computing resources. When the load is not full, automatically increase the number of concurrent processes in the preprocessing stage until it is close to full load. When the load is too high, automatically reduce the number of processes in the preprocessing stage until the computing resources are close to full load.

[0018] Furthermore, in the statistical analysis phase, the table structure and algorithm design for obtaining the most desired results are as follows: Three key tables are designed: Table 1 is the original request table. Key fields include the original request ID (primary key), the original request content, the identity information of the requester, the affiliated organization, and the processing flag. Through the processing flag, data interruption recovery and incremental processing can be realized. Table 2 is the split request table. Key fields include split request ID (primary key), category, original request ID (foreign key), split request content, and summarized request ID (foreign key). Table 3 is a summary request table. Key fields include the summary request ID (primary key) and the summary request content. When the summary thread restarts, it can load the summarized data according to the category and initialize the memory vector array.

[0019] As can be seen from the above design, Table 2 can be linked to Table 1 and Table 3 through foreign keys. Through the linking of the three tables, accurate analysis can be carried out based on multiple dimensions such as the identity information of the person making the request, the organization to which they belong, and the type of request.

[0020] By leveraging foreign key relationships between the three tables, we can achieve multi-dimensional statistical analysis of public demands, accurately obtain the ranking of the most anticipated demands, and guide the direction of government work.

[0021] The present invention also claims a device for summarizing massive government affairs data based on artificial intelligence, comprising: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the above method.

[0022] The present invention also claims a computer-readable medium storing computer instructions that, when executed by a processor, enable the implementation of the above-described method.

[0023] Compared with existing technologies, the method for summarizing massive government service requests based on artificial intelligence proposed in this invention has the following advantages: Leveraging the classification and semantic induction capabilities of large models, this method categorizes and summarizes texts expressed in natural language. It then utilizes lightweight multilingual text representation models (such as paraphrase-multilingual-MiniLM-L12-v2) for semantic similarity screening and verification of induction results. This avoids the attention loss and illusion problems exhibited by large models when processing massive amounts of data. Furthermore, it supports multi-dimensional analysis of the induction results, accurately obtaining the ranking of the most anticipated demands to guide the work direction of government departments. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the preprocessing stage according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the inductive processing stage provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of an avoidance mechanism for attention dilution in large models provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an avoidance mechanism for the large model illusion problem provided in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to specific embodiments.

[0026] This invention provides an artificial intelligence-based method for summarizing massive government service requests. It leverages the classification and semantic induction capabilities of large models to categorize and summarize text expressed in natural language. A lightweight multilingual text representation model (e.g., paraphrase-multilingual-MiniLM-L12-v2) is used for semantic similarity filtering and verification of the summarization results. This avoids the attention loss and illusion problems exhibited by large models when processing massive amounts of data. Furthermore, it supports multi-dimensional analysis of the summarization results, accurately identifying the most anticipated requests and guiding the work direction of government departments.

[0027] The implementation of this method includes a preprocessing stage, an inductive processing stage, and a statistical analysis stage. By dividing the process into multiple stages, parallel processing of the serial inductive process is achieved.

[0028] The preprocessing stage, including splitting, classifying, and polishing, should ensure that the "split" operation comes first, while the order of the other two operations can be arranged arbitrarily. These three steps can be completed separately using the larger model, or they can be merged and then completed by the larger model in one go.

[0029] During the classification phase, by organizing the classification rule table, dynamically constructing large model prompt words, and injecting business rules, the serial inductive operation is parallelized through classification.

[0030] Between the preprocessing stage and the inductive processing stage, message queues are used to smooth out peaks and valleys and decouple processes, thereby improving parallel processing efficiency.

[0031] Optimal resource utilization is achieved by adjusting the number of threads in the preprocessing stage, further improving overall task completion efficiency and computational resource utilization. Following the "serial processing mechanism for request summarization," each category in the summarization stage has one and only one summarization thread, ensuring that the summarization covers all information without omissions. The summarization thread maintains an array of summarization result vectors, using a lightweight multilingual text representation model to compare vectors and select the most relevant records for the larger model to summarize, thereby reducing the problem of attention dilution in the larger model.

[0032] This document describes a memory-based vector array implemented using NumPy, including its content structure, update mechanism, initialization method, and memory usage calculation method. Vector comparison is implemented using a lightweight multilingual text representation model; if the value is below a set threshold, a large model illusion avoidance mechanism is activated. A multi-round re-induction and highest-value evaluation mechanism is employed to avoid the negative risk of filtering out reasonable results during vector comparison.

[0033] To achieve multi-dimensional statistical analysis of public demands, accurately obtain the ranking of the most anticipated demands, and guide the work direction of government departments.

[0034] The specific implementation of this method is as follows: Phase 1: Preprocessing, including splitting, classifying, and polishing, with the workflow as follows: Figure 1 As shown. The main objectives for this phase are as follows: 1. Based on semantics, split the original request containing multiple requests, so that each split request contains only one independent and complete request. Ensure that the splitting operation occurs before the other two operations.

[0035] 2. Based on the rules, classify the split requests.

[0036] 3. Based on the example, refine the split requirements.

[0037] 4. In the message queue, create topics according to categories, and write the split requests after completing the above steps into the message queue.

[0038] Phase Two: Summarizing and concluding phase, the workflow of which is as follows: Figure 2 As shown, the main objectives for this phase are as follows: 1. Retrieve requests within each category of the message queue.

[0039] 2. Within a category, based on semantics, the similarity of each request is compared. If there is high similarity, similar requests are merged into a single integrated request and added to the request list. If there is no high similarity, it is added to the request list as a new independent request.

[0040] Phase Three: Statistical Output. In this phase, utilizing the results of Phase Two's database writing, a list of the most anticipated public demands is output. This allows for flexible statistical analysis based on dimensions such as region and personnel category, using data to guide the work of government departments at all levels.

[0041] In the implementation of this method, in addition to collecting the information on the original requests, the key to data collection is to include the key business dimensions required for statistical analysis, such as: district, personnel identity category, age, gender, etc.

[0042] Based on business rules, establish classification rules for requests. Multi-level classification should be implemented as much as possible, down to the finest business granularity; the finer the granularity, the easier it is for subsequent parallel operations. This classification rule table serves as a dynamic routing label tree for distributed parallel computing at the computer's underlying layer. It is used to accurately schedule and dynamically distribute unstructured request text streams to support subsequent concurrent operations.

[0043] The classification information includes each level of classification and its code, classification description, and examples, as shown in Table 1 below: Table 1 - Classification Information .

[0044] In this method, the first-stage prompt word generation mechanism is as follows: The main tasks at this stage are splitting, classifying, and polishing, corresponding to three categories of prompt words: (1) The large model is required to understand the original demands and split the original demands into one or more split demands. A few split examples are given through the few shots method to improve the split quality.

[0045] (2) Introduce prompt words into the data classification rules above, such as “finest granular classification name”, “finest granular classification code”, “classification rule” and “classification example”, to improve classification accuracy.

[0046] (3) After refining and splitting the large model, the requirements are to provide a few examples using the few shots method to improve the quality of the refinement.

[0047] In actual business scenarios, the three types of prompts can be merged into two or even one based on the text size of the split examples, classification criteria, and polished examples, as well as the context length limitations of the large model and the actual processing quality. This reduces the number of interactions with the large model.

[0048] The second-stage demands summary adopts a serial processing mechanism. Through the classification work in the first stage, similar demands have been divided into the same group, so the summary can be based on the items within the group.

[0049] In summarizing items within a group, a sequential process must be followed. The reason for this is illustrated by the following example: if there are three independent, separately defined request items: (1) Free bus rides are provided for seniors aged 60 and above; (2) Seniors aged 60 and above can ride the subway for free; (3) Disabled veterans are exempt from public transportation (bus and subway). 1. After summarizing Article (1) and Article (2), we can obtain the following summary request: elderly people over 60 years old can ride buses and subways for free; 2. By summarizing the demands in point (3) and the demands summarized in step 1, we can obtain a new summary demand: elderly people over 60 years old and veterans can ride buses and subways for free. Therefore, clause (3) needs to be based on the list of induction results from the first two clauses. If processed in parallel, multiple similar induction results may be generated, thus losing the meaning of induction. Therefore, each category can only have one processing thread to maintain a list of induction requests and process each split request serially.

[0050] This method employs the following avoidance mechanism for attention dilution in large models: In conventional industry practice, an existing list of requests is compiled, incorporating prompts before being submitted to a large model. When the existing request list is small, the model's feedback is relatively accurate. However, in scenarios involving massive amounts of requests, the compiled list grows increasingly large, leading to longer prompt contexts. This causes the model's response time to increase exponentially, and it frequently returns incorrect results, such as incorrect matching indexes or failure to find similar information. Essentially, this is due to the increased probability of attention dilution caused by excessively long contexts.

[0051] To address the aforementioned issues, this method employs a lightweight multilingual text representation model, combined with... Figure 3 As shown, a vectorization field is added to the induction list. Each time a new requirement is received, it is first vectorized, and then compared with the vector from the induction list. Multiple entries (up to N) with an approximation greater than the set threshold X are selected and sent to the large model. This fixes the size of the prompt words and limits the infinite expansion of the prompt word context.

[0052] The reason for setting it to multiple instead of one is mainly because the best matching item obtained by filtering through the lightweight multilingual text representation model may not be the item with the best actual semantic match. Providing N items allows the large model to select the closest one and then summarize it to obtain the one with the closest semantics.

[0053] Therefore, the threshold X should not be set too high. In this system, it is currently set to 0.7, and N can be set to 5 to 10.

[0054] This method addresses the avoidance mechanism for large-scale illusions: The illusion created by the large model cannot be completely eliminated on its own. As the number of calls increases, the illusion will always occur at a certain rate, as manifested in this scenario as follows: The candidate list contains highly matching options, but no match is still returned. If an option is matched but the returned list number is inaccurate, it will result in incorrect overridden content.

[0055] To address the aforementioned issues, this method also introduces a lightweight multilingual text representation model. Through vectorized comparison, it evaluates the similarity between splitting and summarizing requests. If the similarity difference is below a threshold (set to 0.8 to 0.85), hallucinations may occur.

[0056] Combination Figure 4 As shown, the evaluation process is used to rule out the possibility that the correct induction results have a low similarity to hallucinations by summarizing the possible causes. The same request summarization task can be resent to the large model up to 3 times. If the similarity summarization result is higher than the threshold, it will be directly adopted. If none of the 3 times exceed the threshold, it means that the actual summarization result in this scenario may be lower than the threshold. The result with the highest similarity will be taken, and this summarization will be specially marked when it is entered into the database for easy manual review later.

[0057] As a preferred embodiment of the present invention, in order to completely overcome the inherent flaw of large models continuously falling into a logical dead loop and generating continuous hallucinations under the same prompt words, the present invention does not blindly re-issue the task when triggering the re-induction task, but introduces a dynamic parameter intervention mechanism with a feedback control loop. Specifically, when the semantic similarity calculated by the lightweight multilingual text representation model is lower than a set threshold, and the system determines that the current output is suspected to be a hallucination of the large model, it will automatically initiate bidirectional intervention: First, dynamic decay of the temperature coefficient is triggered at the underlying algorithm level. During the first normal call, the large model's temperature is set to 0.7 to ensure the natural refinement of the language. Once a retry is triggered, the system forcibly reduces the temperature parameter to 0.2 in the second call and to 0.0 in the third retry. By progressively reducing the temperature, the large model is forced to abandon creative divergence during generation, completely shifting to extremely conservative literal semantic reasoning, thus suppressing random illusions at the algorithmic source. Second, negative prompts are injected in reverse at the input layer. The system automatically captures erroneous induction results judged as illusions in the previous round and transforms them into negative constraints, inserting them into the context of the prompts in the new round (e.g., explicitly declaring: 'Note that the following induction result has been judged as erroneous by the system and is absolutely forbidden from being output again in this reasoning: [erroneous text]'). By combining dynamic frequency conversion of underlying parameters with reverse feedback at the input end in a closed-loop correction architecture, the authenticity and fidelity of the fully automated summarization of massive periodic survey data are greatly improved, making the final summary request table more objective and valuable for decision-making.

[0058] The performance optimization mechanisms in this method include constructing a memory-based vector array and making full use of computing resources.

[0059] 1. Construct a memory-based vector array The solution for "avoiding attention dilution in large models" utilizes a vector library of induction lists. Each time an induction is completed, the vector library needs to be updated according to the updated induction requirements, and then a search is performed based on the new vector library. The frequency of updating and reading / writing the vector library is very high. When multiple threads are running concurrently, the vector library faces significant pressure, thus slowing down the overall execution efficiency.

[0060] This paper proposes a solution for constructing in-memory vector arrays based on NumPy: (1) According to the serial processing mechanism summarized in the above-mentioned demands, the data in a category is processed by one and only one thread. Therefore, one thread maintains a vector array, and the size of the vector array is only the list of demands summarized in this category.

[0061] (2) The vector library contains: index subscript, summary content and vector value of summary content, wherein the index subscript does not change after the first insertion; after each summary, if an existing summary result is matched, only the two fields of summary content and vector value change, and the index subscript is updated by the matching index returned by the large model; if no existing summary result is matched, a new auto-incrementing subscript is inserted.

[0062] (3) The size of the vector array is limited by the system memory size. The memory usage calculation method is as follows: Taking the paraphrase-multilingual-MiniLM-L12-v2 model as an example, the vector dimension generated by the model is 384 dimensions. The memory usage of a single line varies depending on the precision. For FP32, it is about 1.5KB (parameter of this project), and for FTP16, it is about 0.75KB. The remaining summarization content and index labels can be averaged after testing with a small amount of data. In this project, it is about 0.2KB. According to the formula, 100M of memory can store about 60,000 sets of summarization results, which fully meets the needs of a single classification in general surveys. Whether deployed in a virtual machine or a K8S Pod, allocating a few hundred megabytes of memory is not a problem, and the system resources can fully meet the requirements.

[0063] Calculation formula: Estimated number of totals × (vector memory size + average memory size of summarized content + average memory size of subscripts).

[0064] As a preferred embodiment of the present invention, in order to solve the problem of data skew and message queue backlog caused by sudden social hotspots under specific business categories or secondary categories, this system further introduces a dynamic hotspot splitting and virtual sub-queue diversion mechanism based on semantic entities in the summarization and processing stage. When the computing power resource monitoring module detects that the message queue backlog of a certain category exceeds the set threshold Y, the system no longer maintains a single global queue, but triggers dynamic splitting. The system directly extracts the downstream fine-grained 'core semantic entity keywords' (such as specific nouns such as 'bus' and 'subway') output by the large model in the preprocessing stage as hash routing factors.

[0065] The system utilizes a highly efficient deterministic hash algorithm to calculate the core semantic entity keywords and modulo the total number N of the proposed derived virtual sub-queues, concatenating them to generate virtual sub-queue routing keys with specific hash suffixes (such as 'Travel_01', 'Travel_02'). Through this mechanism, all request texts with the same semantic lineage (such as requests about 'bus,' 'bus,' and 'public bus' converged to the standard word 'public bus' after semantic alignment) will be forcibly and deterministically routed to the same physical partition or virtual sub-queue. On the consumer side, the system introduces a memory segmentation lock design, configuring each virtual sub-queue with mutually isolated local memory vector arrays and exclusive consumer worker threads. Each exclusive thread executes concurrently, but within each virtual sub-queue, strict local serial aggregation and summarization are performed. This architecture design completely avoids I / O contention and deadlocks caused by global large locks, and perfectly ensures that all requests with the same semantics can meet and be integrated, achieving high throughput and the effectiveness of globally summarized data.

[0066] (4) When a task thread exits for various reasons and is restarted, it is necessary to rebuild the memory vector array based on the classification list in the database. This part of the work is time-consuming, but the probability of needing to restart the application is very low and can be ignored.

[0067] Compared to using an external knowledge base, using a local thread array significantly reduces the number of network interactions and completely eliminates disk I / O contention in centralized services. In our test environment, the average execution efficiency is improved by approximately 450%.

[0068] 2. Mechanism for full utilization of computing resources While individual requests may have upstream and downstream relationships between their first and second phases, different requests can have their first and second phases executed in parallel. Therefore, this method uses message queues to connect the first and second phases, achieving decoupling and peak / valley smoothing. After the first phase completes and writes the data to the corresponding topic in the message queue, the corresponding thread for the second phase retrieves records from that topic and performs summarization. Thus, only when the first phase generates enough data to write to the message queues of each category can the worker threads of each category in the second phase execute the summarization operation at full capacity.

[0069] However, in reality, the number of various demands collected varies greatly, often concentrated in a few categories while the number of demands in other categories is relatively small. In other words, the summarization thread for categories with fewer demands may face the predicament of having countless calculable demands, while the summarization thread for categories with more demands will encounter a mechanism where a large amount of unprocessed data accumulates in the message queue due to the "serial processing mechanism of demand summarization," and can only be processed one by one serially.

[0070] Since the "serial processing mechanism for summarizing demands" cannot be changed, in order to improve the overall processing speed and the utilization rate of computing resources, this method proposes the following adjustment mechanism: (1) Monitor the load rate of computing resources. When the load is not full, automatically increase the number of concurrent processes in phase one until it is close to full load.

[0071] (2) When the load is too high, the number of processes in stage one will be automatically reduced until the computing resources are close to full load.

[0072] The third stage involves statistical analysis.

[0073] In addition to the aforementioned classification rule table and other dictionary tables, this system includes three key tables, among which: Table 1 is the original request table. The key fields are Original Request ID (primary key), Original Request Content, Requestor's Identity Information, Affiliated Organization, and Processing Flag. The processing flag enables data recovery from interruption and incremental processing.

[0074] Table 2 is the split request table. The key fields are split request ID (primary key), category, original request ID (foreign key), split request content, and summarized request ID (foreign key).

[0075] Table 3 is a summary request table, with key fields being the summary request ID (primary key) and the summary request content. When the summary thread restarts, it can load the summarized data according to the category and initialize the memory vector array.

[0076] As can be seen from the above design, Table 2 can be linked to Table 1 and Table 3 through foreign keys. Through the linking of the three tables, it is possible to conduct accurate analysis based on multiple dimensions such as the identity information of the person making the request, the organization to which they belong, and the type of request, so as to accurately obtain the ranking of the most anticipated requests and guide the work direction of government departments.

[0077] This invention also provides an artificial intelligence-based device for summarizing massive government affairs data, comprising: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is used to call the machine-readable program to implement the artificial intelligence-based method for summarizing massive government affairs data described in the above embodiments.

[0078] This invention also provides a computer-readable medium storing computer instructions. When executed by a processor, these instructions implement the artificial intelligence-based method for summarizing massive government service requests described in the above embodiments. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the embodiments described above, and enabling the computer (or CPU or MPU) of the system or apparatus to read and execute the program code stored in the storage medium.

[0079] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0080] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0081] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0082] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0083] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the code review methods in the different embodiments. These embodiments are also within the protection scope of the present invention.

Claims

1. A method for summarizing massive government service request data based on artificial intelligence, characterized in that, The implementation of this method includes a preprocessing stage, an inductive processing stage, and a statistical analysis stage. The preprocessing stage includes splitting, classifying, and refining steps. By organizing the classification rule table, large model prompt words are dynamically constructed to realize the injection of business rules. The classification enables the parallelization of serial inductive operations. In the inductive processing stage, requests within each category are obtained. Each category corresponds to one and only one inductive thread, realizing the serial processing of request induction. In the statistical analysis phase, the results of the database writing phase are used to output a list of the most anticipated demands by the public. In the preprocessing stage, the large model breaks down the original request into one or more sub-requests and refines them. Similar requests are grouped into the same group according to classification rules. In the induction processing stage, corresponding induction threads are launched concurrently for different groups to achieve parallel processing. However, within each specific group, the induction thread dedicated to that group performs serial aggregation and induction of the requests in the group according to their order.

2. The method for summarizing massive government affairs request data based on artificial intelligence according to claim 1, characterized in that, The preprocessing first involves a splitting step, followed by a classification or polishing step; The three steps can be completed separately by calling the large model, or they can be merged and then completed by the large model in one go.

3. A method for summarizing massive government affairs request data based on artificial intelligence according to claim 1 or 2, characterized in that, The preprocessing stage specifically includes the following steps: Based on semantics, the original request containing multiple requests is split, and each split request contains only one independent and complete request. Based on the rules, the split demands are categorized; Based on the example, refine the split requirements; In the message queue, topics are created according to categories, and the split requests after completing the above steps are written to the message queue. The inductive processing stage specifically includes the following steps: Retrieve requests within each category of the message queue; Within a category, based on semantics, the similarity of each request is compared. If there is high similarity, similar requests are merged into a single integrated request and added to the request list. If there is no high similarity, the request is added to the request list as a new independent request.

4. The method for summarizing massive government affairs request data based on artificial intelligence according to claim 1, characterized in that, The inductive processing stage also includes a dynamic hotspot splitting and virtual sub-queue routing mechanism based on semantic entities: when the backlog of the message queue corresponding to a certain secondary category exceeds a set threshold Y, the hotspot splitting program is triggered; using the fine-grained semantic entity tags in the multi-level and multi-dimensional tags output in the preprocessing stage as hash keys, a unique physical queue index suffix is ​​calculated through a deterministic hash algorithm, and the request texts with the same or similar semantic entity tags are accurately routed to the corresponding virtual sub-queues; each virtual sub-queue is allocated an exclusive inductive thread and a corresponding memory vector array segment lock, which ensures that requests with the same semantic entity are serially aggregated without omission in their respective virtual sub-queues, while realizing cross-thread concurrent consumption of ultra-large hotspot categories.

5. The method for summarizing massive government affairs request data based on artificial intelligence according to claim 1, characterized in that, A lightweight multilingual text representation model is adopted to add a vectorized field to the summary list. Each time a new requirement is received, it is first vectorized and then compared with the vector from the existing summary list. Multiple entries with similarity exceeding the set threshold X are sent to the large model, thereby fixing the size of the prompt words and limiting the infinite expansion of the prompt word context. Vector comparison is implemented using a lightweight multilingual text representation model. If the value is below a set threshold, a large model illusion avoidance mechanism is activated. By initiating multiple rounds of re-induction and taking the highest value, the negative risk of filtering out reasonable results in vector comparison is avoided.

6. A method for summarizing massive government affairs request data based on artificial intelligence according to claim 1 or 5, characterized in that, The large model hallucination avoidance mechanism also includes dynamic parameter intervention and input reverse feedback control during the retry inference stage: when the similarity difference is lower than a set threshold to trigger a re-induction task, the system dynamically lowers the underlying generation temperature parameter of the large model before the next call to the large model to suppress the randomness and divergence of the large model generation; at the same time, the system injects the previously judged erroneous induction text as a negative prompt word into the new round of input prompt words, explicitly restricting the large model from outputting the erroneous text again in this inference, and reducing the model hallucination rate through the two-way closed-loop feedback control of 'parameter frequency conversion-input correction'.

7. A method for summarizing massive government affairs request data based on artificial intelligence according to claim 1 or 4, characterized in that, Constructing a memory vector array using NumPy Its content structure includes index subscripts, the original text of the summary request, and the vector value of the summary request. The index subscripts do not change after the initial insertion. After each summary, if an existing summary result is matched, only the fields of the original text of the summary request and the vector value of the summary request change, and the index subscripts are updated by the matching index returned by the large model. If no existing summary result is matched, a new auto-incrementing subscript is inserted. Its update mechanism is automatic updating after summarization; The memory usage calculation method is as follows: estimated number of summaries × (memory size occupied by vectors + average memory size occupied by summarized content + average memory size occupied by subscripts); Monitor the load rate of computing resources. When the load is not full, automatically increase the number of concurrent processes in the preprocessing stage until it is close to full load. When the load is too high, automatically reduce the number of processes in the preprocessing stage until the computing resources are close to full load.

8. The method for summarizing massive government affairs request data based on artificial intelligence according to claim 1, characterized in that, In the statistical analysis phase, three key tables were designed: Table 1 is the original request table. Key fields include the original request ID, the original request content, the identity information of the requester, the affiliated organization, and the processing flag. Through the processing flag, data interruption recovery and incremental processing can be realized. Table 2 is the split request table. Key fields include split request ID, category, original request ID, split request content, and summarized request ID. Table 3 is a summary of requests, with key fields including the summary request ID and the summary request content. When the induction thread restarts, it can load the induction data according to the categories and initialize the memory vector array.

9. A device for summarizing massive government affairs requests based on artificial intelligence, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to implement the method according to any one of claims 1 to 8.

10. A computer-readable medium, characterized in that, The computer-readable medium stores computer instructions that, when executed by a processor, enable the implementation of the method described in any one of claims 1 to 8.

Images