Method for optimizing procurement review consistency detection based on reinforcement learning feedback mechanism

Abstract

The application provides a procurement review consistency detection optimization method based on a reinforcement learning feedback mechanism, through constructing a double-branch parallel execution process, a first branch calls an initial consistency detection model to detect procurement review data to output pending inconsistent items, and meanwhile, a second branch trains a reinforcement learning agent based on a labeled sample set to iteratively adjust feature weights and decision logic, solving the technical defects that detection efficiency and model optimization cannot be simultaneously performed; by converting expert review and determination results and correction opinions into feedback reward signals input into the reinforcement learning agent to update model parameters, the technical defects that the model lacks a dynamic feedback mechanism and cannot self-adaptively correct deviations are solved; by the reinforcement learning agent generating a personalized detection rule subset according to the attribute type of a procurement project, replacing general rules in the initial consistency detection model, the problems that rules have poor adaptability and cannot adapt to different types of procurement projects are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an optimization method for procurement review consistency detection based on reinforcement learning feedback mechanism. Background Technology

[0002] Procurement review is a core step in ensuring the fairness and reasonableness of procurement results and a key area for procurement compliance management. In government procurement, centralized corporate procurement, and other scenarios, the review process typically involves multiple review experts scoring suppliers' qualifications, quotations, technical solutions, and other multi-dimensional indicators based on official review rules, and issuing review opinions. However, due to various factors such as differences in the professional backgrounds of review experts, subjective judgment preferences, and misunderstandings of review standards, inconsistent review results often occur in the actual review process, including significant differences in scores from different experts for the same procurement project, contradictory review conclusions for similar projects, and inconsistent criteria for indicator judgment. These problems not only slow down the procurement process and reduce efficiency but may also lead to supplier challenges and even compliance disputes due to unfair results, bringing unnecessary costs and risks to the procuring entity.

[0003] Currently, consistency detection in procurement reviews mainly relies on the following methods: First, pure rule matching, which relies entirely on preset official review rules and identifies review items that clearly violate the rules through logical judgment; second, a single machine learning classification model, which uses historical labeled samples to train a classifier and makes consistency judgments on new review data; third, manual review, which involves experts checking each review result afterward; and fourth, static feature comparison, which compares and analyzes surface features such as scoring results and keywords.

[0004] However, pure rule-based matching can only identify explicit violations that directly conflict with the rules, failing to handle semantically ambiguous expressions in review comments and implicit differences in scoring results, resulting in a high false negative rate. Furthermore, once the rules are fixed, they cannot adapt to the differentiated needs of different types of procurement projects. Single machine learning classification models rely on a large number of fixed labeled samples and lack dynamic feedback mechanisms. When review rules are updated or new procurement projects emerge, model performance rapidly declines, exhibiting insufficient generalization ability. Detection methods primarily based on manual review rely entirely on experts checking each item afterward, resulting in extremely low efficiency. They are also affected by expert subjective judgment and workload, making it difficult to control false negative and false negative rates, and unable to meet the needs of real-time detection for large-scale projects. Static feature comparison methods only compare surface features such as scoring results and keywords, ignoring the semantic relationships between review comments and project attributes and review rules, leading to a high false positive rate. Summary of the Invention

[0005] This invention provides an optimized method for procurement review consistency detection based on a reinforcement learning feedback mechanism, which addresses the technical shortcomings of existing procurement review consistency detection rules, such as poor adaptability, insufficient detection efficiency, and inaccurate accuracy.

[0006] On the one hand, this invention provides an optimization method for procurement review consistency detection based on a reinforcement learning feedback mechanism, comprising: Collect historical procurement review data from multiple dimensions and label sample sets of consistent and inconsistent reviews; Build an initial consistency detection model between the rule matching module and the text feature comparison module; Based on the sample set, a reinforcement learning reward function is defined with the optimization objectives of detection accuracy and false negative rate; Based on the initial consistency detection model and the reinforcement learning reward function, a two-branch parallel process is executed; wherein, the first branch calls the initial consistency detection model to detect the procurement review data and outputs pending inconsistent items; the second branch trains the reinforcement learning agent based on the sample set to iteratively adjust the feature weights and decision logic. The pending inconsistency items are sent to the review and verification port to obtain the expert's review and judgment results and correction opinions. The review and judgment results and correction opinions are converted into feedback reward signals and input into the reinforcement learning agent to update the model parameters of the reinforcement learning agent. Based on the updated reinforcement learning agent, a personalized detection rule subset is generated according to the attribute type of the procurement project. The personalized detection rule subset is synchronized to the initial consistency detection model to replace the general rules. The replaced initial consistency detection model is verified by the newly added review data. If the verification fails, the personalized detection rule subset is readjusted.

[0007] Optionally, the collection of multi-dimensional procurement review historical data and the labeling of sample sets showing consistent and inconsistent reviews include: Extract historical review data, including review comment text, scoring values, review rule documents, and project attribute data, from the procurement system database and API interface. The historical review data is processed by removing missing values ​​and standardizing text format. Through a combination of rule engine pre-screening and manual review, samples that meet the official review rule threshold and whose review comments are logically consistent are marked as consistent samples, while samples with rule conflicts or contradictory review comments are marked as inconsistent samples, thus generating a sample set with binary labels.

[0008] Optionally, the construction of the initial consistency detection model between the rule matching module and the text feature comparison module includes: The review rule document is parsed into logical judgment expressions to construct a rule matching module; A text feature comparison module is constructed by analyzing the review comments text through semantic vector extraction and similarity calculation. By integrating the rule matching module and the text feature comparison module using the sigmoid function, an initial consistency detection model is constructed that can output the consistency probability of the review results.

[0009] Optionally, the step of defining a reinforcement learning reward function based on the sample set, with the optimization objectives of detection accuracy and false negative rate, includes: The detection precision is set as the ratio of the number of correctly detected congruent samples to the total number of correctly detected and incorrectly labeled congruent samples. The false negative rate is set as the ratio of the number of inconsistent samples that were not detected to the total number of inconsistent samples that were correctly detected and those that were not detected. By assigning positive incentives to the detection accuracy and negative penalties to the false negative rate using positive weighting coefficients, a reinforcement learning reward function is obtained.

[0010] Optionally, the step of executing a two-branch parallel process based on the initial consistency detection model and the reinforcement learning reward function includes: The first branch receives procurement review data through multiple threads, calculates the inconsistency score of each review item using the initial consistency detection model, and identifies and stores review items with scores exceeding a preset threshold as pending inconsistency items. The second branch trains a reinforcement learning agent based on the Q-learning framework, encodes the detection performance index as the agent's state, and encodes the adjustment of feature weights and modification of decision logic as the agent's actions. Iteratively updates and optimizes the agent's feature weights and decision logic based on the simulated detection results.

[0011] Optionally, the step of sending the pending inconsistency items to the review and verification port to obtain the expert's review and verification results and correction opinions, and converting the review and verification results and correction opinions into feedback reward signals and inputting them into the reinforcement learning agent to update the model parameters of the reinforcement learning agent, includes: Pending inconsistencies are pushed to the review and verification port via a RESTful API interface. The results of the review, which include inconsistent confirmations, inconsistent denials, or uncertainties made by the experts, are collected. The review and judgment results are quantified into positive, negative, or zero feedback reward signals; The policy gradient algorithm is used to input the feedback reward signal into the reinforcement learning agent, triggering and completing the iterative update of the reinforcement learning agent's parameters.

[0012] Optionally, the step of generating a subset of personalized detection rules based on the updated reinforcement learning agent and the attribute type of the procurement project includes: The attribute types of procurement projects are extracted into multi-dimensional feature vectors that include budget size, procurement type, and supplier qualifications. Using the deep Q-network architecture of the reinforcement learning agent, with the multidimensional feature vector and the initial rule subset as the initial state, the policy network selects discrete rule combination actions to generate a personalized detection rule subset that is suitable for the corresponding procurement project attribute type.

[0013] Optionally, synchronizing the personalized detection rule subset to the initial consistency detection model to replace the general rules, verifying the replaced initial consistency detection model with newly added review data, and readjusting the personalized detection rule subset if the verification fails, includes: The generated subset of personalized detection rules is synchronized to the initial consistency detection model to replace the original general rules. The newly added review data is used as the verification sample to calculate the detection accuracy of the initial consistency detection model after replacement. A preset detection accuracy threshold is set. If the detection accuracy of the initial consistency detection model is lower than the preset detection accuracy threshold, the verification is deemed to have failed. The failure to pass the verification is used as a negative reward input to the reinforcement learning agent, which then drives the agent to re-optimize its strategy and adjust the subset of personalized detection rules.

[0014] Optional, also includes: Based on the validated subset of personalized detection rules, the consistency detection performance index is periodically calculated. When any of the aforementioned consistency detection performance indicators falls below a preset standard, historical data re-collection, sample set update, and retraining of the reinforcement learning agent are triggered.

[0015] Optional, also includes: Collect uncovered procurement review historical data within a preset time period, and select samples whose variance of the procurement review historical data is greater than a preset variance; Oversampling techniques are used to generate a balanced training set to update the sample set; The reinforcement learning agent is retrained based on the policy gradient algorithm, and the policy parameters are adjusted by the Adam optimizer. After training, the updated reinforcement learning agent is deployed after cross-validation.

[0016] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the procurement review consistency detection optimization method based on reinforcement learning feedback mechanism as described above.

[0017] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the procurement review consistency detection optimization method based on reinforcement learning feedback mechanism as described above.

[0018] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the procurement review consistency detection optimization method based on reinforcement learning feedback mechanism as described above.

[0019] This invention provides an optimized procurement review consistency detection method based on a reinforcement learning feedback mechanism. It constructs a dual-branch parallel execution process: the first branch calls the initial consistency detection model to detect procurement review data and output pending inconsistencies, while the second branch trains a reinforcement learning agent based on a labeled sample set to iteratively adjust feature weights and decision logic, thus addressing the technical deficiency of not being able to synchronize detection efficiency and model optimization. Furthermore, by converting expert review results and correction opinions into feedback reward signals and inputting them into the reinforcement learning agent to update model parameters, it addresses the technical deficiency of the model lacking a dynamic feedback mechanism and being unable to adaptively correct deviations. Finally, by having the reinforcement learning agent generate a personalized subset of detection rules based on the attribute type of the procurement project, replacing the general rules in the initial consistency detection model, it solves the problem of poor rule adaptability and inability to adapt to different types of procurement projects. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the optimization method for procurement review consistency detection based on reinforcement learning feedback mechanism provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Figure 1 This is a flowchart illustrating the optimization method for procurement review consistency detection based on reinforcement learning feedback mechanism provided in this embodiment of the invention.

[0024] like Figure 1 As shown in the figure, the procurement review consistency detection optimization method based on reinforcement learning feedback mechanism provided by this embodiment of the invention includes the following steps: 101. Collect historical procurement review data from multiple dimensions and mark the sample sets of consistent and inconsistent reviews.

[0025] Specifically, collect historical procurement review data from multiple dimensions and label sample sets of consistent and inconsistent reviews, including: Extract historical review data, including review comment text, scoring values, review rule documents, and project attribute data, from the procurement system database and API interface. The historical review data is processed by removing missing values ​​and standardizing text format. Through a combination of rule engine pre-screening and manual review, samples that meet the official review rule thresholds and whose review comments are logically consistent are marked as consistent samples, while samples with rule conflicts or contradictory review comments are marked as inconsistent samples, generating a sample set with binary labels.

[0026] Among them, historical review data is extracted in batches from the procurement system database and API interface. The extraction scope includes review opinion text, scoring results, official review rule documents and project attribute data. The project attributes specifically include core indicators such as procurement budget scale, procurement category type, and supplier qualification level. Data extraction from multiple sources can ensure the comprehensiveness and authenticity of historical review data, and build a high-quality data foundation for subsequent sample set construction.

[0027] The extracted raw historical review data is preprocessed by first cleaning the data to remove invalid data entries containing missing or null values, and then standardizing the format of unstructured review opinion text data to ensure the integrity of the multidimensional features of the data. This eliminates the interference of invalid data and format differences on subsequent annotation and model training, effectively improving the usability of the data.

[0028] After preprocessing, standardized data is loaded in batches into a distributed storage system using SQL queries and ETL data processing tools. This storage method provides efficient data reading and processing support for subsequent large-sample model training, significantly improving overall data processing efficiency.

[0029] After data collection and preprocessing, a semi-automatic annotation method combining rule engine pre-screening and manual review is used to construct a sample set. First, the official review rules are parsed into executable logical expressions by the rule engine. Based on the logical expressions, the scoring values ​​and review opinions are initially screened to identify candidate samples that are suspected of being consistent or inconsistent. The rule engine enables automated initial screening of massive amounts of data, which can significantly reduce the workload of manual annotation and improve annotation efficiency. Then, domain review experts manually review the candidate samples and complete the sample judgment according to the annotation standards. Samples whose scoring values ​​meet the threshold of the official review rules and whose review opinion text logic is consistent with the rule requirements are marked as consistent samples. Samples whose scoring values ​​exceed the rule threshold, whose review opinions conflict with the rule requirements, or whose scoring results and review opinions contradict each other in the same review item are marked as inconsistent samples. Through manual review by domain experts, the misjudgment problem of single machine screening can be effectively avoided, ensuring the accuracy of sample annotation.

[0030] The final dataset will have binary labels. ; in, Indicates the first Multidimensional feature vectors of each sample It is a binary label (1 for consistency, 0 for inconsistency), the total number of samples Determined by the actual scale of effective historical data collected, this sample set with multidimensional feature vectors and binary labels can provide accurate and effective training data for the subsequent training of consistency detection models and reinforcement learning agents, thereby improving the detection accuracy and generalization ability of the model from the data source.

[0031] 102. Build an initial consistency detection model between the rule matching module and the text feature comparison module.

[0032] Specifically, an initial consistency detection model is built by fusing the rule matching module and the text feature comparison module, including: The review rule document is parsed into logical judgment expressions to build a rule matching module; A text feature comparison module is constructed by analyzing review comments text through semantic vector extraction and similarity calculation. By integrating the rule matching module and the text feature comparison module using the sigmoid function, an initial consistency detection model is constructed that can output the consistency probability of the review results.

[0033] The initial consistency detection model, which integrates a rule matching module and a text feature comparison module, is built by first performing a structured parsing of the official review rule document. This parses the rule requirements for various review scores and indicator judgments into directly executable logical judgment expressions. Based on this, a rule matching module is constructed. This module can directly perform compliance quantification checks on review score values ​​and calculate the rule matching degree using an indicator function. The rule matching degree is: ; in, This is the actual score. Within the scoring range allowed by the rules, the above quantitative calculation method makes the compliance judgment of the review scores more accurate and quantifiable, effectively avoiding subjective bias in the interpretation of the rules.

[0034] Then, semantic vectors of the review comments are extracted using BERT embedding technology to obtain the semantic vectors of the actual review comments. Simultaneously, semantic vectors corresponding to the expected review comments are generated based on the official review rules. The semantic similarity score between the two is calculated using cosine similarity. In this way, a text feature comparison module is constructed. This module can capture the semantic connotation of the review comments, break through the limitations of surface keyword comparison, accurately identify the implicit logical deviations between the review comments and the rule requirements, and greatly improve the accuracy of text comparison.

[0035] Finally, the rule matching score output by the rule matching module is calculated. The semantic similarity score output by the text feature comparison module. As input to the model, the results are weighted and integrated using the sigmoid function to calculate the consistency probability of the output review results. The consistency probability is: ; in, For the sigmoid function, , For weight parameters, The bias term is used as the weighting parameter, and the weighting parameter and the bias term are optimized through logistic regression training. This is used to construct an initial consistency detection model that can output the consistency probability of the review results. This dual-module weighted integration construction method allows the model to have both the hard compliance judgment capability of rule matching and the flexible semantic analysis capability of text feature comparison. The dual modules work together to improve the robustness and generalization ability of the initial detection model, which can adapt to the consistency detection needs of different types of procurement reviews. At the same time, the quantitative output form of consistency probability also provides a clear and adjustable judgment basis for the subsequent screening of suspected inconsistent items, improving the flexibility of the model in practical applications.

[0036] 103. Based on the sample set, define a reinforcement learning reward function with the optimization objectives of detection accuracy and false negative rate.

[0037] Specifically, based on the sample set, a reinforcement learning reward function is defined with the optimization objectives of detection accuracy and false negative rate, including: The detection precision is set as the ratio of the number of correctly detected congruent samples to the total number of correctly detected and incorrectly labeled congruent samples. The false negative rate is set as the ratio of the number of inconsistent samples that were not detected to the total number of inconsistent samples that were correctly detected and those that were not detected. By assigning positive incentives to detection accuracy and negative penalties to false negative rates using positive weighting coefficients, a reinforcement learning reward function is obtained.

[0038] Specifically, based on the labeled sample set of consistent and inconsistent reviews, a reinforcement learning reward function is defined with the optimization objectives of detection accuracy and false negative rate.

[0039] First, the calculation method for detection precision is set, defining it as the ratio of the number of correctly detected consistent samples to the total number of correctly detected and incorrectly labeled consistent samples. The formula is: ; in, The true positive count represents the number of correctly detected concordant samples. The false positive count represents the number of incorrectly labeled consistent samples. The quantification formula can accurately measure the model's detection accuracy for consistent samples, effectively avoid the problem of the model misjudging consistent samples, and provide an accurate positive optimization basis for the reward function.

[0040] When determining the calculation rule for the false negative rate, the false negative rate is set as the ratio of the number of inconsistent samples that were not detected to the total number of correctly detected and undetected inconsistent samples, as shown in the formula: ; in, The false negative number represents the number of inconsistent samples that were not detected. This formula can accurately quantify the degree to which the model misses inconsistent samples, and in turn, set the bottom line for the optimization of the reward function to ensure the model's detection coverage of risky samples.

[0041] Finally, introduce , Two positive weight coefficients are used to provide positive incentives for detection accuracy and negative penalties for false negatives, respectively. These two metrics are integrated into a unified reinforcement learning reward function, with the following formula: ; in, , The reward function can be flexibly adjusted according to the actual procurement review testing needs, thereby controlling the relative importance of testing accuracy and false negative rate in the reward function. This adjustable coefficient design allows the reward function to adapt to the testing focus of different procurement scenarios. The function form, which combines positive incentives for accuracy with negative penalties for false negative rate, enhances the reliability of model detection by maximizing accuracy and avoiding compliance risks caused by review misjudgments. At the same time, it ensures the model's coverage of all inconsistent samples by minimizing the false negative rate, allowing the model to maintain optimal decision-making efficiency in dynamic procurement review testing scenarios. In addition, the reward function is directly constructed based on the detection indicators of the sample set, which is deeply integrated with the model training process. It can provide clear and quantifiable optimization objectives for the iterative optimization of reinforcement learning agents, and promote the agents to quickly converge to the optimal detection strategy.

[0042] 104. Based on the initial consistency detection model and the reinforcement learning reward function, execute a two-branch parallel process; the first branch calls the initial consistency detection model to detect the procurement review data and outputs pending inconsistent items; the second branch trains the reinforcement learning agent based on the sample set to iteratively adjust the feature weights and decision logic.

[0043] Specifically, based on the initial consistency detection model and the reinforcement learning reward function, a two-branch parallel process is executed, including: The first branch receives procurement review data through multiple threads, calculates the inconsistency score of each review item using the initial consistency detection model, and identifies and stores review items with scores exceeding a preset threshold as pending inconsistency items. The second branch trains a reinforcement learning agent based on the Q-learning framework, encodes the detection performance index as the agent's state, and encodes the adjustment of feature weights and modification of decision logic as the agent's actions. Iteratively updates and optimizes the agent's feature weights and decision logic based on the simulated detection results.

[0044] Based on the established initial consistency detection model and the defined reinforcement learning reward function, a dual-branch parallel execution architecture is built using multi-threading technology. This architecture simultaneously advances the real-time detection process in the first branch and the agent training process in the second branch. The two branches run independently and share data, ensuring real-time detection efficiency while simultaneously optimizing the model iteratively, significantly improving the overall efficiency and adaptability of the detection system. The first branch receives real-time procurement review data streams via multi-threading. This data includes structured and unstructured fields such as supplier quotations, contract terms, and review opinions. It directly calls the pre-deployed initial consistency detection model to process each review item. Feature extraction and calculation are performed using formulas. The inconsistency scores for each review item were obtained, among which, For predefined feature weights, The feature function can be directly quantified using the value benchmark to determine the differences in numerical features. Then, the inconsistency score is compared with a preset threshold. (Can be set to 0.7) for comparison, The review items are identified as pending inconsistent items and directly stored in a temporary database. The multi-threaded receiving and batch calculation method enables real-time processing of review data. The independent storage of the temporary database enables quick retrieval of pending items. At the same time, the model-based initial screening reduces the scope of manual review and significantly reduces the workload of subsequent review.

[0045] The second branch synchronization is based on the labeled sample set. The reinforcement learning agent is trained, and the agent is built on the Q-learning framework. First, the current detection performance metrics (such as accuracy) are set. Encoding as agent state The feature weights will be adjusted. Modify decision logic threshold Operations are encoded as agent actions. Then, the intelligent agent performs simulated testing on the sample set, and the results are analyzed using the formula. Calculate the instant reward In the formula This is the balance coefficient (default setting is 0.6). True positive number For false positives, The recall rate is then used, and the agent parameters are updated using an iterative optimization formula based on the Q-value. The Q-value optimization formula is as follows: ; in, The learning rate (range 0.01-0.1). The discount factor is set to 0.9. In the next state, after multiple rounds of simulation convergence, the agent directly outputs the optimized feature weights and decision logic to the initial consistency detection model of the first branch. The agent training based on the Q-learning framework allows the model to autonomously iterate and adjust according to the simulation detection results. By encoding and mapping the detection performance indicators with optimization actions, the optimization direction of the agent can be more in line with the actual detection needs. The real-time synchronization of parameters between branches allows the initial detection model of the first branch to be continuously optimized as the agent trains, thereby achieving dynamic improvement in detection accuracy and effectively reducing the false alarm rate.

[0046] 105. Send the pending inconsistency items to the review and verification port, obtain the expert's review and judgment results and correction opinions, and convert the review and judgment results and correction opinions into feedback reward signals and input them into the reinforcement learning agent to update the model parameters of the reinforcement learning agent.

[0047] Specifically, pending inconsistencies are sent to the review and verification portal to obtain expert review results and correction opinions. These results and opinions are then converted into feedback reward signals and input into the reinforcement learning agent to update its model parameters, including: Pending inconsistencies are pushed to the review and verification port via a RESTful API interface. The results of the review, which include inconsistent confirmations, inconsistent denials, or uncertainties made by the experts, are collected. The review and judgment results are quantified into positive, negative, or zero feedback reward signals; The policy gradient algorithm is used to input the feedback reward signal into the reinforcement learning agent, triggering and completing the iterative update of the reinforcement learning agent's parameters.

[0048] The process involves sending pending inconsistencies to the review and verification port and updating the parameters of the reinforcement learning agent. First, the pending inconsistencies stored in the temporary database are transmitted in JSON format to the server queue of the review and verification port via a RESTful API architecture interface. This transmission method ensures data integrity and low-latency transmission. The port's web interface will highlight the original input data, model prediction results, and key differences of the review items, providing intuitive judgment criteria for expert review and significantly improving the efficiency and accuracy of expert review.

[0049] After logging in via the web interface, experts manually judge each pending inconsistency item, directly selecting one of three review results: confirmed inconsistency, denied inconsistency, or uncertain. They also simultaneously enter their correction opinions in text form. The system captures the experts' judgment results and correction opinions in real time through an event-driven mechanism. After data verification to ensure that the results are not empty and the format is compliant, the relevant information is stored in a NoSQL database, effectively avoiding interference from invalid input to downstream model updates. At the same time, the structured storage method facilitates subsequent data retrieval and analysis.

[0050] The results of expert review and judgment are quantified to generate feedback reward signals for reinforcement learning. Specifically according to The rules assign values ​​based on inconsistencies. A positive reward signal of +1 is assigned to confirm inconsistencies, which reinforces the agent's behavior of correctly identifying inconsistent items. A negative reward signal of -1 is assigned to negate inconsistencies, which punishes the agent's incorrect prediction behavior. If the inconsistency is uncertain, a value of 0 is assigned. This binary reward signal design simplifies the calculation process while retaining the core semantic information of expert feedback, which can effectively improve the convergence efficiency of the agent.

[0051] Feedback reward signal Compared with the current state Intelligent agent actions Combined, the input is fed into the experience replay buffer of the reinforcement learning framework, and the agent model parameters are iteratively updated using the policy gradient algorithm, with the update rule following the formula. .

[0052] in, Let be the parameter vector of the agent. This is the learning rate (usually set to 0.001). For gradient operators, For the agent in state Next action Based on the probability distribution, this algorithm can maximize the expected cumulative reward. The driving parameters are adjusted in the direction of optimization. After the parameters are updated, they are directly synchronized to the model weights of the reinforcement learning agent to complete one iteration cycle. The review experience of human experts is integrated into the training of the agent in the form of reward signals. This allows the model to adaptively correct detection biases, greatly improve the generalization ability and robustness of the agent in complex procurement review data scenarios, reduce the need for subsequent manual review, and make the parameter optimization of the agent more targeted by the precise update of the policy gradient algorithm, thus driving the continuous improvement of the model's detection performance.

[0053] 106. Based on the updated reinforcement learning agent, generate a subset of personalized detection rules according to the attribute type of the procurement project, synchronize the subset of personalized detection rules to the initial consistency detection model to replace the general rules, verify the replaced initial consistency detection model with the newly added review data, and readjust the subset of personalized detection rules if the verification fails.

[0054] Specifically, based on the updated reinforcement learning agent, a subset of personalized detection rules is generated according to the attribute types of the procurement project, including: The attribute types of procurement projects are extracted into multi-dimensional feature vectors that include budget size, procurement type, and supplier qualifications. By using a deep Q-network architecture for reinforcement learning agents, with multi-dimensional feature vectors and an initial rule subset as the initial state, a policy network selects discrete rule combinations to generate a personalized detection rule subset that is suitable for the corresponding procurement project attribute type.

[0055] Specifically, based on the updated reinforcement learning agent, a subset of personalized detection rules is generated according to the attribute types of procurement projects. First, feature extraction and vectorization processing are performed on the attribute information of the procurement projects, transforming core attribute indicators such as project budget, procurement type, and supplier qualifications into multi-dimensional feature vectors. ,in, As an attribute dimension, multidimensional feature vectorization can transform unstructured project attributes into numerical features that the agent can recognize and compute. This allows the agent to accurately capture the attribute differences of different procurement projects, providing precise feature basis for personalized rule generation. Then, based on the updated reinforcement learning agent, a deep Q-network (DQN) architecture is built to process the aforementioned multidimensional feature vectors. With the initial rule subset in the current system Together as the initial state of the agent , The time step index is used, and the selection and combination operations of various rules are defined as discrete rule combination actions, and the corresponding action space is constructed. The agent uses a policy network. Autonomously select and combine discrete rules from the action space to create actions. The network parameters were optimized using a gradient descent strategy. Then, a subset of personalized consistency detection rules adapted to the attribute type of this type of procurement project is generated. The optimization of the policy network aims to minimize the loss function. For the goal.

[0056] in, For experience replay buffer, As a discount factor, As an immediate reward based on subsequent verification results, and Unlike online and target Q-networks, this deep Q-network architecture allows intelligent agents to autonomously learn optimal rule generation strategies based on project attribute characteristics, breaking through the limitations of fixed general rules. The selection method of discrete rule combination actions allows the generated rule subsets to better fit the actual detection needs of procurement projects of different categories and sizes, greatly improving the adaptability and targeting of rules. At the same time, the gradient descent parameter optimization method allows the strategy network to continuously iterate, ensuring the generation accuracy of personalized rule subsets, reducing the iteration cost of manually designing rules, and allowing detection rules to be dynamically adjusted according to changes in the attributes of procurement projects.

[0057] In addition, a subset of personalized detection rules is synchronized to the initial consistency detection model to replace the general rules. The replaced initial consistency detection model is then validated using newly added review data. If the validation fails, the subset of personalized detection rules is readjusted, including: The generated subset of personalized detection rules is synchronized to the initial consistency detection model to replace the original general rules. The newly added review data is used as the verification sample to calculate the detection accuracy of the initial consistency detection model after replacement. A preset detection accuracy threshold is set. If the detection accuracy of the initial consistency detection model is lower than the preset detection accuracy threshold, the verification is deemed unsuccessful. The failure to pass validation is used as a negative reward input to the reinforcement learning agent, driving the agent to re-optimize its strategy and adjust a subset of personalized detection rules.

[0058] Specifically, after generating a subset of personalized detection rules adapted to the attribute types of the procurement project, this subset of rules is directly synchronized to the initial consistency detection model, replacing the original general rules of the model. During the synchronization process, the corresponding model parameters are updated to ensure the accuracy of the model's detection function. The new personalized detection rule subset can be directly invoked for consistency testing. This synchronous update method allows the initial detection model to quickly adapt to the detection needs of procurement projects with different attributes, without the need to rebuild the model, greatly improving the model's adaptation efficiency; subsequently, the newly added procurement review data will be... As a verification sample, it is loaded into the initial consistency detection model after the replacement rules to conduct iterative verification, using the formula... The detection accuracy of the calculation model is as follows: To verify the total number of samples, For the true labels of the samples, For indicator functions, For a subset of personalized detection rules, This provides the input review data features. The above quantitative calculation method can measure the actual detection performance of the model after rule replacement, providing an objective and quantifiable basis for judging the verification results.

[0059] At the same time, a detection accuracy threshold is set in advance. (Usually set to 0.95), compare the actual calculated detection accuracy with this threshold. If the detection accuracy is... If the rule is not properly matched, the model after the rule replacement is directly deemed to have failed validation. This threshold determination method makes the validation results clearer and more explicit, and can quickly identify subsets of personalized rules with insufficient adaptability. Once validation is deemed to have failed, the validation failure signal is immediately treated as a negative reward. The input is fed into the reinforcement learning agent and injected into the Q-learning update process, driving the agent to re-optimize the policy network. Based on the optimized strategy, discrete rule combinations are selected again to generate a modified personalized detection rule subset. This modified rule subset is then resynchronized to the initial consistency detection model for verification until the detection accuracy reaches a preset threshold. By feeding back the verification failure signal as a negative reward to the agent, the agent can accurately identify deviations in rule generation and optimize the rule generation strategy accordingly. The closed-loop design of cyclical verification and rule adjustment ensures that the final personalized detection rule subset has a high detection accuracy, significantly improving the detection accuracy and robustness of the initial consistency detection model in different attribute procurement project scenarios, and effectively avoiding detection misjudgments and omissions caused by insufficient rule adaptability.

[0060] In some embodiments, the procurement review consistency detection optimization method based on reinforcement learning feedback mechanism provided by the present invention further includes: Based on the validated subset of personalized detection rules, the consistency detection performance index is periodically calculated. When any consistency detection performance index falls below the preset standard, historical data re-collection, sample set update, and retraining of the reinforcement learning agent are triggered.

[0061] Specifically, after the personalized detection rule subset is validated and put into actual detection application, the system automatically performs statistical work on consistency detection performance indicators on a monthly basis. Based on the full-process detection log data, it accurately extracts the full output results of intelligent detection for the month and the complete records of expert review, using a formula... Calculate the accuracy of intelligent detection, where, The number of correctly detected positive examples, The number of negative cases detected correctly. For false positives, The number is a false negative, passed Calculate the expert review correction rate, where, The number of cases that were revised after expert review This refers to the total number of cases reviewed that month, and also through... The system calculates the false negative rate. These three core indicators comprehensively characterize the detection effect from the dimensions of detection accuracy, degree of manual correction, and risk of missed detection. The quantitative calculation method makes the indicator results objective and traceable, providing accurate data support for subsequent optimization and judgment. The system pre-sets preset standards for each indicator based on historical detection benchmarks, including intelligent detection accuracy thresholds. Expert review correction rate threshold False negative rate threshold The threshold parameters are stored in a configuration file for the system to retrieve in real time. When any statistically obtained indicator meets the threshold... or or At that time, the entire process optimization operation is automatically triggered. First, it re-collects historical procurement review data from the distributed database that was not covered in the past 6 months, prioritizing high-variance samples to enhance data diversity. Then, it uses SMOTE oversampling technology to process the re-collected data, and uses the formula... Generate a balanced training set, in which, For minority class samples, For its nearest neighbor samples, A random weighting factor between 0 and 1 is used to update the sample set, solve the problems of data drift and class imbalance, and retrain the model to adapt to the latest business trends.

[0062] Subsequently, based on the updated sample set, the reinforcement learning agent was retrained using the policy gradient algorithm. The loss function is defined as θ, where θ is the policy network parameter, st is the state vector, at is the action, rt is the immediate reward, γ is the discount factor, and T is the trajectory length. After training, the model performance is verified through cross-validation. Once the verification is successful, the agent is redeployed. Monthly periodic metric statistics can promptly capture the changing trends of the model's detection performance, avoiding a continuous decline in detection effectiveness due to data drift or business changes. Automatically triggered optimization when metrics are not met can complete data replenishment, sample updates, and agent retraining without manual intervention. This reduces the cost of manual maintenance and ensures that the reinforcement learning agent always adapts to the latest procurement review data features, guaranteeing the long-term stability of the initial consistency detection model's detection performance. At the same time, high-variance sample replenishment and oversampling sample updates can improve the model's robustness to marginal cases. The policy gradient algorithm combined with the Adam optimizer's retraining method allows the agent to quickly converge to the optimal detection strategy, ultimately forming a continuous self-optimizing closed loop for procurement review consistency detection, effectively addressing dynamically changing procurement review scenarios.

[0063] In some embodiments, the procurement review consistency detection optimization method based on reinforcement learning feedback mechanism provided by the present invention further includes: Collect uncovered historical procurement review data within a preset time period, and select samples whose variance of the historical procurement review data is greater than the preset variance; Oversampling techniques are used to generate a balanced training set to update the sample set; The reinforcement learning agent is retrained based on the policy gradient algorithm, and the policy parameters are adjusted by the Adam optimizer. After training, the updated reinforcement learning agent is deployed after cross-validation.

[0064] Specifically, historical procurement review data that was not covered within a preset time period (usually the most recent 12 months) is collected. Raw data is extracted from multiple data sources such as the procurement platform database and the historical review file system through distributed crawlers. After data cleaning to remove duplicate records, outliers and data with format errors, a standardized historical dataset is obtained.

[0065] Analysis of variance (ANOVA) is used to calculate the characteristic variance of the dataset. Samples with variances greater than a preset variance threshold (dynamically set according to the business scenario, generally 1.5 times the mean characteristic variance of similar projects) are selected. These samples typically include marginal cases with complex procurement needs, special review standards, or significant differences in supplier qualifications, effectively improving the diversity and representativeness of the sample set. Subsequently, the selected sample set is subjected to class balancing using SMOTE oversampling technology. For minority class samples, the K-nearest neighbor algorithm is used to find their nearest neighbor samples until the ratio of samples in each class reaches 1:1.2, forming a balanced training set to update the original sample library.

[0066] Based on the updated balanced training set, the reinforcement learning agent is retrained using the policy gradient algorithm. The training set data is divided into training and validation sets in an 8:2 ratio. The optimization objective is the policy network parameters. The Adam optimizer (learning rate set to 0.001, β1=0.9, β2=0.999) is used to minimize the loss function. Iterative training is performed for 200 epochs, with detection accuracy calculated every 50 epochs using the validation set. Training stops when the accuracy fails to improve after three consecutive validation iterations. After training, 5-fold cross-validation is used to evaluate model performance. If the average accuracy reaches a preset standard (typically ≥0.96), the updated reinforcement learning agent is deployed to the production environment, replacing the original agent. If the standard is not met, the process returns to the sample set update step for re-optimization. This process addresses the uneven sample distribution problem through targeted data collection and oversampling. The combination of policy gradient and Adam optimizer accelerates the agent's convergence speed, and cross-validation ensures the model's generalization ability, ultimately enabling the reinforcement learning agent to continuously adapt to dynamic procurement review scenarios.

[0067] Figure 2 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention.

[0068] like Figure 2 As shown, the electronic device may include a processor 210, a communications interface 220, a memory 230, and a communication bus 240. The processor 210, communications interface 220, and memory 230 communicate with each other via the communication bus 240. The processor 210 can call logical instructions from the memory 230 to execute an optimization method for procurement review consistency detection based on a reinforcement learning feedback mechanism.

[0069] Furthermore, the logical instructions in the aforementioned memory 230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the procurement review consistency detection optimization method based on reinforcement learning feedback mechanism provided by the above methods.

[0071] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the procurement review consistency detection optimization method based on reinforcement learning feedback mechanism provided by the above methods.

[0072] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optimization method for procurement review consistency detection based on reinforcement learning feedback mechanism, characterized in that, include: Collect historical procurement review data from multiple dimensions and label sample sets of consistent and inconsistent reviews; Build an initial consistency detection model between the rule matching module and the text feature comparison module; Based on the sample set, a reinforcement learning reward function is defined with the optimization objectives of detection accuracy and false negative rate; Based on the initial consistency detection model and the reinforcement learning reward function, a two-branch parallel process is executed; wherein, the first branch calls the initial consistency detection model to detect the procurement review data and outputs pending inconsistent items; the second branch trains the reinforcement learning agent based on the sample set to iteratively adjust the feature weights and decision logic. The pending inconsistency items are sent to the review and verification port to obtain the expert's review and judgment results and correction opinions. The review and judgment results and correction opinions are converted into feedback reward signals and input into the reinforcement learning agent to update the model parameters of the reinforcement learning agent. Based on the updated reinforcement learning agent, a personalized detection rule subset is generated according to the attribute type of the procurement project. The personalized detection rule subset is synchronized to the initial consistency detection model to replace the general rules. The replaced initial consistency detection model is verified by the newly added review data. If the verification fails, the personalized detection rule subset is readjusted.

2. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 1, characterized in that, The collection of multi-dimensional procurement review historical data and the labeling of sample sets showing consistent and inconsistent reviews include: Extract historical review data, including review comment text, scoring values, review rule documents, and project attribute data, from the procurement system database and API interface. The historical review data is processed by removing missing values ​​and standardizing text format. Through a combination of rule engine pre-screening and manual review, samples that meet the official review rule threshold and whose review comments are logically consistent are marked as consistent samples, while samples with rule conflicts or contradictory review comments are marked as inconsistent samples, thus generating a sample set with binary labels.

3. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 2, characterized in that, The initial consistency detection model for building the fusion rule matching module and the text feature comparison module includes: The review rule document is parsed into logical judgment expressions to construct a rule matching module; A text feature comparison module is constructed by analyzing the review comments text through semantic vector extraction and similarity calculation. By integrating the rule matching module and the text feature comparison module using the sigmoid function, an initial consistency detection model is constructed that can output the consistency probability of the review results.

4. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 1, characterized in that, Based on the sample set, a reinforcement learning reward function is defined with the optimization objectives of detection accuracy and false negative rate, including: The detection precision is set as the ratio of the number of correctly detected congruent samples to the total number of correctly detected and incorrectly labeled congruent samples. The false negative rate is set as the ratio of the number of inconsistent samples that were not detected to the total number of inconsistent samples that were correctly detected and those that were not detected. By assigning positive incentives to the detection accuracy and negative penalties to the false negative rate using positive weighting coefficients, a reinforcement learning reward function is obtained.

5. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 1, characterized in that, The process of executing a two-branch parallel procedure based on the initial consistency detection model and the reinforcement learning reward function includes: The first branch receives procurement review data through multiple threads, calculates the inconsistency score of each review item using the initial consistency detection model, and identifies and stores review items with scores exceeding a preset threshold as pending inconsistency items. The second branch trains a reinforcement learning agent based on the Q-learning framework, encodes the detection performance index as the agent's state, and encodes the adjustment of feature weights and modification of decision logic as the agent's actions. Iteratively updates and optimizes the agent's feature weights and decision logic based on the simulated detection results.

6. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 1, characterized in that, The step of sending the pending inconsistency entries to the review and verification port, obtaining the expert's review and verification results and correction opinions, and converting the review and verification results and correction opinions into feedback reward signals and inputting them into the reinforcement learning agent to update the model parameters of the reinforcement learning agent includes: Pending inconsistencies are pushed to the review and verification port via a RESTful API interface. The results of the review, which include inconsistent confirmations, inconsistent denials, or uncertainties made by the experts, are collected. The review and judgment results are quantified into positive, negative, or zero feedback reward signals; The policy gradient algorithm is used to input the feedback reward signal into the reinforcement learning agent, triggering and completing the iterative update of the reinforcement learning agent's parameters.

7. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 1, characterized in that, The updated reinforcement learning agent generates a subset of personalized detection rules based on the attribute types of the procurement projects, including: The attribute types of procurement projects are extracted into multi-dimensional feature vectors that include budget size, procurement type, and supplier qualifications. Using the deep Q-network architecture of the reinforcement learning agent, with the multi-dimensional feature vector and the initial rule subset as the initial state, the policy network selects discrete rule combination actions to generate a personalized detection rule subset that is suitable for the corresponding procurement project attribute type.

8. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 1, characterized in that, The process of synchronizing the personalized detection rule subset to the initial consistency detection model to replace the general rules, verifying the replaced initial consistency detection model with newly added review data, and readjusting the personalized detection rule subset if the verification fails includes: The generated subset of personalized detection rules is synchronized to the initial consistency detection model to replace the original general rules. The newly added review data is used as the verification sample to calculate the detection accuracy of the initial consistency detection model after replacement. A preset detection accuracy threshold is set. If the detection accuracy of the initial consistency detection model is lower than the preset detection accuracy threshold, the verification is deemed to have failed. The failure to pass the verification is used as a negative reward input to the reinforcement learning agent, which then drives the agent to re-optimize the strategy and adjust the personalized detection rule subset.

9. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 1, characterized in that, Also includes: Based on the validated subset of personalized detection rules, the consistency detection performance index is periodically calculated. When any of the aforementioned consistency detection performance indicators falls below a preset standard, historical data re-collection, sample set update, and retraining of the reinforcement learning agent are triggered.

10. The procurement review consistency detection optimization method based on reinforcement learning feedback mechanism according to claim 1, characterized in that, Also includes: Collect uncovered historical procurement review data within a preset time period, and select samples whose variance of the historical procurement review data is greater than a preset variance; Oversampling techniques are used to generate a balanced training set to update the sample set; The reinforcement learning agent is retrained based on the policy gradient algorithm, and the policy parameters are adjusted by the Adam optimizer. After training, the updated reinforcement learning agent is deployed after cross-validation.

Images