Purchasing document checking method and device based on multi-modal and rule optimization

Abstract

The application discloses a kind of based on multimodal and rule optimization's procurement document checking method and device, comprising: obtaining electric power procurement to be checked file;Extract the image feature and non-image feature in electric power procurement to be checked file and carry out alignment;Non-image feature is semantically encoded, and procurement semantic feature is obtained;By semantic and visual collaborative feature extraction strategy, procurement semantic feature and image feature are spliced, and procurement vector is obtained;Based on the procurement audit decision model constructed in advance, in combination with differentiable reward function, the procurement vector is analyzed and judged, and the procurement document checking result is given, and the electric power procurement to be checked file is labeled, and procurement document checking report is generated. Through the extraction alignment multimodal feature, the efficient collaborative processing of multimodal data in procurement document checking is realized, the semantic and visual correlation of procurement document is captured using the semantic and visual collaborative feature extraction strategy, and the accuracy and robustness of the audit are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent review, and particularly relates to a procurement document checking method and device based on multi-modal and rule optimization. BACKGROUND

[0002] Electric power procurement plan document review is that electric power material procurement plan experts review the procurement plan technical specification document before the procurement project bidding according to the issued procurement plan review points, and the procurement plan personnel correct the technical specification document according to the review results of the experts.

[0003] With the expansion of the scale of electric power material procurement and the improvement of equipment complexity, the procurement plan audit faces the dual challenges of multi-modal data integration and dynamic rule adaptation. The traditional audit system mainly depends on the shallow analysis of single modal data, and it is difficult to fully capture the key features in the electrical diagram and wiring diagram. At the same time, the rules in the field of equipment procurement are complex and updated frequently, and the hard constraints and implicit business logic are intertwined, which leads to low efficiency and errors in manual audit.

[0004] Patent application CN112446649A discloses a kind of material procurement plan's review method and device, comprising: obtaining the material procurement plan to be reviewed, wherein the material procurement plan to be reviewed at least includes: the description information of the material to be purchased, procurement mode;According to the corresponding relationship between the pre-established description information of material and review points, the target review point corresponding to the description information of the material to be purchased is determined;According to the pre-set review rule corresponding to target review point, the material procurement plan to be reviewed is reviewed, and the review result is obtained, wherein the review result is used to indicate whether the material procurement plan to be reviewed is qualified, and / or mark the error information existing in the material procurement plan to be reviewed;The review result of the material procurement plan to be reviewed is output, solves the problem that the material procurement plan review efficiency and accuracy are low.

[0005] The above related technology only compares and reviews the description information of the material and the review points, without considering other information in the material procurement plan, and does not comprehensively review the points in the material procurement plan. How to process and audit the multi-modal information in the material procurement plan to improve the review accuracy and efficiency of the material procurement plan is a problem to be solved at present. SUMMARY

[0006] In view of the defects in the prior art, the application provides a procurement document verification method and device based on multi-modal and rule optimization, which comprises the following steps: obtaining a power procurement document to be verified; extracting image features and non-image features in the power procurement document to be verified to form multi-modal features, and aligning the multi-modal features; performing semantic coding on the non-image features to obtain procurement semantic features; splicing the procurement semantic features and the image features by a semantic and visual collaborative feature extraction strategy to obtain a procurement vector; analyzing and judging the procurement vector based on a pre-constructed procurement review decision model combined with a differentiable reward function to give a procurement document verification result, wherein the differentiable reward function is obtained by compiling hard constraints of procurement review; and labeling the power procurement document to be verified according to the procurement document verification result and generating a procurement document verification report.

[0007] By extracting multi-modal features in the power procurement document to be verified and aligning the multi-modal features, efficient collaborative processing of multi-modal data in procurement document verification is realized; by splicing the procurement semantic features and the image features by the semantic and visual collaborative feature extraction strategy to obtain the procurement vector, the limitation of single modal data in traditional review is broken, the semantic and visual correlation of the procurement document can be fully captured, and the accuracy and robustness of the review are improved; the procurement vector is analyzed and judged by the procurement review decision model, the procurement document verification result is given, the power procurement document to be verified is labeled, and the procurement document verification report is generated, which is convenient for subsequent tracing and rectification, and improves the transparency and operability of the review work.

[0008] In a first aspect, the application provides a procurement document verification method based on multi-modal and rule optimization, which specifically comprises the following steps:

[0009] Obtaining a power procurement document to be verified;

[0010] Extracting image features and non-image features in the power procurement document to be verified to form multi-modal features, and aligning the multi-modal features;

[0011] Performing semantic coding on the non-image features to obtain procurement semantic features;

[0012] Splicing the procurement semantic features and the image features by a semantic and visual collaborative feature extraction strategy to obtain a procurement vector;

[0013] Analyzing and judging the procurement vector based on a pre-constructed procurement review decision model combined with a differentiable reward function to give a procurement document verification result, wherein the differentiable reward function is obtained by compiling hard constraints of procurement review;

[0014] Labeling the power procurement document to be verified according to the procurement document verification result and generating a procurement document verification report.

[0015] Further, the image feature is obtained by the following steps:

[0016] An original procurement image matrix in the power procurement to-be-verified file is acquired;

[0017] According to a visual attention allocation function, combined with visual learning parameters, a procurement key area is extracted from the original procurement image matrix to obtain a procurement visual vector;

[0018] According to a text semantic mapping function, combined with text learning parameters, a procurement image semantic vector related to the original procurement image matrix is extracted from a pre-constructed procurement corpus;

[0019] Based on a vector conversion function, the procurement visual vector and the procurement image semantic vector are weightedly fused to obtain an image feature.

[0020] Further, the image feature is specifically represented as:

[0021] .

[0022] Wherein, V caption is a fixed-dimension image feature used to describe the main content of the image in the power procurement to-be-verified file; F image is an original procurement image matrix containing pixel-level visual information; Ψ(·,Θ Ψ ) is a visual attention allocation function, Θ Ψ is a visual learning parameter used to filter a procurement key area from the original procurement image matrix; C text is a pre-constructed procurement corpus; Ω(·,Θ ω ) is a text semantic mapping function, Θ ω is a text learning parameter used to generate a procurement image semantic vector related to the original procurement image matrix from the procurement corpus; ⊙ is a weighted fusion operation of the procurement visual vector and the procurement semantic vector; Φ(·) is a vector conversion function mapping the fused vector into a fixed-dimension image feature.

[0023] Further, through a semantic and visual collaborative feature extraction strategy, the procurement semantic feature and the image feature are spliced to obtain a procurement vector, specifically including:

[0024] The procurement semantic feature and the image feature are respectively preprocessed to obtain a standard procurement semantic feature and a standard image feature;

[0025] The similarity of each standard procurement semantic feature and standard image feature is analyzed to give a first procurement correlation degree;

[0026] Based on the first procurement correlation degree, a procurement feature pair of the standard procurement semantic feature and the standard image feature is constructed, and cross-modal learning is performed on the standard procurement semantic feature and the standard image feature in the procurement feature pair to adjust the standard procurement semantic feature and the standard image feature, so as to obtain semantic learning features and image learning features;

[0027] The semantic learning features and the image learning features are subjected to dimension consistency processing and splicing to obtain a procurement vector.

[0028] Further, the first procurement correlation degree is specifically represented as:

[0029] ;

[0030] Wherein, Sim(F text , F image ) is the similarity of the standard procurement semantic feature and the standard image feature, (F text , F image ) is the procurement feature pair, F text is the standard procurement semantic feature, F image is the standard image feature, ||F text || is the modulus of the standard procurement semantic feature, and ||F image || is the modulus of the standard image feature.

[0031] Further, based on the pre-constructed procurement review decision model, the procurement vector is analyzed and judged in combination with the differentiable reward function to give a procurement file verification result, specifically including:

[0032] Deep semantic understanding and feature conversion are performed on the procurement vector to obtain a model procurement vector;

[0033] Based on the differentiable reward function, the degree of fit of each hard constraint in the procurement review and the model procurement vector is analyzed and fused to obtain a procurement reward value corresponding to the model procurement vector;

[0034] Different values of the procurement file verification result are searched, and the procurement reward value corresponding to the model procurement vector is combined to obtain a selection probability and an evaluation result corresponding to the different values of the procurement file verification result.

[0035] Further, the differentiable reward function is obtained through the following steps:

[0036] Based on the matching degree of the hard constraint of the procurement review and the procurement vector, a matching degree function is constructed;

[0037] Based on the semantic similarity of the hard constraint of the procurement review and the procurement vector, a constraint correlation function is constructed;

[0038] The contribution proportions of the balance matching degree function and the constraint association function are fused in combination with a preset balance coefficient to give a fitting function of each hard constraint.

[0039] The fitting functions of each hard constraint are weighted and summed based on a preset constraint coefficient to obtain a differentiable reward function.

[0040] Further, the differentiable reward function is specifically represented as:

[0041] ;

[0042] wherein Rdiff(R i ,S) is a reward value of the differentiable reward function corresponding to the i-th hard constraint R i and the procurement vector S, σ(·) is a Sigmoid activation function for mapping the result to the range of [0, 1], λ i is a constraint coefficient of the i-th hard constraint, δ(R i ,S) is a matching degree function, φ(R i ,S) is a constraint association function, γ is a balance coefficient, γ ∈ (0, 1), and n is the total number of hard constraints.

[0043] Further, the procurement review decision model comprises a multi-layer self-attention mechanism, a feedforward neural network, and a multi-layer fully connected network comprising a procurement output network and a procurement evaluation network.

[0044] The feedforward neural network is connected with the multi-layer self-attention mechanism and receives a semantic understanding vector output by the multi-layer self-attention mechanism for feature conversion.

[0045] The multi-layer fully connected network is connected with the feedforward neural network and outputs different values of the procurement file verification result and probability distributions corresponding to the different values.

[0046] The procurement output network comprises a first input layer, a first hidden layer, and a first output layer. The first input layer receives a feature conversion vector output by the feedforward neural network and transmits the feature conversion vector to the first output layer through the first hidden layer to obtain different values of the intermediate review result and selection probabilities corresponding to the different values.

[0047] The procurement evaluation network comprises a second input layer, a second hidden layer, and a second output layer. The second input layer receives different values of the intermediate review result and selection probabilities corresponding to the different values and transmits the different values and the selection probabilities to the second output layer through the second hidden layer to adjust the selection probabilities of the different values of the intermediate review result and feed back the adjusted selection probabilities to the first hidden layer.

[0048] Further, the network parameters of the multi-layer fully connected network are determined by the following steps:

[0049] all parameters of the multi-layer self-attention mechanism and the feedforward neural network are set to a frozen state;

[0050] define different values of an output end of the multi-layer fully connected network connected with the feedforward neural network and an output end of the multi-layer fully connected network outputting the procurement document verification result and a probability distribution corresponding to the different values;

[0051] convert the feature conversion vector output by the feedforward neural network into input data, and use a historical audit result label set as supervision data, wherein the historical audit result label set includes an audit label for historical power procurement document auditing;

[0052] use a small batch gradient descent method to adjust the network parameters of the multi-layer fully connected network, and calculate the cross-entropy loss between the probability distribution output by the multi-layer fully connected network and the historical audit result label set;

[0053] update the network parameters of the multi-layer fully connected network based on the cross-entropy loss through back propagation, and repeat iteration until the cross-entropy loss converges.

[0054] Further, the first hidden layer includes a shared encoder, and the first output layer includes a main classification head, an auxiliary regression head, and an auxiliary binary classification head;

[0055] The shared encoder captures core features related to the intermediate audit result in the feature conversion vector input by the feedforward neural network;

[0056] The main classification head is configured to give the intermediate audit result according to the core features output by the shared encoder;

[0057] The auxiliary regression head is configured to analyze the deviation range of the central audit result based on the core features, give a deviation result, and feed back to the second input layer;

[0058] The auxiliary binary classification head is configured to analyze the risk of bid rigging of the central audit result based on the core features, give a risk result, and feed back to the second input layer.

[0059] In a second aspect, the present application also provides a procurement document verification device based on multi-modal and rule optimization, which adopts the procurement document verification method based on multi-modal and rule optimization as described above, and comprises:

[0060] A data acquisition module is configured to acquire a power procurement document to be verified;

[0061] A feature extraction module is configured to extract image features and non-image features in the power procurement document to be verified, form multi-modal features, and align the multi-modal features;

[0062] A feature encoding module is configured to perform semantic encoding on the non-image features to obtain procurement semantic features;

[0063] a feature fusion module configured to splice the procurement semantic features and the image features by a semantic and visual collaborative feature extraction strategy to obtain a procurement vector;

[0064] a feature analysis module configured to analyze and judge the procurement vector based on a pre-constructed procurement review decision model in combination with a differentiable reward function to give a procurement file verification result, wherein the differentiable reward function is obtained by compiling hard constraints of the procurement review;

[0065] a report output module configured to label the power procurement file to be verified according to the procurement file verification result and generate a procurement file verification report.

[0066] The procurement file verification method and device based on multi-modal and rule optimization provided by the application have at least the following beneficial effects:

[0067] (1) The multi-modal features in the power procurement file to be verified are extracted and aligned to realize efficient collaborative processing of multi-modal data in the procurement file verification. The procurement semantic features and the image features are spliced by the semantic and visual collaborative feature extraction strategy to obtain the procurement vector, which breaks the limitation of single modal data in the traditional review and can fully capture the semantic and visual correlation of the procurement file, thereby improving the accuracy and robustness of the review. The procurement review decision model is used to analyze and judge the procurement vector to give the procurement file verification result, label the power procurement file to be verified, generate the procurement file verification report, facilitate subsequent tracing and rectification, and improve the transparency and operability of the review work.

[0068] (2) The hard constraints of the procurement review are compiled into the differentiable reward function, so that the procurement review decision model can dynamically adjust the strategy and select different values of the procurement file verification result through the search strategy, thereby realizing the intelligentization of the review process and improving the review efficiency.

[0069] (3) The non-image features are semantically encoded to convert the text information into a feature vector containing semantics and convert the high-dimensional data into a low-dimensional vector for easy processing and analysis. Meanwhile, the low-dimensional vector can reduce the consumption of computing resources, improve the processing and analysis efficiency, and uniformly represent the data of different modalities (such as text and table) as a vector through semantic encoding for easy fusion processing. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The flowchart of the procurement file verification method based on multi-modal and rule optimization provided by the embodiment of the application;

[0071] Figure 2 The flowchart of determining the image features provided by the embodiment of the application;

[0072] Figure 3 The flowchart for obtaining the procurement vector provided in the embodiment of the present application;

[0073] Figure 4 The flowchart for giving the procurement file checking result provided in the embodiment of the present application;

[0074] Figure 5 The flowchart for constructing the differentiable reward function provided in the embodiment of the present application;

[0075] Figure 6 The architecture diagram of the procurement review decision model provided in the embodiment of the present application;

[0076] Figure 7 The structural block diagram of the procurement file checking device based on multi-modal and rule optimization provided in the embodiment of the present application.

[0077] Among them, 201, data acquisition module; 202, feature extraction module; 203, feature coding module; 204, feature fusion module; 205, feature analysis module; 206, report output module. DETAILED DESCRIPTION

[0078] In order to better understand the above technical solutions, the above technical solutions will be described in detail in conjunction with the description of the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0079] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Multiple" generally includes at least two.

[0080] It should also be noted that the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the goods or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or include the elements inherent in such goods or devices. Without more limitation, the element defined by the sentence "including a" does not exclude the presence of other identical elements in the goods or devices including the element.

[0081] Traditional procurement document verification technology has many defects. One is that the multi-modal data processing capability is insufficient. The traditional system can only process structured text or table data, and the analysis of electrical diagrams and wiring diagrams relies on manual annotation, resulting in incomplete and low-efficiency information extraction. Two is that the rule adaptation is rigid. The hard constraints are realized by a static rule engine, which cannot dynamically adjust the priority or handle implicit business logic, and frequent manual intervention is required for rule library updating. Three is that the model iteration is lagging. The existing audit model is trained based on a closed data set, which is difficult to incorporate new rules or device types, resulting in poor adaptability to new business scenarios. In addition, the traditional system lacks explainability, and the returned decision only provides a simple result, which cannot locate the specific field or graphic element level error, increasing the review cost.

[0082] Although the related art attempts to introduce a machine learning model, the problems of multi-modal data alignment difficulty and rule expression rigidity still restrict the level of audit intelligence. The present application provides a procurement document verification method and device based on multi-modal and rule optimization. The method comprises the following steps: obtaining an electric power procurement document to be verified; extracting image features and non-image features in the electric power procurement document to be verified to form multi-modal features, and aligning the multi-modal features; performing semantic encoding on the non-image features to obtain procurement semantic features; using a semantic and visual collaborative feature extraction strategy to splice the procurement semantic features and the image features to obtain a procurement vector; based on a pre-constructed procurement audit decision model, combining a differentiable reward function, analyzing and judging the procurement vector, and giving a procurement document verification result, wherein the differentiable reward function is obtained by compiling hard constraints of procurement audit; and based on the procurement document verification result, labeling the electric power procurement document to be verified and generating a procurement document verification report.

[0083] By extracting multi-modal features in the electric power procurement document to be verified and aligning the multi-modal features, efficient collaborative processing of multi-modal data in procurement document verification is realized. By using a semantic and visual collaborative feature extraction strategy to splice the procurement semantic features and the image features to obtain a procurement vector, the limitations of single modal data in traditional audits are broken, the semantic and visual correlation of the procurement document can be fully captured, and the accuracy and robustness of the audit are improved. By using a procurement audit decision model to analyze and judge the procurement vector, a procurement document verification result is given, the electric power procurement document to be verified is labeled, and a procurement document verification report is generated, which facilitates subsequent tracing and rectification, and improves the transparency and operability of the audit work.

[0084] As shown in Figure 1 The present application provides a procurement document verification method based on multi-modal and rule optimization, which specifically comprises the following steps:

[0085] S101: Obtain an electric power procurement document to be verified.

[0086] Specifically, the power procurement to be verified file is a power equipment procurement plan file submitted by the procurement department, including a power equipment procurement plan table and relevant drawings of each power equipment, such as an electrical wiring diagram. The plan table includes device model, device quantity, device technical parameters and other information.

[0087] S102: Extract image features and non-image features in the power procurement to be verified file, form multi-modal features, and align the multi-modal features.

[0088] Wherein, the multi-modal features include image features and non-image features. Further, referring to Figure 2 , the image features in the power procurement to be verified file are extracted by the following steps:

[0089] Obtain the original procurement image matrix in the power procurement to be verified file;

[0090] According to the visual attention allocation function, combined with the visual learning parameter, the procurement key area is extracted from the original procurement image matrix to obtain the procurement visual vector;

[0091] According to the text semantic mapping function, combined with the text learning parameter, the procurement image semantic vector related to the original procurement image matrix is extracted from the pre-constructed procurement corpus;

[0092] Based on the vector conversion function, the procurement visual vector and the procurement image semantic vector are weighted and fused to obtain the image features.

[0093] In a specific embodiment, first, the key information of the text, table and related drawings including the electrical wiring diagram is extracted from the power procurement to be verified file to generate structured records, and the originally scattered and unstructured information is converted into regular structured data to lay the foundation for subsequent processing. Among them, by extracting the text and table from the power procurement to be verified file for structured processing, the non-image features are obtained. For image features, the image description generation algorithm is used to process the related drawings to generate a vector that can describe the main content of the corresponding drawing, i.e. image features. For example, the image description generation algorithm is used to process the electrical wiring diagram to obtain a vector that describes the main content of the electrical wiring diagram, i.e. the corresponding image features. The image features convert the complex image information in the related drawings into understandable vector form, solving the problem that image information is difficult to directly participate in analysis and calculation. Finally, the text, table and image features are aligned to ensure the accuracy of the correlation between different types of data, avoid the review deviation caused by data correlation errors, and ensure the consistency of multi-modal data in subsequent processing.

[0094] Further, the image features are specifically represented as:

[0095] ;

[0096] wherein V caption is a fixed-dimension image feature used to describe the main content of the image in the power procurement to-be-verified file; F image is an original procurement image matrix containing pixel-level visual information; Ψ(·, Θ Ψ ) is a visual attention allocation function, Θ Ψ is a visual learning parameter used to filter the procurement key area from the original procurement image matrix; C text is a pre-constructed procurement corpus; Ω(·, Θ ω ) is a text semantic mapping function, Θ ω is a text learning parameter used to generate a procurement image semantic vector related to the original procurement image matrix from the procurement corpus; is a weighted fusion operation of the procurement visual vector and the procurement semantic vector; Φ(·) is a vector conversion function that maps the fused vector into a fixed-dimension image feature.

[0097] In one specific example, the relevant drawing is an electrical wiring diagram, and the electrical wiring diagram is substituted into the calculation formula of the image description generation algorithm, i.e., the electrical wiring diagram is the original procurement image matrix F image . The electrical wiring diagram is substituted into the visual attention allocation function Ψ(·, Θ Ψ ), and the procurement key area is extracted according to the adjustment of the visual learning parameter to obtain a procurement visual vector. At the same time, the electrical wiring diagram is substituted into the text semantic mapping function Ω(·, Θ ω ), and a procurement image semantic vector related to the original procurement image matrix is extracted from the pre-constructed procurement corpus according to the adjustment of the text learning parameter. The visual learning parameter and the text learning parameter can be defined according to requirements or experience, or can be determined through model training, and no limitation is made to this. Finally, the obtained procurement visual vector and procurement image semantic vector are substituted into the vector conversion function Φ(·), and are fused by weighting to obtain an image feature.

[0098] S103: The non-image features are semantically encoded to obtain procurement semantic features.

[0099] In one specific embodiment, the non-image features are data cleaned to remove noise data, duplicate data and invalid data in the structured data. For example, empty values are deleted, and incorrect formats are corrected. After data cleaning, the structured data is converted into a unified format, and data standardization is completed.

[0100] After standardization, structured data containing text is processed for word segmentation, part-of-speech tagging, named entity recognition, and other processes. Then, the context information in the structured data is analyzed to understand the relationships between the data. For example, in a procurement plan table, the relationship between "product name" and "purchase quantity" is understood. The structured data is then encoded according to actual needs, and the encoding method is selected, such as using word embedding for semantic encoding: mapping words or phrases in text data to high-dimensional vector space. For example, using Word2Vec or GloVe to map "transformer" to a vector. Sentence embedding can also be used for semantic encoding: mapping sentences to vectors. For example, using BERT or Sentence-BERT to map "the device must have overload protection function and can run for 1 hour at 1.2 times rated current" to a vector. According to the needs, the encoded vectors are spliced or converted to obtain procurement semantic features.

[0101] In a specific example, the power equipment procurement plan file includes the sentence "Need to purchase high-voltage switchgear with model ABB-1234, rated voltage 12kV, rated current 1250A." The sentence is decomposed into words or phrases such as "need to purchase", "model", "ABB-1234", "high-voltage switchgear", "rated voltage", "12kV", "rated current", "1250A". Key information such as "ABB-1234", "12kV", "1250A" is extracted. Use word embedding technology (such as Word2Vec or BERT) to convert each word or phrase into a vector and combine it into a comprehensive vector.

[0102] For example, the power equipment procurement plan file includes the sentence "The device must have overload protection function and can run for 1 hour at 1.2 times rated current." The sentence is decomposed into words or phrases such as "device", "must have", "overload protection function", "can run for", "1.2 times rated current", "for 1 hour". Extract key information such as "overload protection function", "1.2 times rated current", "1 hour". Use sentence embedding technology (such as Sentence-BERT) to convert the entire sentence into a vector.

[0103] By performing semantic encoding on non-image features, text information is converted into feature vectors containing semantics, high-dimensional data is converted into low-dimensional vectors, facilitating processing and analysis. At the same time, low-dimensional vectors can reduce the consumption of computing resources and improve processing and analysis efficiency. And through semantic encoding, data of different modalities (such as text, tables) are uniformly represented as vectors, facilitating fusion processing.

[0104] S104: Concatenate the procurement semantic features and image features through the semantic and visual collaborative feature extraction strategy to obtain a procurement vector.

[0105] The semantic and visual collaborative feature extraction strategy is a method for comprehensively processing multi-modal data by fusing semantic information (such as text description, structured data, etc.) and visual information (such as images, videos, etc.) to extract features with better representation and semantic consistency, thereby providing more effective input for the downstream procurement vector review task.

[0106] The semantic and visual collaborative feature extraction strategy is a feature extraction method that collaboratively processes semantic features and visual features to mine the internal correlation and complementarity between them, thereby generating a fused feature vector that can reflect both semantic information and visual information. The core is to align semantic features and visual features in the same feature space through cross-modal learning, and to improve the representation ability and semantic consistency of the features through collaborative optimization.

[0107] The semantic and visual collaborative feature extraction strategy can effectively mine the complementarity of semantic and visual information, providing stronger feature support for the procurement vector review task.

[0108] Further, with reference to Figure 3 , the procurement semantic features and image features are concatenated through the semantic and visual collaborative feature extraction strategy to obtain a procurement vector, specifically including:

[0109] The procurement semantic features and image features are preprocessed respectively to obtain standard procurement semantic features and standard image features;

[0110] The similarity of each standard procurement semantic feature and standard image feature is analyzed to give a first procurement correlation degree;

[0111] Based on the first procurement correlation degree, a procurement feature pair of standard procurement semantic features and standard image features is constructed, and cross-modal learning is performed on the standard procurement semantic features and standard image features in the procurement feature pair to adjust the standard procurement semantic features and standard image features, obtaining semantic learning features and image learning features;

[0112] The semantic learning features and image learning features are processed for dimension consistency and concatenated to obtain a procurement vector.

[0113] Further, the first procurement correlation degree is specifically expressed as:

[0114] ;

[0115] wherein Sim(F text , F image ) is the similarity of the standard procurement semantic features and the standard image features, (Ftext , F image ) is a standard purchase semantic feature, F text is a standard purchase semantic feature, F image is a standard image feature, ||F text is a standard purchase semantic feature, ||F image is a standard image feature, ||F

[0116] In a specific embodiment, the purchase semantic features are preprocessed, including removing redundant information, standardizing the format, to obtain standard purchase semantic features, and at the same time, the image features are preprocessed, including adjusting the size, normalizing the operation, to obtain standard image features. Then the similarity between the standard purchase semantic features and the standard image features is calculated and analyzed to measure the semantic correlation degree, and the first purchase correlation degree is given. Based on the first purchase correlation degree, the purchase feature pair of the standard purchase semantic feature and the standard image feature is constructed, for example, taking the standard purchase semantic feature A as an example, the first purchase correlation degree with each standard image feature is a1, a2, a3, …, an, the first purchase correlation degree is sorted in descending order, and it is judged whether the maximum value in the first purchase correlation degree reaches the correlation threshold value, if it reaches, the purchase feature pair is established, otherwise, the purchase feature pair is not established. The standard purchase semantic features and the standard image features in the purchase feature pair are subjected to cross-modal learning, and the standard purchase semantic features and the standard image features are adjusted to obtain semantic learning features and image learning features. Based on the construction result of the purchase feature pair, cross-modal contrast learning is carried out, so that the standard purchase semantic features and the standard image features related to semantics are more close in the feature space. The semantic learning features and the image learning features after cross-modal contrast learning are subjected to dimension consistency processing; the processed semantic learning features and image learning features are spliced to form a purchase vector.

[0117] By using the semantic and visual collaborative feature extraction strategy, cross-modal contrast learning is carried out on the standard purchase semantic features and the standard image features, so that the standard purchase semantic features and the standard image features related to semantics are closer in the feature space, which enhances the association between text and image, and can more comprehensively understand the overall content of the power equipment procurement plan document. Finally, the processed features are spliced into a purchase vector, which integrates multi-modal information and provides comprehensive and comprehensive feature representation for the subsequent audit decision module, reducing the limitations of single modal information.

[0118] S105: Based on the pre-constructed purchase audit decision model, the purchase vector is analyzed and judged in combination with the differentiable reward function, and the purchase file verification result is given.

[0119] The differentiable reward function is compiled through hard constraints on procurement review, which are internal constraints on the procurement of power equipment, including equipment specifications and procurement budget limits.

[0120] Furthermore, referring to Figure 4 Based on a pre-built procurement review and decision-making model, combined with a differentiable reward function, the procurement vector is analyzed and judged, and the procurement document verification results are given, including:

[0121] Deep semantic understanding and feature transformation are performed on the procurement vector to obtain the model procurement vector;

[0122] Based on the differentiable reward function, we analyze and integrate the fit between various hard constraints in the procurement review and the model procurement vector to obtain the procurement reward value corresponding to the model procurement vector.

[0123] By searching the different values ​​of the procurement document verification results corresponding to the procurement vector in the model, and combining them with the procurement reward value corresponding to the procurement vector in the model, we can obtain the selection probability and evaluation result corresponding to the different values ​​of the procurement document verification results.

[0124] Furthermore, referring to Figure 5 The differentiable reward function is obtained through the following steps:

[0125] Based on the matching degree between the hard constraints of procurement review and the procurement vector, a matching degree function is constructed;

[0126] Based on the hard constraints of procurement review and the semantic similarity of procurement vectors, a constraint association function is constructed.

[0127] By combining the preset balance coefficients, the contribution ratios of the balance matching function and the constraint association function, and integrating them, the fit function for each hard constraint is given.

[0128] Based on preset constraint coefficients, the fit functions of each hard constraint are weighted, summed, and mapped to obtain a differentiable reward function.

[0129] Furthermore, the differentiable reward function is specifically expressed as:

[0130] ;

[0131] Among them, Rdiff(R i S) represents the i-th hard constraint R. i The reward value corresponding to the differentiable reward function for the purchase vector S, σ(·) is the Sigmoid activation function used to map the result to the range [0,1], and λ i Let δ(R) be the constraint coefficient of the i-th hard constraint. i S) is the matching degree function, φ(R)i S) is a constraint association function, γ is a balance coefficient, γ ∈ (0, 1), and n is the total number of hard constraints.

[0132] In a specific embodiment, the hard constraints of the procurement review are compiled to obtain a differentiable reward function. Wherein, Rdiff(R i S) ∈ [0, 1], that is, the reward value is mapped to 0 to 1, λ i The priority for embodying the hard constraint is determined through historical power procurement file review data statistical analysis. When S completely matches R i , the matching degree function δ(R i S) outputs 1, and outputs -1 when it is completely inconsistent, and outputs a continuous value between -1 and 1 when it is partially consistent. The semantic similarity between S and R i outputs a continuous value between 0 and 1, the balance coefficient is set according to the actual business demand and rule characteristics, and is used to adjust the contribution proportion of the matching degree function and the constraint association function, and is used to adjust the contribution proportion of the rule matching degree and the association degree. σ(·) is a Sigmoid activation function, which maps the final reward value to 0 to 1, forms a differentiable reward value, i is the serial number of the hard constraint of the procurement review, and n is the total number of the hard constraint of the procurement review.

[0133] By constructing a differentiable reward function, the rigid hard constraint can be involved in the analysis and judgment of the procurement vector in a calculable and optimized form, so that the procurement file verification result is more in line with the actual business rules.

[0134] Further, with reference to Figure 6 , the procurement review decision model includes a multi-layer self-attention mechanism, a feedforward neural network, and a multi-layer fully connected network including a procurement output network and a procurement evaluation network;

[0135] The feedforward neural network is connected with the multi-layer self-attention mechanism and receives the semantic understanding vector output by the multi-layer self-attention mechanism for feature conversion;

[0136] The multi-layer fully connected network is connected with the feedforward neural network, and outputs different values of the procurement file verification result and probability distributions corresponding to the different values;

[0137] The procurement output network includes a first input layer, a first hidden layer and a first output layer, the first input layer receives the feature conversion vector output by the feedforward neural network, and transmits the feature conversion vector to the first output layer through the first hidden layer to obtain different values of the intermediate review result and selection probabilities corresponding to the different values.

[0138] The procurement evaluation network comprises a second input layer, a second hidden layer and a second output layer, the second input layer receives different values of the intermediate audit result and selection probabilities corresponding to the different values, the selection probabilities of the different values of the intermediate audit result are propagated to the second output layer through the second hidden layer, and the selection probabilities of the different values of the intermediate audit result are fed back to the first hidden layer.

[0139] In a specific embodiment, the procurement output network is an Actor network, and the procurement evaluation network is a Critic network. The procurement output network is used to learn an optimal strategy, that is, to select an action a according to the feature transformation vector output by the feedforward neural network, so as to maximize the cumulative reward of the differentiable reward function. The first input layer receives the feature transformation vector output by the feedforward neural network. The first hidden layer comprises a plurality of fully connected layers, each followed by a nonlinear activation function. The output of the first output layer is a probability distribution of the action, that is, the output layer is a Softmax layer, which outputs the probability of each action. The feature transformation vector output by the feedforward neural network is input into the first input layer, and the output of the first hidden layer is calculated through forward propagation of the first hidden layer. The action is generated according to the output of the first output layer.

[0140] The procurement evaluation network is used to evaluate the quality of the action selected by the procurement evaluation network. The second input layer is based on a state-action value function, and the feature representation of the current state and action. The second hidden layer comprises a plurality of fully connected layers, each followed by a nonlinear activation function. The output of the second output layer is a state-action value, and the state-action value is fed back to the first hidden layer.

[0141] In a specific example, a provincial information center is preparing to purchase 120 AI servers with a budget of 6 million yuan. Based on the power procurement to-be-verified file, a procurement vector is obtained, which encodes the procurement demand, technical indicators, budget, historical prices, policy provisions, supplier qualifications and the like into a 512-dimensional dense vector.

[0142] The procurement vector is input into a multi-layer self-attention mechanism comprising 6 layers of self-attention modules for semantic understanding. For example, the first layer associates "120 servers" and "600 million budget" and finds that "unit price 50,000 per unit" falls within the market price range of 45,000-52,000, and is preliminarily marked as "normal price". For another example, the sixth layer compares "supplier qualifications" with "historical bid records" and finds that a supplier has won bids 4 times in the past 3 years, but each time has won the bid at the lowest price by 0.3%, thus generating a hidden state of "suspected bid rigging". Through processing of the procurement vector by the multi-layer self-attention mechanism, the semantic understanding results of each layer of self-attention modules are fused to output a 256-dimensional semantic understanding vector. It can be understood that the multi-layer self-attention mechanism can comprise a plurality of layers of self-attention modules, each layer of self-attention module analyzes different contents, and the number of self-attention modules is configured according to actual needs, which is not limited.

[0143] Then the feedforward neural network converts the 256-dimensional semantic understanding vector output by the multi-layer self-attention mechanism into a 128-dimensional feature conversion vector, maps the "semantic level" features to "decision level" features, for example, "normal price" is converted to a continuous value of 0.12 (close to 0 means no risk). The feature conversion vector is input into the procurement output network, and after passing through the first input layer, the first hidden layer and the first output layer, the intermediate audit result and its probability are obtained, for example, the probability of "pass" is 15%, the probability of "return" is 25%, and the probability of "modification" is 60%. The procurement evaluation network analyzes and evaluates the intermediate audit result and its probability, splices the "pass / return / modify" three one-hot vectors with their respective probabilities, and gives the expected risk after passing through the second hidden layer and the second output layer, for example, the score corresponding to "pass" is -82 points, the score corresponding to "return" is -10 points, and the score corresponding to "modification" is -5 points. Based on this, a 3-dimensional "correction signal" c = [-0.20, +0.05, +0.15] is output, which is used to improve the "modification" probability and lower the "pass" probability. The correction signal c is fed back to the first hidden layer of the procurement output network to adjust the probability.

[0144] Based on the foregoing, it can be understood that the procurement audit decision model constructed by the present application is not a conventional single neural network model, but a "hybrid architecture with external evaluation-feedback loop".

[0145] Further, the first hidden layer includes a shared encoder, and the first output layer includes a main classification head, an auxiliary regression head and an auxiliary binary classification head.

[0146] The shared encoder captures the core features related to the intermediate audit result in the feature conversion vector input into the feedforward neural network;

[0147] The main classification head is used to give the intermediate audit result according to the core features output by the shared encoder;

[0148] The auxiliary regression head is used to analyze the deviation range of the central audit result based on the core features, give the deviation result, and feed back to the second input layer;

[0149] The auxiliary binary classification head is used to analyze the risk of bid rigging of the central audit result based on the core features, give the risk result, and feed back to the second input layer.

[0150] In the procurement output network, the shared encoder belongs to the hidden layer and is responsible for converting the input features into an intermediate representation suitable for subsequent classification and regression tasks. The main classification head, the auxiliary regression head and the auxiliary binary classification head are all modules for further processing based on the output of the shared encoder, and their relationship with the shared encoder is as follows:

[0151] The shared encoder converts the feature vectors of the feedforward neural network input into a higher-level feature representation that captures information relevant to the classification and regression tasks.

[0152] The main classification head predicts the audit result (pass / reject / modify) of the procurement plan based on the output of the shared encoder. The main classification head directly receives the output of the shared encoder and converts it into the output of the classification task.

[0153] The auxiliary regression head predicts the budget deviation based on the output of the shared encoder. The auxiliary regression head directly receives the output of the shared encoder and converts it into the output of the regression task.

[0154] The auxiliary binary classification head predicts whether there is a risk of bid rigging (a binary classification task) based on the output of the shared encoder. The auxiliary binary classification head includes a fully connected layer with an activation function that maps the output of the shared encoder to the probability of bid rigging. The auxiliary binary classification head directly receives the output of the shared encoder and converts it into the output of the binary classification task.

[0155] In a specific example, the procurement vector is a 256-dimensional vector representing various features of the procurement plan (such as budget, quantity, technical specifications, etc.). The input dimension of the shared encoder is 256, and the output is a 64-dimensional feature vector. The main classification head maps the 64-dimensional feature vector output by the shared encoder to output the probability distribution of the audit result: 15% pass, 25% reject, and 60% modify. The auxiliary regression head maps the 64-dimensional feature vector output by the shared encoder to output the budget deviation: the budget deviation is 0.05. The auxiliary binary classification head maps the 64-dimensional feature vector output by the shared encoder to output the probability of bid rigging: the bid rigging risk is 0.87.

[0156] The shared encoder is a hidden layer responsible for feature extraction. The main classification head, auxiliary regression head, and auxiliary binary classification head are output layers responsible for different tasks, but all based on the output of the shared encoder for further processing.

[0157] Further, the network parameters of the multi-layer fully connected network are determined by the following steps:

[0158] Set all parameters of the multi-layer self-attention mechanism and the feedforward neural network to a frozen state;

[0159] Define the input of the multi-layer fully connected network connected to the feedforward neural network, the different values of the output of the multi-layer fully connected network outputting the procurement document verification result, and the probability distribution corresponding to the different values.

[0160] The feature conversion vector output by the feedforward neural network is taken as input data, and a historical audit result label set is taken as supervised data, wherein the historical audit result label set includes an audit label for historical power purchase document auditing;

[0161] The network parameters of the multi-layer fully connected network are adjusted by using a small batch gradient descent method, and the cross-entropy loss between the probability distribution output by the multi-layer fully connected network and the historical audit result label set is calculated.

[0162] The network parameters of the multi-layer fully connected network are updated based on the cross-entropy loss through backpropagation, and the iteration is repeated until the cross-entropy loss converges.

[0163] At the same time, the results of manual auditing are extracted from the historical auditing database to generate a label set for model training, providing a historical experience reference for auditing decisions, so that the model can learn the logic and standards of manual auditing, improving the rationality and accuracy of auditing.

[0164] In a specific embodiment, a procurement audit decision model is used as the basis, and the subject of the procurement audit decision model adopts a Transformer architecture to perform deep semantic understanding and feature conversion on the input procurement vector, which can capture complex semantic relationships and potential information in the power procurement document to be verified. The output of the multi-layer fully connected network is a probability distribution of the procurement document verification result taking values of "pass", "return" and "modify". At the same time, a procurement output network and a procurement evaluation network are mounted. In this example, the procurement output network is an actor network, and the procurement evaluation network is a critic network. The actor network is used to output the selection probability of different values of the procurement document verification result, and the critic network is used to evaluate the value of the current procurement document verification result. Through an ε-greedy strategy, different values of the procurement document verification result are searched, balancing the exploration of new actions and the use of known effective actions. According to a differentiable reward function, a corresponding procurement reward value is obtained to obtain reward feedback. By updating the network parameters of the procurement audit decision model, the selection probability and evaluation result corresponding to different values of the final procurement document verification result are output, so that the procurement audit decision model can optimize the decision, improve the accuracy and adaptability of the audit.

[0165] The procurement review decision model collects feedback data in the daily review process, provides the latest business data support for procurement review decision model updating, periodically re-trains the procurement review decision model with appropriate learning rate using incremental learning technology, so that the model can learn new knowledge without forgetting old knowledge, and adapt to the dynamic changes of business; at the same time, through the generation of edge error samples by the generative adversarial network, the robustness test is carried out to test the performance of the procurement review decision model in extreme or special cases, to ensure that the procurement review decision model can evolve dynamically with new procurement rules and equipment types, and continuously improve the reliability and stability of the review. Among them, the generator of the generative adversarial network adopts the U-Net structure, takes the historical error samples as the input, and generates variation samples by random noise injection; the discriminator is a convolutional neural network, which outputs the distribution difference value of the generated samples and the real samples, and selects the samples with difference value less than 10% for robustness test.

[0166] S106: According to the procurement document verification result, the power procurement to be verified document is labeled, and a procurement document verification report is generated.

[0167] It can be understood that when the review result is passed, the result is clear and no further explanation of the review process is needed, but when the procurement document verification result is returned or modified, the field and graphic element level highlight explanation is generated for the returned or modified power procurement to be verified document, which points out the part that does not meet the hard constraint or review rule, so that the procurement department can intuitively understand the problem and reduce the communication cost. At the same time, these explanations are written into the review report to form a standard and formal feedback document, which is convenient for subsequent tracing and rectification, and improves the transparency and operability of the review work.

[0168] In one specific example, there is an audit task of an audit plan, and the headquarters of a convenience store formulates a beverage procurement plan for three new stores A, B and C in July, which needs to be audited.

[0169] First, the purchase to be verified file is obtained. For example, the demand table of each store, including the store area, the surrounding customer portrait, the historical sales, the promotion period, the demand of various drinks, etc. Then the image features and non-image features are extracted and aligned. Among them, the non-image features are obtained by structuring the various information in the demand table. The non-image features are semantically encoded to obtain the purchase semantic features. For example, "customer portrait = young white-collar" is converted into a one-hot vector, and "historical sales 1200 pieces" is standardized by Z-Score, that is, the numerical field is standardized, the category field is encoded, and the missing value is filled in. Finally, 30-dimensional standard purchase semantic features are obtained. The image features can be the street view and the real photo of each store. There are cars, trees, and pedestrians in the street view. First, use Mask-RCNN to remove irrelevant pixels, then adjust the brightness to 0.5 average, image denoising, size uniformity, and color normalization. Finally, 2048-dimensional standard image features are obtained.

[0170] Through the semantic and visual collaborative feature extraction strategy, the purchase semantic features and the image features are spliced to obtain the purchase vector. The first purchase correlation degree of the purchase semantic features and the image features is analyzed. For example, it is found that "store A semantic says 'dense office buildings'" and "store A street view appears a large number of coffee cup logos" have a cosine similarity of 0.87 in the feature space, that is, the similarity matrix of the semantic features vs. the image features is calculated by the cosine distance, and the first purchase correlation degree (numerical value 0~1) is obtained. "Store A semantic" and "store A image" are bound into a pair; low correlation (such as store C semantic and store A image) is discarded. Based on the first correlation degree threshold (>0.7), the purchase feature pair is constructed. Use the cross-modal Transformer to make the semantic features and the image features notice each other: the "customer" dimension in the semantic features is activated by the "school uniform" in the image, and the "coffee logo" in the image features is activated by the "office building" in the semantic. Output 30-dimensional semantic learning features + 2048-dimensional image learning features, which are aligned to the same latent space. The semantic learning features are 30-dimensional, the image learning features are 2048-dimensional, the image is reduced to 64-dimensional by 1x1 convolution, and the semantic is increased to 64-dimensional, that is, the linear projection ensures the consistency of the dimensions. The 64-dimensional semantic + 64-dimensional image is directly concatenated into a 128-dimensional vector to obtain a 128-dimensional purchase vector.

[0171] Based on the pre-constructed procurement review decision model, combined with the differentiable reward function, the procurement vector is analyzed and judged, the procurement file verification result is given, and the power procurement to-be-verified file is labeled, and the procurement file verification report is generated. For example, input the 128-dimensional vector into the procurement review decision model, output "pass" 75%, that is, the probability of passing the to-be-verified file is 75%, indicating that the procurement plan in the to-be-verified file passes the review, and the procurement can be carried out accordingly. For another example, input the 128-dimensional vector into the procurement review decision model, output "return" 70%, that is, the probability of returning the to-be-verified file is 70%, and the to-be-verified file needs to be modified, and the data in the procurement plan is judged in turn, for example, the number of A drinks in the procurement plan is 22.5 boxes, the number of B drinks is 5000 boxes, and the number of C drinks is 200 boxes, wherein the number of A drinks cannot be a decimal number, the procurement plan of A drinks is labeled, the number of B drinks exceeds the procurement threshold, and the procurement plan of B drinks is highlighted. The number of C drinks meets the requirements and does not need to be labeled, and the procurement file verification report is generated combined with the labeled procurement plan.

[0172] In summary, based on the procurement file verification method based on multi-modal and rule optimization, in the power equipment procurement scene, the structured and aligned multi-modal data is realized through cross-modal learning and feature splicing, the procurement vector is generated, the differentiable reward function is constructed, the hard constraint of procurement review is integrated, the procurement file verification result is output combined with the procurement review decision model, and the procurement file verification report is generated based on the labeled power procurement to-be-verified file. The auditing efficiency, accuracy and dynamic adaptability are greatly improved, and the auditing demand of complex procurement plan in the power industry is met.

[0173] Reference Figure 7 The embodiment of the application provides a procurement file verification device based on multi-modal and rule optimization, comprising:

[0174] The data acquisition module 201 is used for acquiring the power procurement to-be-verified file;

[0175] The feature extraction module 202 is used for extracting the image features and non-image features in the power procurement to-be-verified file, forming multi-modal features, and aligning the multi-modal features;

[0176] The feature encoding module 203 is used for performing semantic encoding on the non-image features to obtain procurement semantic features;

[0177] The feature fusion module 204 is used for splicing the procurement semantic features and the image features through a semantic and visual collaborative feature extraction strategy to obtain a procurement vector;

[0178] The feature analysis module 205 is configured to analyze and judge the purchase vector based on a pre-constructed purchase auditing decision model and in combination with a differentiable reward function, and give a purchase file verification result, wherein the differentiable reward function is obtained by compiling hard constraints of the purchase auditing;

[0179] The report output module 206 is configured to mark the power purchase file to be verified according to the purchase file verification result, and generate a purchase file verification report.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described modules can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0181] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to these embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various modifications and changes to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and changes of the present application fall within the scope of the claims of the present application and their equivalents, the present application also intends to include these modifications and changes.

Claims

1. A method for purchase document verification based on multi-modal and rule optimization, characterized in that, The method comprises the following steps: acquire a power purchase file to be verified; extract image features and non-image features from the power purchase file to be verified, form multi-modal features, and align the multi-modal features; semantically encode the non-image features to obtain purchase semantic features; use a semantic and visual collaborative feature extraction strategy to splice the purchase semantic features and the image features to obtain a purchase vector; based on a pre-constructed purchase review decision model, in combination with a differentiable reward function, analyze and judge the purchase vector to give a purchase file verification result, wherein the differentiable reward function is obtained by compiling hard constraints of the purchase review, specifically including: constructing a matching degree function based on the matching degree of the hard constraints of the purchase review and the purchase vector; constructing a constraint association function based on the semantic similarity of the hard constraints of the purchase review and the purchase vector; balancing the contribution proportions of the matching degree function and the constraint association function in combination with a preset balance coefficient and fusing to give a fitting function of each hard constraint; weighting and summing the fitting functions of each hard constraint based on a preset constraint coefficient and mapping to obtain the differentiable reward function; based on the purchase file verification result, label the power purchase file to be verified and generate a purchase file verification report.

2. The method for procurement document verification based on multimodal and rule optimization as claimed in claim 1 wherein, extract image features from the power purchase file to be verified by the following steps: acquire an original purchase image matrix in the power purchase file to be verified; extract a purchase key area from the original purchase image matrix based on a visual attention allocation function in combination with visual learning parameters to obtain a purchase visual vector; extract a purchase image semantic vector related to the original purchase image matrix from a pre-constructed purchase corpus based on a text semantic mapping function in combination with text learning parameters; based on a vector conversion function, weight and fuse the purchase visual vector and the purchase image semantic vector to obtain image features.

3. The method for procurement document verification based on multimodal and rule optimization as claimed in claim 2, wherein, The image features are specifically represented as: ； wherein V caption is a fixed-dimension image feature; F image is an original purchase image matrix; Ψ(·, Θ Ψ ) is a visual attention allocation function, Θ Ψ is a visual learning parameter; C text is a pre-constructed purchase corpus; Ω(·, Θ ω ) is a text semantic mapping function, Θ ω is a text learning parameter; and is a weighted fusion operation of the purchase visual vector and the purchase semantic vector; and Φ(·) is a vector conversion function.

4. The method for procurement document verification based on multimodal and rule optimization as claimed in claim 1 wherein, use a semantic and visual collaborative feature extraction strategy to splice the purchase semantic features and the image features to obtain a purchase vector, specifically including: preprocess the purchase semantic features and the image features respectively to obtain standard purchase semantic features and standard image features; analyze the similarity of each standard purchase semantic feature and standard image feature to give a first purchase correlation degree; based on the first purchase correlation degree, construct a purchase feature pair of the standard purchase semantic features and the standard image features, and perform cross-modal learning on the standard purchase semantic features and the standard image features in the purchase feature pair to adjust the standard purchase semantic features and the standard image features to obtain semantic learning features and image learning features; perform dimension consistency processing on the semantic learning features and the image learning features and splice them to obtain the purchase vector.

5. The method for procurement document verification based on multimodal and rule optimization as claimed in claim 1 wherein, based on a pre-constructed purchase review decision model, in combination with a differentiable reward function, analyze and judge the purchase vector to give a purchase file verification result, specifically including: perform deep semantic understanding and feature conversion on the purchase vector to obtain a model purchase vector; based on the differentiable reward function, analyze the fitting degree of each hard constraint in the purchase review and the model purchase vector and fuse to obtain a purchase reward value corresponding to the model purchase vector; The search model procurement vector corresponds to different values of the procurement document verification result, and the model procurement vector corresponds to the procurement reward value, so as to obtain the selection probability and evaluation result corresponding to the different values of the procurement document verification result.

6. The method for procurement document verification based on multimodal and rule optimization as claimed in claim 5, wherein, The procurement review decision model comprises a multi-layer self-attention mechanism, a feedforward neural network, and a multi-layer fully connected network comprising a procurement output network and a procurement evaluation network. The feedforward neural network is connected to the multi-layer self-attention mechanism and receives the semantic understanding vector output by the multi-layer self-attention mechanism for feature conversion. The multi-layer fully connected network is connected to the feedforward neural network and outputs different values of the procurement document verification result and the probability distribution corresponding to the different values. The procurement output network comprises a first input layer, a first hidden layer, and a first output layer. The first input layer receives the feature conversion vector output by the feedforward neural network and transmits it to the first output layer through the first hidden layer to obtain different values of the intermediate review result and the selection probability corresponding to the different values. The procurement evaluation network comprises a second input layer, a second hidden layer, and a second output layer. The second input layer receives the different values of the intermediate review result and the selection probability corresponding to the different values, and transmits them to the second output layer through the second hidden layer to adjust the selection probability of the different values of the intermediate review result and feed it back to the first hidden layer.

7. The method for procurement document verification based on multimodal and rule optimization as claimed in claim 6, wherein, The network parameters of the multi-layer fully connected network are determined by the following steps: Set all parameters of the multi-layer self-attention mechanism and the feedforward neural network to a frozen state. Define the input end of the multi-layer fully connected network to be connected to the feedforward neural network, and the output end of the multi-layer fully connected network to output different values of the procurement document verification result and the probability distribution corresponding to the different values. Take the feature conversion vector output by the feedforward neural network as input data and the historical review result label set as supervision data, wherein the historical review result label set comprises review labels for historical power procurement documents. Use the mini-batch gradient descent method to adjust the network parameters of the multi-layer fully connected network, calculate the cross-entropy loss between the probability distribution output by the multi-layer fully connected network and the historical review result label set, and update the network parameters of the multi-layer fully connected network based on the cross-entropy loss through back propagation. The first hidden layer comprises a shared encoder, and the first output layer comprises a main classification head, an auxiliary regression head, and an auxiliary binary classification head.

8. The method for procurement document verification based on multimodal and rule optimization as claimed in claim 6, wherein, The shared encoder captures the core features in the feature conversion vector input by the feedforward neural network that are related to the intermediate review result. The main classification head is used to give the intermediate review result based on the core features output by the shared encoder. The auxiliary regression head is used to analyze the deviation range of the central review result based on the core features and give the deviation result, which is fed back to the second input layer. The auxiliary binary classification head is used to analyze the risk of bid rigging of the central review result based on the core features and give the risk result, which is fed back to the second input layer. The method comprises the following steps:

9. A procurement document verification device based on multi-modal and rule optimization, characterized by, A data acquisition module is used to acquire the power procurement document to be verified. A feature extraction module is used to extract image features and non-image features from the power procurement document to be verified, form multi-modal features, and align the multi-modal features. ​ The feature coding module is configured to perform semantic coding on the non-image features to obtain procurement semantic features; The feature fusion module is configured to splice the procurement semantic features and the image features by a semantic and visual collaborative feature extraction strategy to obtain a procurement vector; The feature analysis module is configured to analyze and judge the procurement vector based on a pre-constructed procurement review decision model and in combination with a differentiable reward function to give a procurement document verification result, wherein the differentiable reward function is obtained by compiling hard constraints of procurement review, and specifically includes: constructing a matching degree function based on a matching degree between the hard constraints of the procurement review and the procurement vector; constructing a constraint association function based on a semantic similarity between the hard constraints of the procurement review and the procurement vector; balancing a contribution proportion of the matching degree function and the constraint association function in combination with a preset balance coefficient and fusing to give a fitting function of each hard constraint; and obtaining the differentiable reward function by weighting and summing the fitting functions of the hard constraints and mapping based on a preset constraint coefficient; The report output module is configured to label the power procurement document to be verified according to the procurement document verification result and generate a procurement document verification report.

Images