Multi-dimensional document aggregation calculation method and device, equipment and medium

Abstract

The invention relates to a multi-dimensional document aggregation calculation method and device, equipment and a medium, which are applied to the technical field of data processing, and the method comprises the following steps: obtaining a document data set, an aggregation label of the document data set, a special efficacy level label, a target dimension direction and an initial dimension weight corresponding to the target dimension direction; performing feature data extraction on the document data set based on the target dimension direction, and outputting dimension data; constructing a to-be-trained data set based on the document data set, the aggregation tag, the special efficacy level tag and the dimension data; constructing a weight adjustment model based on the to-be-trained data set and the initial dimension weight; obtaining a to-be-processed material document; and performing weight calculation on the to-be-processed material document based on the weight adjustment model to generate a document aggregation result. The method and the device have the effect of accurately arranging the files according to the sequence.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a multi-dimensional document aggregation calculation method, device, equipment and medium. Background Art

[0002] When processing files, it is necessary to ensure the coherence of the document content being processed. When users upload documents, they often only upload a whole PDF file. However, since some files contain multiple documents of different types, such as contracts, bill details, and personnel information, most users upload and integrate them by taking photos, thereby splitting each type of document into multiple pictures for mixed upload. As a result, the pictures between documents are mixed disorderly, making the final generated file context incoherent, that is, multiple documents are mixed disorderly into one file. This form of file will greatly affect the file processing efficiency. Therefore, how to ensure the continuity of the content of all pages of the merged file is extremely important. Summary of the Invention

[0003] In order to accurately arrange files in order, the present application provides a multi-dimensional document aggregation calculation method, device, equipment and medium.

[0004] In the first aspect, the present application provides a multi-dimensional document aggregation calculation method, which adopts the following technical solutions: A multi-dimensional document aggregation calculation method, comprising: Obtaining a document dataset, an aggregation label of the document dataset, a special effectiveness level label, a target dimension direction, and an initial dimension weight corresponding to the target dimension direction; Extracting feature data from the document dataset based on the target dimension direction, and outputting dimension data; Constructing a dataset to be trained based on the document dataset, the aggregation label, the special effectiveness level label, and the dimension data; Constructing a weight adjustment model based on the dataset to be trained and the initial dimension weights; Obtain documents of materials to be processed; The weight of the material document to be processed is calculated based on the weight adjustment model to generate a document aggregation result.

[0005] By adopting the above technical solution, existing document data is collected to construct a document dataset, feature data is extracted from the obtained document dataset to obtain dimensional data, the obtained dimensional data is correspondingly bound and integrated with the document dataset, aggregation label and special effectiveness level label to obtain a dataset to be trained, and a weight adjustment model is constructed for the preset and initial dimensional weights using the dataset to be trained. The weight adjustment model is used to calculate the weight of each document in the to-be-processed material documents to obtain aggregation data between documents, and the to-be-processed material documents are aggregated and arranged according to the aggregation data to obtain the final document aggregation result. When performing the aggregation processing, the preset initial dimensional weights are adjusted according to the actual document situation, so that the final weight is more in line with the actual situation of the document, thereby accurately arranging the files in order.

[0006] Optionally, constructing a training dataset based on the document dataset, the aggregation label, the special effectiveness level label, and the dimension data includes: Adding the corresponding aggregation tag and the special effectiveness level tag to each document data in the document data set and binding them to generate a labeled document data set; Combining the label document data in the label document dataset based on the dimension data to generate a page number pair combination; The page numbers are annotated based on manual annotation and a preset large model to generate positive and negative samples; The positive and negative samples are integrated to construct a training data set.

[0007] Optionally, constructing a weight adjustment model based on the dataset to be trained and the initial dimension weights includes: Get the GBDT base model and training stop conditions; Calculate the sorting loss based on the dataset to be trained, the initial dimension weights, and the GBDT base model; Generate Lambda gradient based on the ranking loss; Training a preset LambdaMART model based on the Lambda gradient to generate a training model; Determining whether the training model meets the training stop condition; If the training model meets the training stop condition, the trained LambdaMART model is used as the weight adjustment model; If the training model does not meet the training stop condition, the step of training the preset LambdaMART model based on the Lambda gradient is repeated until the training stop condition is met.

[0008] Optionally, the training of a preset LambdaMART model based on the Lambda gradient includes: Obtain hard constraint rule base, special constraints, base model and training stop conditions; Constructing a special constraint model based on the hard constraint rule base and the special constraints; Calculate the loss gradient based on the Lambda gradient and the special constraint model; Constructing a decision tree based on the loss gradient; Adjusting the initial dimension weights based on the decision tree to generate adjusted weights; Adjusting and judging the decision tree based on the special constraint model to generate an adjusted decision tree; The preset LambdaMART model is trained based on the adjusted weights and the adjusted decision tree.

[0009] Optionally, performing weight calculation on the to-be-processed material documents based on the weight adjustment model to generate a document aggregation result includes: Identify and process the material document to be processed based on the target dimensional direction to generate dimensional data to be processed; Performing weight calculation on the material document to be processed based on the dimensional data to be processed and the weight adjustment model to generate a weight calculation result; The to-be-processed material documents are sorted based on the weight calculation result to generate a document aggregation result.

[0010] Optionally, performing weight calculation on the material document to be processed based on the dimensional data to be processed and the weight adjustment model to generate a weight calculation result includes: Determining the material type of the material document to be processed based on the dimension data to be processed; Determining a confidence compensation value and a weight adjustment value corresponding to each document in the to-be-processed material documents based on the material type and a preset adjustment engine; A weight calculation is performed on the processing material document based on the weight adjustment model, the confidence compensation value, and the weight adjustment value to generate a weight calculation result.

[0011] Optionally, after sorting the documents to be processed based on the weight calculation result and generating a document aggregation result, the method further includes: Obtaining manual verification results and feedback data of the document aggregation results; The preset adjustment engine is updated based on the manual verification result and the feedback data.

[0012] In a second aspect, the present application provides a multi-dimensional document aggregation computing device, which adopts the following technical solution: A multi-dimensional document aggregation computing device comprises: An initial data acquisition module is configured to acquire a document data set, an aggregation label of the document data set, a special effectiveness level label, a target dimension direction, and an initial dimension weight corresponding to the target dimension direction; A dimension data output module is configured to perform feature data extraction on the document data set based on the target dimension direction and output dimension data; A to-be-trained data construction module is configured to construct a to-be-trained data set based on the document data set, the aggregation label, the special effectiveness level label, and the dimension data; An adjustment model construction module is configured to construct a weight adjustment model based on the to-be-trained data set and the initial dimension weight; A material document acquisition module is configured to acquire a to-be-processed material document; An aggregation result generation module is configured to perform weight calculation on the to-be-processed material document based on the weight adjustment model and generate a document aggregation result.

[0013] By adopting the above technical solution, the existing document data is collected to construct a document data set, the obtained document data set is subjected to feature data extraction to obtain dimension data, the obtained dimension data is correspondingly bound and integrated with the document data set, the aggregation label, and the special effectiveness level label to obtain a to-be-trained data set, the to-be-trained data set is used to construct a weight adjustment model for preset and initial dimension weight adjustment, the weight adjustment model is used to perform weight calculation on each document in the to-be-processed material document to obtain aggregation data between documents, the to-be-processed material document is arranged and processed according to the aggregation data to obtain a final document aggregation result, and the preset initial dimension weight is adjusted according to the actual document condition during the aggregation processing, so that the final weight is more consistent with the actual condition of the document, thereby accurately arranging the files in order.

[0014] In a third aspect, the present application provides an electronic device, which adopts the following technical solution: An electronic device comprises a processor coupled with a memory; The processor is configured to execute a computer program stored in the memory to enable the electronic device to execute the computer program of the multi-dimensional document aggregation computing method of any one of the first aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and execute the multi-dimensional document aggregation calculation method described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of a multi-dimensional document aggregation calculation method provided in an embodiment of the present application.

[0017] Figure 2 This is a structural block diagram of a multi-dimensional document aggregation computing device provided in an embodiment of the present application.

[0018] Figure 3 This is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The present application is further described in detail below with reference to the accompanying drawings.

[0020] Embodiments of the present application provide a multi-dimensional document aggregation computing method that can be executed by an electronic device, which can be a server or a terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, a tablet computer, a desktop computer, etc., but is not limited thereto.

[0021] Figure 1 A flowchart of a multi-dimensional document aggregation calculation method provided in an embodiment of the present application.

[0022] like Figure 1 As shown, the main process of the method is described as follows (steps S101 to S106): Step S101: Obtain a document dataset, an aggregation label of the document dataset, a special effectiveness level label, a target dimension direction, and an initial dimension weight corresponding to the target dimension direction.

[0023] In this embodiment, existing documents are collected and integrated to obtain a document data set. The document data in the document data set are all processed complete document combinations with correct form, that is, various types of multi-page documents composed of multiple separate image document data. In the initial stage of document aggregation processing, it is necessary to clarify the dimensional direction mainly used in the aggregation processing, as well as whether the document needs to be aggregated and the effectiveness level of the special standard document corresponding to this type of document. The special document here is a standard document issued by an official department that is binding on citizens. The special effectiveness level is the constraint level corresponding to the document. For example, there are three standard documents ABC. Standard document A has the greatest binding force, standard document B is second, and standard document C has the least binding force. When formulating the level, it can be set in increments according to the size of the binding force, that is, the binding force of standard document A is level 3, the binding force of standard document B is level 2, the binding force of standard document C is level 3, and so on. Different types of document data correspond to different special effectiveness levels, thereby adding different effectiveness level labels.

[0024] Similarly, different document data also require different aggregation information. For example, some two document data cannot be aggregated, while some two document data must be aggregated together. Therefore, corresponding aggregation tags are added to each document data according to actual needs.

[0025] In this application, there are four target dimension directions, namely document type matching, semantic similarity, visual continuity and page number proximity. When calculating the similarity between two documents, a comprehensive calculation is performed based on the above four directions, and the document data surface with the greatest similarity to the current page is used as the next page of the current page. The initial dimension weights corresponding to the target dimension directions are preliminarily set according to the existing data, which are 0.4, 0.3, 0.2 and 0.1 respectively. The overall similarity aggregation calculation formula is: Similarity = 0.4*Document Type Matching + 0.3*Semantic Similarity + 0.2*Visual Continuity + 0.1*Page Number Proximity. In actual application, the numerical values ​​corresponding to the weights will be adjusted to obtain the page similarity that best fits the current document data situation, thereby improving the accuracy of page aggregation.

[0026] Step S102 : extracting feature data from the document dataset based on the target dimension direction, and outputting dimension data.

[0027] In this embodiment, after obtaining the document data set, the document data in the document data set are combined in pairs to calculate the similarity of the four dimensions between the two pages. When performing the similarity calculation, the existing big data model can be used and the corresponding code program can be set according to actual needs. The specific calculation method is not specifically limited here.

[0028] Before calculating the document type similarity, it is necessary to first determine the document type of the current document data. The calculation of the document type includes three levels, which need to be classified and judged one by one, that is, the first level is used for judgment first. If the type judgment fails, the judgment method of the next level will be used step by step. The purpose of dividing into three levels is to balance efficiency and quality. If the file type can be quickly determined by the first-level small model, the subsequent judgment will not be performed. It is efficient and can quickly pass through key information. If the first level cannot be identified, it will be judged by the second-level small model. However, due to the small training data set of the small model, the judgment speed is slow, but it can identify types that cannot be judged at the first level. When the second level cannot be judged, it will finally be judged by the third-level large model. Similarly, the third-level large model also has a slow judgment speed, but the effect is the best and can accurately identify the correct type.

[0029] The specific identification methods are: Level 1: Coarse-grained classification (FastText core layer) 1. Data enhancement: Build a specialized vocabulary for specific fields: Extract keywords from tens of thousands of bankruptcy documents using TF-IDF+TextRank; Use back-translation technology (Chinese ⇄ English ⇄ Chinese) to expand training samples and improve model generalization ability; 2. Model Improvement: Improved FastText's n-gram window: Dynamically adjust the weight of 2-5 character combinations to improve recognition of special terms; The label attention mechanism is introduced, with an accuracy of 96.2% on the ICLR-2023 dataset; Output: Output major categories (such as identity certificates / debt certificates / special department documents, etc.) and confidence levels; Second level: fine-grained classification (hybrid neural network layer) Accurately classify special documents through the collaborative architecture of TextCNN+BiGRU sub-models; Level 3: Failure Recovery (Large Language Model Inference Layer) Trigger condition: Automatically activated when the confidence level of the secondary model is less than 85%; It supports interaction with Deepseek's large model, enhances prompt engineering through special knowledge, identifies file types, and provides a feedback mechanism. It adds LLM judgment results to the training set and updates the secondary model parameters every week.

[0030] Step S103: constructing a training dataset based on the document dataset, aggregation labels, special effectiveness level labels and dimension data.

[0031] For step S103, the corresponding aggregation label and special effectiveness level label are added to each document data in the document data set and bound to generate a labeled document data set; the labeled document data in the labeled document data set are combined based on the dimensional data to generate a page number pair combination; the page number pair combination is labeled based on manual labeling and the preset large model labeling to generate positive and negative samples; the positive and negative samples are integrated to construct the data set to be trained.

[0032] In this embodiment, when generating the document data to be trained, the aggregation label and level effectiveness label of each document data are first added to the document data and bound, thereby obtaining labeled document data carrying aggregation information and special effectiveness information, that is, a single document page carrying aggregation information and special effectiveness information. When extracting dimensional data, the four calculated values ​​are all calculated based on two document pages. Therefore, when constructing the training dataset, it is also necessary to perform page number pair processing on the labeled document data to generate multiple page number pair combinations, and add the dimensional data of each page number pair combination to the page number pair combination. For example, a 10-page document will output 45 feature vectors. Feature examples include: f1: Type matching degree between the P5 contract body and the P8 supplementary clause = 0.92, f2: Semantic similarity between the P8 clause and the P5 main contract = 0.85, f3: Visual continuity between the P5 table and the P8 continuation table = 0.78 (due to the cross-page table), f4: Page number decay value = exp(-0.32 * 3)=0.38 (interval 3 pages), on this basis, the manual annotation and preset large model annotation page numbers are combined to generate positive and negative samples, and the training data set is obtained after summing up all the positive and negative samples.

[0033] When adding information such as dimensional data, it can be implemented through programming language, specifically: {"sample_id":"2023破字第12345-5-8", / / Document TD+ page "features":[0.92, 0.85, 0.78, 0.38], / / Four-dimensional feature vector "label":2, / / Legal effectiveness level (1-5) "legal_metadata":{ "doc_type":"Declaration","issuing_authority":"Unit","reference laws"": [ Article 47 of the First Document, Article 642 of the Second Document ], "evidence_chain index": 0.82 / / Evidence chain integrity score }, "visual_proof": "s3: / / legal-bucket / 2023破12345 / p5-8_layout.png" / / Publisher Analysis Chart} Step S104: construct a weight adjustment model based on the dataset to be trained and the initial dimension weights.

[0034] For step S104, obtain the GBDT base model and the training stop condition; calculate the sorting loss based on the data set to be trained, the initial dimension weights and the GBDT base model; generate the Lambda gradient based on the sorting loss; train the preset LambdaMART model based on the Lambda gradient to generate a training model; determine whether the training model meets the training stop condition; if the training model meets the training stop condition, use the trained LambdaMART model as the weight adjustment model; if the training model does not meet the training stop condition, repeat the step of training the preset LambdaMART model based on the Lambda gradient until the training stop condition is met.

[0035] In this embodiment, the ranking learning algorithm LambdaMART is used to optimize the initial dimension weights, and the model is trained using labeled data to adjust the weights of each feature. The preset LambdaMART model = gradient boosting tree (GBDT) + ranking loss optimization + legal constraint injection, where the gradient boosting tree (GBDT) is implemented using the GBDT base model. By iteratively constructing a decision tree, the weight deviation is gradually corrected to achieve gradient boosting; the Lambda gradient converts the change in ranking indicators such as NDCG into a gradient signal, soft constraint regularization is performed, and special rule constraint terms are embedded in the loss function.

[0036] Furthermore, training the preset LambdaMART model based on Lambda gradient includes: obtaining a hard constraint rule base, special constraints, a base model and training stop conditions; constructing a special constraint model based on the hard constraint rule base and special constraints; calculating the loss gradient based on Lambda gradient and the special constraint model; constructing a decision tree based on the loss gradient; adjusting the initial dimension weights based on the decision tree to generate adjusted weights; adjusting and judging the decision tree based on the special constraint model to generate an adjusted decision tree; and training the preset LambdaMART model based on the adjusted weights and the adjusted decision tree.

[0037] In this embodiment, special constraint modeling includes two key parts. The first part is the hard constraint rule base, which contains the mandatory hard constraints, such as the signature position must have a signature, the seal position must have an official seal, and the claim declaration form must be prioritized with the claim formation process. The second part is soft constraint regularization, which injects special constraints and adds a regularization phase to the loss function: ; in, is the initial dimension weight, For violation of special restrictions in Article j.

[0038] The default LambdaMART model architecture is: class LegalLambdaMART: def _init_(self): self.trees =[ ] # Decision tree collection self.learning_rate = 0.1 self.legal_rules =load legal_constraints() def fit(self,x,y,qids): lambdas = self.calc lambda(y,predictions)# Calculate the Lambda gradient of each tree lambdas = apply_legal_correction(lambdas, self.legal_rules)# Special constraint gradient correction tree =build tree(X,lambdas)self.trees.append(tree)#Build a policy tree to fit the residuals.

[0039] Steps of Gradient Boosting Tree (GBDT) + Sorting Loss Optimization + Special Constraint Injection: 1. Gradient boosted tree (GBDT), which refers to the generation of tree structures in special constraint modeling and the basic learner in LambdaMART model construction; Tree structure generation (decision tree): The core role of tree structure generation in special constraint modeling is the key process of converting special rules into interpretable decision paths, achieving: a. Explicit special logic: Encode the Bankruptcy Law and other provisions into tree splitting rules; b. Feature selection constraints: prohibit the use of illegal features; c. Decision traceability: Each leaf node is associated with a specific basis (e.g., Article 142 of the Civil Code); When generating a tree, special constraints need to be injected (as mentioned in the previous article, i.e. soft constraint injection), for example: feature disabling, skipping sensitive features (such as ID card number) when splitting, the special basis is Article 51 of the Personal Information Protection Law.

[0040] Base learner (algorithm) in LambdaMART model construction: In the LambdaMART framework, the base learner is a single decision tree in the gradient boosted tree (GBDT). Its core function is to gradually fit the residual (pseudo gradient) through feature splitting; The tree structure in special constraint modeling provides special knowledge to the base learner, and the base learner provides feedback on the quality of the structure tree. Tree structure → Base learner: Inject special rules to ensure that the split of each decision tree conforms to judicial logic; Basic Learner → Tree Structure: Iteratively optimizes the numerical expression of special rules through gradient boosting.

[0041] 2. Sorting loss optimization refers to the sample sorting and labeling in training data construction and the Lambda gradient calculation in LambdaMART model construction. This is a cyclic process. By default, the base model is used to generate Lambda gradients and build a decision tree. Then, according to special regularization, the GBDT base model is updated.

[0042] 3. Special constraint injection: regularization term design in special constraint modeling.

[0043] Step S105: Obtain the material document to be processed.

[0044] In this embodiment, after the weight adjustment model is constructed, an aggregation calculation process is performed on the newly obtained documents to obtain the documents that have not been processed and are dispersed into multiple single pages, and all the document pages are used as material documents to be processed.

[0045] Step S106: performing weight calculation on the material documents to be processed based on the weight adjustment model to generate a document aggregation result.

[0046] For step S106, the material documents to be processed are identified and processed based on the target dimension direction to generate dimension data to be processed; weight calculation is performed on the material documents to be processed based on the dimension data to be processed and the weight adjustment model to generate weight calculation results; the material documents to be processed are sorted based on the weight calculation results to generate document aggregation results.

[0047] In this embodiment, after obtaining the material document to be processed, dimensional analysis is also performed, that is, the dimensional data of the material document to be processed is calculated and processed using the target dimensional direction to obtain the dimensional data to be processed, and the weight calculation of the material document to be processed is performed using the weight adjustment model and the dimensional data to be processed to obtain the weight data between each pair of document pages. Similarity calculation is performed based on the weight data and the similarity calculation formula mentioned above, so that the two document pages with the highest similarity are aggregated and sorted to obtain the document aggregation result.

[0048] Furthermore, weight calculation is performed on the material documents to be processed based on the dimensional data to be processed and the weight adjustment model, and the weight calculation results are generated, including: determining the material type of the material documents to be processed based on the dimensional data to be processed; determining the confidence compensation value and weight adjustment value corresponding to each document in the material documents to be processed based on the material type and the preset adjustment engine; weight calculation is performed on the processed material documents based on the weight adjustment model, the confidence compensation value and the weight adjustment value, and the weight calculation results are generated.

[0049] In this embodiment, in addition to the weights continuously calculated by the weight adjustment model, dynamic adjustments are also required during use, and the weights of the actual material documents to be processed are adjusted using a preset adjustment engine. The adjustment of the preset adjustment engine is based on special constraint modeling, material type, confidence and other information, and is equipped with confidence assessment and compensation rules. According to the confidence obtained during the material type identification process, the different impacts on each weight under the current material type and confidence are determined, and special constraint modeling is applied for boundary constraints to obtain the weight calculation result between the two document pages that best suits the current material type.

[0050] In this embodiment, manual verification results and feedback data of document aggregation results are obtained; and a preset adjustment engine is updated based on the manual verification results and feedback data.

[0051] After obtaining the document aggregation results, manual verification is performed to obtain the verification results and response feedback data, and the manual verification results and feedback data are adjusted and corrected with the hard constraint rule library.

[0052] Figure 2 A structural block diagram of a multi-dimensional document aggregation computing device 200 provided in an embodiment of the application.

[0053] like Figure 2 As shown, the multi-dimensional document aggregation computing device 200 mainly includes: Initial data acquisition module 201, used to acquire a document dataset, an aggregation label of the document dataset, a special effectiveness level label, a target dimension direction, and an initial dimension weight corresponding to the target dimension direction; The dimension data output module 202 is used to extract feature data from the document dataset based on the target dimension direction and output dimension data; A training data construction module 203 is used to construct a training data set based on the document data set, the aggregation label, the special effect level label and the dimension data; An adjustment model building module 204 is used to build a weight adjustment model based on the training data set and the initial dimension weights; The material document acquisition module 205 is used to acquire the material document to be processed; The aggregation result generating module 206 is used to perform weight calculation on the material documents to be processed based on the weight adjustment model to generate document aggregation results.

[0054] As an optional implementation of this embodiment, the training data construction module 203 is specifically used to add corresponding aggregation tags and special effectiveness level tags to each document data in the document data set to bind them to generate a labeled document data set; combine the labeled document data in the labeled document data set based on dimensional data to generate page number pair combinations; label the page number pair combinations based on manual labeling and preset large model labeling to generate positive and negative samples; integrate the positive and negative samples to construct the training data set.

[0055] As an optional implementation of this embodiment, the adjustment model building module 204 includes: Stop condition acquisition module, used to obtain the GBDT base model and training stop conditions; The sorting loss calculation module is used to calculate the sorting loss based on the training data set, initial dimension weights and GBDT base model; Gradient data generation module, used to generate Lambda gradients based on ranking loss; The training model generation module is used to train the preset LambdaMART model based on Lambda gradient to generate a training model; The training condition judgment module is used to judge whether the training model meets the training stop condition; The weight model generation module is used to use the trained LambdaMART model as the weight adjustment model; The training step repetition module is used to repeat the steps of training the preset LambdaMART model based on the Lambda gradient until the training stop condition is met.

[0056] In this optional embodiment, the training model generation module is specifically used to obtain a hard constraint rule base, special constraints, a base model and training stop conditions; construct a special constraint model based on the hard constraint rule base and the special constraints; calculate the loss gradient based on the Lambda gradient and the special constraint model; construct a decision tree based on the loss gradient; adjust the initial dimension weights based on the decision tree to generate adjusted weights; adjust and judge the decision tree based on the special constraint model to generate an adjusted decision tree; and train the preset LambdaMART model based on the adjusted weights and the adjusted decision tree.

[0057] As an optional implementation of this embodiment, the aggregation result generation module 206 includes: The material document processing module is used to identify and process the material document to be processed based on the target dimension direction and generate the dimension data to be processed; A weight result generation module is used to perform weight calculation on the material documents to be processed based on the dimension data to be processed and the weight adjustment model, and generate a weight calculation result; The document result generation module is used to sort the documents of the materials to be processed based on the weight calculation results and generate document aggregation results.

[0058] In this optional embodiment, the weight result generation module is specifically used to determine the material type of the material document to be processed based on the dimensional data to be processed; determine the confidence compensation value and weight adjustment value corresponding to each document in the material document to be processed based on the material type and the preset adjustment engine; perform weight calculation on the processed material document based on the weight adjustment model, confidence compensation value and weight adjustment value, and generate a weight calculation result.

[0059] As an optional implementation of this embodiment, the multi-dimensional document aggregation computing device 200 includes: Result data acquisition module, used to obtain manual verification results and feedback data of document aggregation results; The data update processing module is used to update the preset adjustment engine based on the manual verification results and feedback data.

[0060] In one example, the module in any of the above devices can be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0061] For another example, when the modules in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0062] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0063] Figure 3 This is a structural block diagram of the electronic device 300 provided in an embodiment of the present application.

[0064] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302 , and may further include an information input / information output (I / O) interface 303 , one or more communication components 304 , and a communication bus 305 .

[0065] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps of the multi-dimensional document aggregation calculation method described above. The memory 302 is used to store various types of data to support the operation of the electronic device 300. For example, these data may include instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0066] The I / O interface 303 provides an interface between the processor 301 and other interface modules, which can be a keyboard, a mouse, a button, etc. The buttons can be virtual buttons or physical buttons. The communication component 304 is configured to perform wired or wireless communication between the electronic device 300 and other devices. The wireless communication, for example, Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, and thus the communication component 304 can include a Wi-Fi component, a Bluetooth component, and an NFC component.

[0067] The electronic device 300 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for performing the multi-dimensional document aggregation calculation method according to the embodiments.

[0068] The communication bus 305 can include a path for transmitting information between the components. The communication bus 305 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 305 can be divided into an address bus, a data bus, a control bus, etc.

[0069] The electronic device 300 can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet PC), a PMP (Portable Multimedia Player), a car terminal (e.g., a car navigation terminal), etc., and a fixed terminal such as a digital TV, a desktop computer, etc., and can also be a server, etc.

[0070] The present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the multi-dimensional document aggregation calculation method described above.

[0071] The computer readable storage medium can include a variety of media that can store program codes, such as a U disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

[0072] The terms "comprise", "contain", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements not only includes those elements, but also includes other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0073] The above description is merely preferred embodiments of the present application and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with technical features with similar functions applied in the present application (but not limited to) to form technical solutions.

Claims

1. A multi-dimensional document aggregation calculation method, characterized in that: include: Obtaining a document dataset, an aggregation label of the document dataset, a special effectiveness level label, a target dimension direction, and an initial dimension weight corresponding to the target dimension direction; Extracting feature data from the document dataset based on the target dimension direction, and outputting dimension data; Constructing a dataset to be trained based on the document dataset, the aggregation label, the special effectiveness level label, and the dimension data; Constructing a weight adjustment model based on the dataset to be trained and the initial dimension weights; Obtain documents of materials to be processed; The weight of the material document to be processed is calculated based on the weight adjustment model to generate a document aggregation result.

2. The method according to claim 1, characterized in that The constructing of a training dataset based on the document dataset, the aggregation label, the special effectiveness level label, and the dimension data includes: Adding the corresponding aggregation tag and the special effectiveness level tag to each document data in the document data set and binding them to generate a labeled document data set; Combining the label document data in the label document dataset based on the dimension data to generate a page number pair combination; The page numbers are annotated based on manual annotation and a preset large model to generate positive and negative samples; The positive and negative samples are integrated to construct a training data set.

3. The method according to claim 1, characterized in that The constructing of a weight adjustment model based on the to-be-trained data set and the initial dimension weights includes: Get the GBDT base model and training stop conditions; Calculate the sorting loss based on the dataset to be trained, the initial dimension weights, and the GBDT base model; Generate Lambda gradient based on the ranking loss; Training a preset LambdaMART model based on the Lambda gradient to generate a training model; Determining whether the training model meets the training stop condition; If the training model meets the training stop condition, the trained LambdaMART model is used as the weight adjustment model; If the training model does not meet the training stop condition, the step of training the preset LambdaMART model based on the Lambda gradient is repeated until the training stop condition is met.

4. The method according to claim 3, characterized in that The training of the preset LambdaMART model based on the Lambda gradient includes: Obtain hard constraint rule base, special constraints, base model and training stop conditions; Constructing a special constraint model based on the hard constraint rule base and the special constraints; Calculate the loss gradient based on the Lambda gradient and the special constraint model; Constructing a decision tree based on the loss gradient; Adjusting the initial dimension weights based on the decision tree to generate adjusted weights; Adjusting and judging the decision tree based on the special constraint model to generate an adjusted decision tree; The preset LambdaMART model is trained based on the adjusted weights and the adjusted decision tree.

5. The method according to claim 4, characterized in that The weight calculation of the to-be-processed material documents based on the weight adjustment model to generate a document aggregation result includes: Identify and process the material document to be processed based on the target dimensional direction to generate dimensional data to be processed; Performing weight calculation on the material document to be processed based on the dimensional data to be processed and the weight adjustment model to generate a weight calculation result; The to-be-processed material documents are sorted based on the weight calculation result to generate a document aggregation result.

6. The method according to claim 5, characterized in that The performing weight calculation on the material document to be processed based on the dimensional data to be processed and the weight adjustment model to generate a weight calculation result includes: Determining the material type of the material document to be processed based on the dimension data to be processed; Determining a confidence compensation value and a weight adjustment value corresponding to each document in the to-be-processed material documents based on the material type and a preset adjustment engine; A weight calculation is performed on the processing material document based on the weight adjustment model, the confidence compensation value, and the weight adjustment value to generate a weight calculation result.

7. The method according to claim 6, characterized in that After sorting the to-be-processed material documents based on the weight calculation result and generating a document aggregation result, the method includes: Obtaining manual verification results and feedback data of the document aggregation results; The preset adjustment engine is updated based on the manual verification result and the feedback data.

8. A multi-dimensional document aggregation computing device, characterized in that: include: An initial data acquisition module is used to acquire a document dataset, an aggregation label of the document dataset, a special effectiveness level label, a target dimension direction, and an initial dimension weight corresponding to the target dimension direction; A dimension data output module, configured to extract feature data from the document dataset based on the target dimension direction and output dimension data; A training data construction module, configured to construct a training data set based on the document data set, the aggregation label, the special effectiveness level label, and the dimension data; An adjustment model construction module, configured to construct a weight adjustment model based on the dataset to be trained and the initial dimension weights; Material document acquisition module, used to obtain material documents to be processed; The aggregation result generation module is used to perform weight calculation on the material documents to be processed based on the weight adjustment model to generate document aggregation results.

9. An electronic device, characterized in that: comprising a processor coupled to a memory; The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The method comprises a computer program or an instruction, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images