Data processing method and device, computer equipment, readable storage medium and program product
The data extraction and verification of preset documents through the data extraction model solves the problem of low efficiency of manual extraction and quality inspection in traditional methods, and realizes efficient and accurate data processing and model optimization.
Patent Information
- Application Number
- CN202411508134.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-07
AI Technical Summary
In data processing scenarios, traditional methods require manual extraction and quality inspection of data, resulting in large workload, low efficiency and poor accuracy.
The preset document is extracted through the data extraction model, automatically obtains the pending data, and iteratively updates the model based on the target data and position data to realize automatic data extraction and verification.
It improves data processing efficiency, reduces manual workload, enhances the accuracy and reliability of data extraction, and realizes a virtuous cycle of automatic data extraction-data verification-model optimization.
Smart Images
Figure CN119396915A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] With the development of big data and the explosion of multi-dimensional data, the market has higher and higher requirements for data quality. In some data processing scenarios, it is necessary to extract the required data from a large amount of multi-dimensional data, process the required data, and then push the processed data to relevant business parties.
[0003] In traditional technology, in order to ensure the accuracy and reliability of the pushed data, it is necessary to manually extract the data and perform quality checks on the extracted data to avoid the problem of poor data quality caused by incorrect extracted data. However, this method requires a large amount of manual work, is inefficient, and has a low accuracy rate. Summary of the invention
[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve efficiency and accuracy in response to the above technical problems.
[0005] In a first aspect, the present application provides a data processing method, comprising:
[0006] Acquire the data to be processed, wherein the data to be processed is obtained by extracting data from a preset document using a data extraction model, and the data extraction model matches the data type of the data to be processed;
[0007] In response to receiving a verification completion instruction, obtaining target data corresponding to the data to be processed and location data of the target data in the preset document;
[0008] The target data and the position data are input into the data extraction model, so that the data extraction model is iteratively updated according to the corresponding relationship between the target data and the position data to obtain an updated data extraction model.
[0009] In one embodiment, obtaining the data to be processed includes:
[0010] In response to receiving a data acquisition instruction, determining a target data type according to the data acquisition instruction;
[0011] Data whose message subject matches the target data type is obtained from the first message queue to obtain data to be processed.
[0012] In one embodiment, inputting the target data and the location data into the data extraction model comprises:
[0013] storing the target data and the position data in a second message queue, wherein the message subject of the target data is the same as the message subject of the data to be processed;
[0014] Determining, according to the message subject of the target data, a data extraction model matching the target data from candidate data extraction models;
[0015] The target data and the location data are input into the data extraction model.
[0016] In one embodiment, after obtaining the target data corresponding to the data to be processed, the method further includes:
[0017] Determining initial data on a target platform that matches the target data;
[0018] When the target data passes the review, the initial data is updated using the target data to obtain updated data.
[0019] In one embodiment, the method for determining the data extraction model includes:
[0020] Obtaining a training document sample set, wherein the training data sample set includes training documents marked with location tags;
[0021] Constructing an initial data extraction model, wherein the initial data extraction model is provided with training parameters;
[0022] Inputting the training data sample set into the initial data extraction model, and outputting the predicted position through the initial data extraction model;
[0023] Based on the difference between the predicted position and the corresponding position label, the training parameters are iteratively adjusted until the difference meets the preset conditions to obtain a data extraction model.
[0024] In a second aspect, the present application further provides a data processing device, comprising:
[0025] A first acquisition module is used to acquire data to be processed, wherein the data to be processed is obtained by extracting data from a preset document using a data extraction model, and the data extraction model matches the data type of the data to be processed;
[0026] A second acquisition module, configured to acquire, in response to receiving an instruction indicating that verification is completed, target data corresponding to the data to be processed and location data of the target data in the preset document;
[0027] An input module is used to input the target data and the position data into the data extraction model, so that the data extraction model is iteratively updated according to the corresponding relationship between the target data and the position data to obtain an updated data extraction model.
[0028] In one embodiment, the acquisition module is further used for:
[0029] In response to receiving a data acquisition instruction, determining a target data type according to the data acquisition instruction;
[0030] Data whose message subject matches the target data type is obtained from the first message queue to obtain data to be processed.
[0031] In one embodiment, the input module is further used for:
[0032] storing the target data and the position data in a second message queue, wherein the message subject of the target data is the same as the message subject of the data to be processed;
[0033] Determining, according to the message subject of the target data, a data extraction model matching the target data from candidate data extraction models;
[0034] The target data and the location data are input into the data extraction model.
[0035] In one embodiment, the device further comprises:
[0036] A determination submodule, used to determine initial data on a target platform that matches the target data;
[0037] The updating module is used to update the initial data with the target data to obtain updated data when the target data is approved.
[0038] In one embodiment, the device further comprises a module for determining the data extraction model, and the module for determining the data extraction model comprises:
[0039] An acquisition submodule is used to acquire a training document sample set, wherein the training data sample set includes training documents marked with position tags;
[0040] A construction module, used to construct an initial data extraction model, wherein the initial data extraction model is provided with training parameters;
[0041] An input submodule, used for inputting the training data sample set into the initial data extraction model, and outputting a predicted position through the initial data extraction model;
[0042] The adjustment module is used to iteratively adjust the training parameters based on the difference between the predicted position and the corresponding position label until the difference meets the preset conditions to obtain a data extraction model.
[0043] In a third aspect, an embodiment of the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the embodiments of the present disclosure are implemented.
[0044] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.
[0045] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.
[0046] The above-mentioned data processing method, device, computer equipment, computer-readable storage medium and computer program product obtain the data to be processed after extracting data from the preset document through the data extraction model, without manual data extraction, and automatically extracting the data based on the model, which effectively improves the data processing efficiency and reduces the manual workload; after receiving the instruction of verification completion, the target data corresponding to the data to be processed and the position data of the target data in the preset document are obtained, so that the target data can be verified while ensuring the data extraction efficiency, and the accuracy and reliability of the extracted data can be improved, and the problem of data extraction errors can be avoided. Further, the data extraction model is iteratively updated using the target data and the position data to obtain an updated data extraction model. In the process of data extraction verification, the model is continuously adjusted and optimized, which can improve the model extraction performance. In the process of data extraction, the model training samples are enriched, and a virtuous cycle of automatic data extraction-data verification-model optimization-automatic data extraction is realized. It is suitable for more application scenarios, taking into account the efficiency and accuracy of data extraction, and in the scenario of massive multi-dimensional data, the required target data can be quickly and accurately extracted. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 is a flow chart of a data processing method in one embodiment;
[0049] Figure 2 is a flow chart of a data processing method in another embodiment;
[0050] Figure 3 A schematic diagram of a flow chart of a method for determining a data extraction model in one embodiment;
[0051] Figure 4 is a flow chart of a data processing method in another embodiment;
[0052] Figure 5 is a structural block diagram of a data processing device in one embodiment;
[0053] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0055] In one embodiment, Figure 1 As shown, a data processing method is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0056] Step S110, obtaining data to be processed, wherein the data to be processed is obtained by extracting data from a preset document using a data extraction model, and the data extraction model matches the data type of the data to be processed;
[0057] Exemplarily, the data extraction model can be trained in advance according to the actual application scenario, and different business scenarios and data extraction requirements can correspond to different data extraction models. In one example, the data extraction model can be determined from the data extraction model library. In one example, in the data extraction process, the corresponding data extraction model can be determined according to the actual business scenario, data extraction requirements, and the type of document to be extracted.
[0058] In one example, the preset document may be a document determined according to an actual application scenario for information extraction. For example, the preset document may include but is not limited to judicial documents, government announcements, etc. The data extraction model matches the data type of the data to be processed. For example, if the data type of the data to be processed is business operation information, the data extraction model may be a model for extracting business operation information.
[0059] Optionally, after extracting data from a preset document through a data extraction model, initial data is obtained, and after further screening the initial data, data to be processed is obtained. In one example, the initial data can be judged by preset rules to determine that data that may contain errors is data to be processed; in another example, the initial data can be allocated to the corresponding business department for use, and in response to a data warning message returned by the business department, the data that may contain errors is determined to be data to be processed.
[0060] Step S120, in response to receiving the instruction of verification completion, obtaining the target data corresponding to the data to be processed and the location data of the target data in the preset document;
[0061] Exemplarily, after the data to be processed is obtained, the data can be verified by a verification department or a verification model. After the verification is completed, the verification end, such as the verification department, the verification model, etc., sends a verification completion instruction.
[0062] In one example, during the verification process, it is determined whether there is an extraction error in the data to be processed based on the data to be processed and the preset document. If so, the data to be processed is modified to obtain the target data, and the location data of the target data in the preset document is marked.
[0063] In an example, the position data may include, but is not limited to, a matrix position identifier, which may be determined based on an actual application scenario.
[0064] Step S130: input the target data and the position data into the data extraction model, so that the data extraction model is iteratively updated according to the corresponding relationship between the target data and the position data to obtain an updated data extraction model.
[0065] Exemplarily, the target data and the position data are input into the data extraction model, so that the data extraction model is iteratively updated according to the corresponding relationship between the target data and the position data.
[0066] In one example, the target data and the location data may be input as training samples into the data extraction model, so that the data extraction model is updated and optimized according to the verified data.
[0067] In one example, a preset document may also be input into the data extraction model so that the data extraction model can be trained according to the document content and location data, thereby accurately extracting the target data.
[0068] Optionally, the updated data extraction model continues to be used to extract data from the document.
[0069] In the disclosed embodiment, after extracting data from a preset document through a data extraction model, the data to be processed is obtained. There is no need for manual data extraction, and preliminary data extraction is automatically performed based on the model, which effectively improves the data processing efficiency and reduces the manual workload; after receiving the instruction of verification completion, the target data corresponding to the data to be processed and the position data of the target data in the preset document are obtained, so that the target data can be verified while ensuring the data extraction efficiency, improving the accuracy and reliability of the extracted data, and avoiding the problem of data extraction errors. Further, the data extraction model is iteratively updated using the target data and the position data to obtain an updated data extraction model. In the process of data extraction verification, the model is continuously adjusted and optimized, which can improve the model extraction performance. In the process of data extraction, the model training samples are enriched, and a virtuous cycle of automatic data extraction-data verification-model optimization-automatic data extraction is realized. It is suitable for more application scenarios, taking into account the efficiency and accuracy of data extraction, and in the scenario of massive multi-dimensional data, the required target data can be quickly and accurately extracted.
[0070] In one embodiment, Figure 2 As shown, the obtaining of the data to be processed includes:
[0071] Step S111, in response to receiving a data acquisition instruction, determining a target data type according to the data acquisition instruction;
[0072] Step S112, obtaining data whose message subject matches the target data type from the first message queue to obtain data to be processed.
[0073] Exemplarily, the data acquisition instruction may be actively triggered or passively triggered, which may be determined according to the actual application scenario. For example, the data acquisition instruction may be sent through external operation, or may be periodically automatically triggered.
[0074] In one example, the data acquisition instruction may include data type information of the data to be acquired, such as a data type identifier, and the target data type, that is, the data type of the data to be acquired, is determined according to the data acquisition instruction.
[0075] Optionally, the relevant data is stored in the first message queue. The first message queue may store data extracted from different or the same documents by different data extraction models, and different data types correspond to different message topics.
[0076] According to the correspondence between the data type and the message subject, the target message subject corresponding to the target data type is determined, and the data of the target message subject is obtained from the first message queue to obtain the data to be processed.
[0077] In the embodiment of the present disclosure, when a data acquisition instruction is received, the target data type is determined according to the instruction, and the data to be processed is obtained from the first message queue according to the message subject and the target data type, so that the corresponding data to be processed can be automatically acquired and consumed from the message queue according to the data type. The data is stored and consumed through the first message queue, and the data processing steps in different links of the data processing flow are connected. There is no need to share specific data information between different data processing steps, and a cross-departmental data processing flow can be realized, which is suitable for more application scenarios.
[0078] In one embodiment, inputting the target data and the location data into the data extraction model comprises:
[0079] storing the target data and the position data in a second message queue, wherein the message subject of the target data is the same as the message subject of the data to be processed;
[0080] Determining, according to the message subject of the target data, a data extraction model matching the target data from candidate data extraction models;
[0081] The target data and the location data are input into the data extraction model.
[0082] Exemplarily, the data verification step and the model update step can be implemented in different environments and different data processing stages. Therefore, the two steps are isolated from each other, and the data information is not interoperable. A data bridge can be built through the second message queue.
[0083] Optionally, the target data and the location data are stored in the second message queue, and the message subject corresponding to the target data is the message subject of the corresponding data to be processed. The second message queue is used to store the verified data, and different data correspond to different message subjects.
[0084] In the model updating step, according to the message subject of the target data, a data extraction model matching the target data can be determined from the candidate data extraction models, and the target data and the location data can be input into the data extraction model. In one example, after the data is stored in the second message queue, the data extraction model can automatically select the matching data according to the message subject, and automatically update the model according to the matched data.
[0085] In the embodiment of the present disclosure, when inputting the data extraction model, the target data and the location data are first stored in the second message queue, and then the matching data extraction model is determined according to the message subject of the target data, and the data is input, thereby enabling cross-departmental and cross-step collaboration between the two data processing steps of data verification and model optimization, which is suitable for more application scenarios.
[0086] In one embodiment, after obtaining the target data corresponding to the data to be processed, the method further includes:
[0087] Determining initial data on a target platform that matches the target data;
[0088] When the target data passes the review, the initial data is updated using the target data to obtain updated data.
[0089] Exemplarily, the extracted and verified data can be used to update the data on the target platform. The target platform can be a platform for data display and data query determined according to the actual application scenario. Users can obtain the required data through the target platform.
[0090] Determine the initial data on the target platform that matches the target data. The target data and the initial data have the same attributes. For example, if the target data is business operation information, the initial data may also be business operation information on the target platform.
[0091] In one example, the target data needs to be audited, and the audit can be performed by the audit department or the audit model, which can be determined according to the actual application scenario. If the target data passes the audit, it can be considered that the target data is confirmed to be correct and can be put on the online platform.
[0092] Optionally, when the target data is used to update the initial data, the target data and the initial data may be compared to determine that the data with differences is the data to be updated, and the corresponding target data may be used to replace the data to be updated to obtain updated data.
[0093] In the disclosed embodiment, after acquiring the target data, the corresponding initial data on the target platform is determined, and after the data is reviewed and approved, the target data is used to update the initial data to obtain updated data, so that the data on the target platform can be updated according to the target data, thereby ensuring the validity and real-time nature of the data on the target platform.
[0094] In one embodiment, Figure 3 As shown, the method for determining the data extraction model includes:
[0095] Step S310, obtaining a training document sample set, wherein the training data sample set includes training documents marked with location tags;
[0096] Step S320, constructing an initial data extraction model, wherein the initial data extraction model is provided with training parameters;
[0097] Step S330, inputting the training data sample set into the initial data extraction model, and outputting the predicted position through the initial data extraction model;
[0098] Step S340, based on the difference between the predicted position and the corresponding position label, iteratively adjust the training parameters until the difference meets the preset conditions to obtain a data extraction model.
[0099] Exemplarily, the data extraction model is trained based on a training document sample set. The training document sample set includes training documents annotated with location tags, wherein the training documents can be obtained through historical data.
[0100] An initial data extraction model is constructed, and training parameters are set in the initial data extraction model. In an example, the model algorithm based on the initial data extraction model may include but is not limited to a named entity recognition algorithm, a support vector machine algorithm, a neural network algorithm, etc., which can be determined based on the actual application scenario.
[0101] The training data sample set is input into the initial data extraction model, and the predicted position is output by the initial data extraction model. Since the initial data extraction model is an untrained model at this time, the output accuracy of the model is poor. Therefore, the training parameters can be iteratively adjusted according to the difference between the predicted position and the corresponding position label.
[0102] When the difference between the predicted position and the corresponding position label meets the preset conditions, it can be considered that the model output accuracy is within the error tolerance range, the model can be used for data extraction, and the model is determined to be a data extraction model. In one example, the preset conditions can be set according to actual application scenarios.
[0103] In the disclosed embodiment, when determining the data extraction model, a training document sample set is first obtained, an initial data extraction model is constructed, and the sample set is input into the initial data extraction model. The initial data extraction model outputs a predicted position, and the training parameters are iteratively adjusted according to the difference between the predicted position and the corresponding position label until the difference meets the preset conditions, thereby obtaining a data extraction model. In this way, the data extraction model can be trained quickly and accurately, providing a model basis for the subsequent automatic extraction of data.
[0104] Figure 4 is a flow chart of a data processing method according to an exemplary embodiment, referring to Figure 4 As shown, the data to be processed enters the big data platform, and after verification, the target data and location data are obtained, which are submitted for review. After the review is passed, the target data and location data are pushed to the second message queue, and the data extraction model consumes the data in the second message queue and updates the model. In one example, the reviewed data is also used for data cleaning and then put online to take effect, and the initial data on the target platform is updated.
[0105] Through this embodiment, the quality of the extracted data can be improved, and the efficiency and accuracy of data extraction can be improved, the manual workload and labor costs can be reduced, and the timeliness of data extraction can be guaranteed in scenarios with large amounts of multi-dimensional data, making it suitable for more application scenarios.
[0106] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0107] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.
[0108] In an exemplary embodiment, Figure 5As shown, a data processing device 500 is provided, comprising:
[0109] A first acquisition module 510 is used to acquire data to be processed, wherein the data to be processed is obtained by extracting data from a preset document using a data extraction model, and the data extraction model matches the data type of the data to be processed;
[0110] A second acquisition module 520 is used to acquire target data corresponding to the data to be processed and position data of the target data in the preset document in response to receiving the instruction of verification completion;
[0111] The input module 530 is used to input the target data and the position data into the data extraction model, so that the data extraction model is iteratively updated according to the corresponding relationship between the target data and the position data to obtain an updated data extraction model.
[0112] In one embodiment, the acquisition module is further used for:
[0113] In response to receiving a data acquisition instruction, determining a target data type according to the data acquisition instruction;
[0114] Data whose message subject matches the target data type is obtained from the first message queue to obtain data to be processed.
[0115] In one embodiment, the input module is further used for:
[0116] storing the target data and the position data in a second message queue, wherein the message subject of the target data is the same as the message subject of the data to be processed;
[0117] Determining, according to the message subject of the target data, a data extraction model matching the target data from candidate data extraction models;
[0118] The target data and the location data are input into the data extraction model.
[0119] In one embodiment, the device further comprises:
[0120] A determination submodule, used to determine initial data on a target platform that matches the target data;
[0121] The updating module is used to update the initial data with the target data to obtain updated data when the target data is approved.
[0122] In one embodiment, the device further comprises a module for determining the data extraction model, and the module for determining the data extraction model comprises:
[0123] An acquisition submodule is used to acquire a training document sample set, wherein the training data sample set includes training documents marked with position tags;
[0124] A construction module, used to construct an initial data extraction model, wherein the initial data extraction model is provided with training parameters;
[0125] An input submodule, used for inputting the training data sample set into the initial data extraction model, and outputting a predicted position through the initial data extraction model;
[0126] The adjustment module is used to iteratively adjust the training parameters based on the difference between the predicted position and the corresponding position label until the difference meets the preset conditions to obtain a data extraction model.
[0127] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0128] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be processed and other data involved in the method described in this embodiment. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.
[0129] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0130] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0132] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0134] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0136] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data processing method, characterized in that: The method comprises: Acquire the data to be processed, wherein the data to be processed is obtained by extracting data from a preset document using a data extraction model, and the data extraction model matches the data type of the data to be processed; In response to receiving a verification completion instruction, obtaining target data corresponding to the data to be processed and location data of the target data in the preset document; The target data and the position data are input into the data extraction model, so that the data extraction model is iteratively updated according to the corresponding relationship between the target data and the position data to obtain an updated data extraction model.
2. The method according to claim 1, characterized in that: The obtaining of the data to be processed comprises: In response to receiving a data acquisition instruction, determining a target data type according to the data acquisition instruction; Data whose message subject matches the target data type is obtained from the first message queue to obtain data to be processed.
3. The method according to claim 2, characterized in that The inputting the target data and the location data into the data extraction model comprises: storing the target data and the position data in a second message queue, wherein the message subject of the target data is the same as the message subject of the data to be processed; Determining, according to the message subject of the target data, a data extraction model matching the target data from candidate data extraction models; The target data and the location data are input into the data extraction model.
4. The method according to claim 1, characterized in that After obtaining the target data corresponding to the data to be processed, the method further includes: Determining initial data on a target platform that matches the target data; When the target data passes the review, the initial data is updated using the target data to obtain updated data.
5. The method according to claim 1, characterized in that: The method for determining the data extraction model includes: Obtaining a training document sample set, wherein the training data sample set includes training documents marked with location tags; Constructing an initial data extraction model, wherein the initial data extraction model is provided with training parameters; Inputting the training data sample set into the initial data extraction model, and outputting the predicted position through the initial data extraction model; Based on the difference between the predicted position and the corresponding position label, the training parameters are iteratively adjusted until the difference meets the preset conditions to obtain a data extraction model.
6. A data processing device, characterized in that: The device comprises: A first acquisition module is used to acquire data to be processed, wherein the data to be processed is obtained by extracting data from a preset document using a data extraction model, and the data extraction model matches the data type of the data to be processed; A second acquisition module, configured to acquire, in response to receiving an instruction indicating that verification is completed, target data corresponding to the data to be processed and location data of the target data in the preset document; An input module is used to input the target data and the position data into the data extraction model, so that the data extraction model is iteratively updated according to the corresponding relationship between the target data and the position data to obtain an updated data extraction model.
7. The device according to claim 6, characterized in that The acquisition module is also used for: In response to receiving a data acquisition instruction, determining a target data type according to the data acquisition instruction; Data whose message subject matches the target data type is obtained from the first message queue to obtain data to be processed.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.