Data processing method and device, computer readable storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a computer readable storage medium and electronic equipment. The method comprises the following steps: detecting whether target file information is written into a target data node of a target storage system or not; when it is detected that the target file information is written into the target data node, obtaining the target file information; on the basis of the target file information and the file information of the multiple fragment files, the integrity of the target file is detected, a detection result is obtained, a target processing mode is determined on the basis of the detection result, and the target processing mode is used for guiding the target application to send the target file to a next processing system or guiding the target application to send the target file to the next processing system. And guiding the target application to prohibit sending the target file to the next processing system. Through the file processing method and device, the problem that the file accuracy is poor due to the fact that the combined file is directly sent to the downstream for further processing after the application combines the fragmented files in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of distributed systems, and more specifically, to a data processing method, device, computer-readable storage medium, and electronic device. Background Art

[0002] In distributed systems that need to process large amounts of data (e.g., distributed applications), in order to improve processing efficiency and reduce the load on a single node, data is usually split into multiple shard files (or set files) and stored in different storage areas. When these data need to be sent to other systems, they are merged into a complete file and then sent to downstream systems, such as data warehouses, data analysis platforms, etc., for subsequent data analysis and business processing.

[0003] At present, in the related technology, after merging the fragmented files, the merged file is directly sent to the downstream for further processing. However, the merged file may have anomalies, which may lead to the problem of poor file accuracy.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The main purpose of the present application is to provide a data processing method, device, computer-readable storage medium and electronic device to solve the problem in the related art that after merging the fragmented files, the merged files are directly sent to the downstream for further processing, resulting in poor file accuracy.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data processing method is provided. The method includes: detecting whether target file information is written in a target data node of a target storage system, wherein the target storage system stores data based on multiple data nodes, the target data node belongs to multiple data nodes, the target file information refers to the file information of the target file obtained after the target application merges multiple shard files, and the target file information is written to the target data node after the target application merges multiple shard files; in the case of detecting that the target file information is written in the target data node, the target file information is obtained; based on the target file information and the file information of multiple shard files, the integrity of the target file is detected to obtain a detection result, and a target processing method is determined based on the detection result, wherein the target processing method is used to guide the target application to send the target file to the next processing system, or to guide the target application to prohibit sending the target file to the next processing system.

[0007] Furthermore, the data processing method also includes: extracting the job name to which the target file belongs from the target file information; obtaining a first matching relationship between multiple job names and multiple file verification methods, and determining the target file verification method from multiple file verification methods based on the first matching relationship and the job name to which the target file belongs; according to the target file verification method, based on the target file information and the file information of multiple segmented files, detecting the integrity of the target file to obtain a detection result.

[0008] Furthermore, the data processing method also includes: when the target file verification method includes a row verification method, extracting the number of rows of the target file from the target file information, and extracting the number of rows of the slice file from the file information of each slice file; based on the number of rows of the target file and the number of rows of multiple slice files, detecting the integrity of the target file; when the target file verification method includes a content verification method, extracting the storage address information of the target file from the target file information, and extracting the storage address information of the slice file from the file information of each slice file; obtaining the target file according to the storage address information of the target file, and obtaining multiple slice files according to the storage address information of multiple slice files; detecting the integrity of the target file according to the content of the target file and the content of the multiple slice files.

[0009] Furthermore, the data processing method also includes: extracting the job name to which the target file belongs from the target file information; obtaining a second matching relationship between multiple job names and multiple result processing configurations, and determining a target result processing configuration from multiple result processing configurations based on the second matching relationship and the job name to which the target file belongs; and determining a target processing method based on the target result processing configuration and the detection result.

[0010] Furthermore, the data processing method also includes: determining a target executor to be subjected to integrity detection from a plurality of executors through a monitoring module in the target processing system, wherein the plurality of executors belong to a data processing module in the target processing system; importing executor information of the target executor into a first data node in the target storage system through the monitoring module; controlling the target executor through the data processing module to detect the integrity of the target file based on the target file information and the file information of the plurality of shard files according to the executor information in the first data node, and obtaining a detection result.

[0011] Furthermore, the data processing method also includes: after determining the target processing method based on the detection result, updating the target file information in the target data node based on the target processing method, and sending the target information to the target application based on the target processing method, wherein the target information is used to indicate whether the target application is allowed to send the target file to the next processing system.

[0012] Furthermore, the data processing method further includes: updating the to-be-verified flag in the target file information to a verification success flag, or updating the to-be-verified flag in the target file information to a verification failure flag, or deleting the target file information.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data processing device is provided. The device includes: a detection module, which is used to detect whether the target file information is written in the target data node of the target storage system, wherein the target storage system stores data based on multiple data nodes, the target data node belongs to multiple data nodes, and the target file information refers to the file information of the target file obtained after the target application merges multiple shard files, and the target file information is written to the target data node after the target application merges multiple shard files; an acquisition module, which is used to acquire the target file information when it is detected that the target file information is written in the target data node; a determination module, which is used to detect the integrity of the target file based on the target file information and the file information of multiple shard files, obtain the detection result, and determine the target processing method based on the detection result, wherein the target processing method is used to guide the target application to send the target file to the next processing system, or guide the target application to prohibit sending the target file to the next processing system.

[0014] Furthermore, the determination module also includes: an extraction sub-module, used to extract the job name to which the target file belongs from the target file information; an acquisition sub-module, used to obtain a first matching relationship between multiple job names and multiple file verification methods, and determine the target file verification method from multiple file verification methods based on the first matching relationship and the job name to which the target file belongs; a first detection sub-module, used to detect the integrity of the target file according to the target file verification method, based on the target file information and the file information of multiple segment files, and obtain a detection result.

[0015] Furthermore, the first detection submodule also includes: a first extraction unit, which is used to extract the number of lines of the target file from the target file information and the number of lines of the slice file from the file information of each slice file when the target file verification method includes a line verification method; a first detection unit, which is used to detect the integrity of the target file based on the number of lines of the target file and the number of lines of multiple slice files; a second extraction unit, which is used to extract the storage address information of the target file from the target file information and the storage address information of the slice file from the file information of each slice file when the target file verification method includes a content verification method; a first acquisition unit, which is used to acquire the target file according to the storage address information of the target file and acquire multiple slice files according to the storage address information of multiple slice files; a second detection unit, which is used to detect the integrity of the target file according to the content of the target file and the content of the multiple slice files.

[0016] Furthermore, the determination module also includes: a third extraction unit, used to extract the job name to which the target file belongs from the target file information; a first acquisition unit, used to obtain a second matching relationship between multiple job names and multiple result processing configurations, and determine the target result processing configuration from the multiple result processing configurations based on the second matching relationship and the job name to which the target file belongs; a determination unit, used to determine the target processing method based on the target result processing configuration and the detection result.

[0017] Furthermore, the determination module also includes: a determination sub-module, used to determine the target executor to be subjected to integrity detection from multiple executors through the monitoring module in the target processing system, wherein the multiple executors belong to the data processing module in the target processing system; a processing sub-module, used to import the executor information of the target executor into the first data node in the target storage system through the monitoring module; and a second detection sub-module, used to control the target executor through the data processing module according to the executor information in the first data node, to detect the integrity of the target file based on the target file information and the file information of multiple shard files, and obtain a detection result.

[0018] Furthermore, the data processing device also includes: a processing module, which is used to update the target file information in the target data node based on the target processing method, and send the target information to the target application based on the target processing method, wherein the target information is used to indicate whether the target application is allowed to send the target file to the next processing system.

[0019] Furthermore, the processing module also includes: a first update module, used to update the to-be-verified flag in the target file information to a verification success flag, or a second update module, used to update the to-be-verified flag in the target file information to a verification failure flag, or a deletion module, used to delete the target file information.

[0020] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer-readable storage medium is provided, and the computer-readable storage medium includes a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned data processing method.

[0021] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an electronic device is provided, the electronic device comprising a memory storing an executable program; and a processor for running the program, wherein the above-mentioned data processing method is executed when the program is running.

[0022] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a computer program product is provided, including computer instructions, which implement the steps of the above-mentioned data processing method when executed by a processor.

[0023] In an embodiment of the present application, by detecting whether the target file information is written in the target data node of the target storage system, and setting the target file information to be written to the target data node after the target application merges multiple shard files, effective detection of whether the target application merges to obtain a new file is achieved. By detecting that the target file information is written in the target data node, the integrity of the target file is detected based on the target file information and the file information of multiple shard files to obtain a detection result. When the target application merges to obtain a new file, effective verification of the integrity of the merged file is achieved. By determining the target processing method based on the detection result, it is achieved to determine whether the target application is allowed to send the merged file to the downstream based on the integrity of the merged file, thereby avoiding sending wrong files to the downstream and causing production problems, thereby effectively improving the accuracy of the file.

[0024] It can be seen that the solution provided by the present application achieves the purpose of performing integrity detection on the file obtained by merging the application, thereby realizing the technical effect of improving the accuracy of the file, and further solving the technical problem in the related technology that after the application merges the fragmented files, the merged file is directly sent to the downstream for further processing, resulting in poor file accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 is a hardware structure block diagram of a computer terminal provided according to an embodiment of the present application;

[0027] Figure 2 is a flow chart of a data processing method provided according to an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of a target processing system provided according to an embodiment of the present application;

[0029] Figure 4 It is a working diagram of the monitoring module and the data processing module provided according to an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of a data processing method provided according to an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of a data processing device provided according to an embodiment of the present application;

[0032] Figure 7 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] It should be noted that the collected information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0036] Example 1

[0037] According to an embodiment of the present application, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0039] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0043] Under the above operating environment, this application provides Figure 2 The data processing method shown. Figure 2 is a flow chart of a data processing method provided according to an embodiment of the present application, such as Figure 2 As shown, the method is applied to a target processing system and comprises the following steps:

[0044] Step S201, detect whether the target file information is written in the target data node of the target storage system, wherein the target storage system stores data based on multiple data nodes, the target data node belongs to multiple data nodes, and the target file information refers to the file information of the target file obtained after the target application merges multiple shard files, and the target file information is written to the target data node after the target application merges multiple shard files.

[0045] In some embodiments, Figure 3 is a schematic diagram of a target processing system provided according to an embodiment of the present application, such as Figure 3 As shown, the target processing system may include an input module, a monitoring module, a data processing module, a result processing module, and a configuration module. The input module is connected to the target storage system, the monitoring module is used to monitor the target storage system, the configuration module is connected to the monitoring module, the data processing module, and the result processing module, and the data processing module is connected to the result processing module.

[0046] In some embodiments, the type of target application and the content of the shard file may vary depending on the actual application scenario. For example, the target application may be an application of a financial institution, and the content of the shard file may be customer information of customers in the financial institution.

[0047] In some embodiments, the target storage system adopts a distributed service framework. The target storage system stores all data in memory and provides data storage based on a directory node tree similar to a file system. The data model is a tree, which includes multiple data nodes. Each data node stores its own data content and a series of attribute information. Each data node has a unique path.

[0048] In some embodiments, the target storage system is used to store file information of multiple slice files of the target application, and file information of the target file obtained after the target application merges multiple slice files. For example, the target storage system includes at least multiple file generation nodes and at least one file merging node, the file generation node is used to store file information of multiple slice files, and the file merging node (equivalent to the above-mentioned target data node) is used to store file information of the target file. In addition, an event listener can be registered on the file merging node to monitor data changes in the file merging node.

[0049] For example, the business processing layer of the target application includes N shard executors. The N shard executors receive instructions for a file generation job, execute corresponding programs, and shard the original files specified in the job to obtain N files (i.e., shard files). Optionally, if each file obtained by sharding needs to be further split into M files, the total number of files (i.e., shard files) is N*M, where N and M are both positive integers greater than 1.

[0050] Optionally, after obtaining a plurality of slice files, the target application may store the plurality of slice files in different storage areas and generate file information of each slice file. Then the file information of the slice files is sent to an input module in the target processing system so as to write the file information of the slice files into the target storage system through the input module. Optionally, the file information of the slice files may include the job name of the job to which the slice files belong, the job session number of the job to which the slice files belong, the file name of the slice files, the file path, the file generation date, the file type, the number of file lines, etc., wherein the job name and the job session number of the plurality of slice files obtained in the same job and the same batch are the same, and the file name, the file path, the file generation date, the file type, and the number of file lines are not completely the same. For example, the job name and the job session number of the above-mentioned N slice files (or N*M allocation files) are the same, and the file name, the file path, the file generation date, the file type, and the number of file lines are not completely the same.

[0051] Optionally, after obtaining the file information of multiple shard files, the input module creates multiple file generation nodes in the target storage system, and stores the file information of different shard files through different file generation nodes. For example, if there are N shard files, N file generation nodes are created.

[0052] Optionally, when the target application needs to send the content of the above original file to other systems (e.g., applications other than the target application), the target application can execute a file merging operation program to merge multiple shard files to obtain a target file, generate file information of the target file, and then send the file information of the target file to the input module in the target processing system to write the file information of the target file into the target storage system through the input module. If the merging operation is performed normally, the target file is the same as the above original file, and if the merging operation is not performed normally, the target file is different from the above original file. Optionally, the file merging operation is performed in the public layer of the target application, and the number of target files obtained after executing the file merging operation is 1. In some embodiments, if the shard executor in the business processing layer further splits the file obtained by sharding, that is, there are N*M shard files, then the business processing layer of the target application will also perform a file merging operation to merge the N*M shard files into N shard files. Optionally, the file information of the target file may include the job name of the job to which the target file belongs, the job session number of the job to which the target file belongs, the file name of the target file, the file path, the file generation date, the file type, the number of file lines, etc.

[0053] Optionally, after obtaining the file information of the target file, the input module creates a file merging node in the target storage system, and stores the file information of the target file through the file merging node.

[0054] Optionally, the monitoring module in the target processing system may monitor data changes in the file merging node based on an event listener, that is, monitor whether the target file information is written into the file merging node.

[0055] Step S202: when it is detected that the target file information is written into the target data node, the target file information is obtained.

[0056] Optionally, when the target processing system detects that the target file information is written into the target data node, the target processing system may obtain the target file information from the target data node.

[0057] Step S203, based on the target file information and the file information of multiple shard files, detect the integrity of the target file, obtain the detection result, and determine the target processing method based on the detection result, wherein the target processing method is used to guide the target application to send the target file to the next processing system, or to guide the target application to prohibit sending the target file to the next processing system.

[0058] Optionally, the target processing system may obtain file information of each shard file from the file generation node, for example, find the file information of the corresponding shard file from the target storage system according to the job name and job session number in the target file information.

[0059] Optionally, after obtaining the target file information and the file information of the multiple shard files, the target processing system can detect the integrity of the target file based on the target file information and the file information of the multiple shard files to obtain a detection result. For example, it is determined whether the number of file lines in the target file information is the same as the sum of the number of file lines in the file information of the multiple shard files, and the detection result is determined based on the determination result. For another example, the target file is obtained according to the file path in the target file information, and the multiple shard files are obtained according to the file path in the file information of the multiple shard files, and then the contents of the target file and the multiple shard files are compared, and the detection result is determined based on the comparison result. For another example, the target processing system can determine the detection result in combination with the aforementioned determination result and comparison result.

[0060] In some embodiments, the detection result may be one of the following: detection passed, detection failed, detection abnormality, detection failure, etc.

[0061] Optionally, after obtaining the detection result, the target processing system can determine the target processing method according to the detection result. For example, the target processing system stores the correspondence between multiple detection results and multiple processing methods, and the target processing system can determine the target processing method corresponding to it from multiple processing methods according to the detection result, and then execute the target processing method. For example, in the case where the detection result is that the detection is passed, a message of detection passed is sent to the target processing system to guide the target application to send the target file to the next processing system. In the case where the detection result is that the detection is not passed, a message of detection not passed is sent to the target processing system to guide the target application to prohibit the target file from being sent to the next processing system. In the case where the detection result is that the detection is abnormal, a message of detection abnormality is sent to the target processing system to guide the target application to prohibit the target file from being sent to the next processing system, and re-merge the files. In the case where the detection result is that the detection fails, a message of detection failure is sent to the target processing system to guide the target application to prohibit the target file from being sent to the next processing system, and generate an early warning message.

[0062] In some embodiments, the above-mentioned next processing system refers to a system for processing the target file after the target application, and the next processing system may be a data warehouse, a data analysis platform, etc.

[0063] In an embodiment of the present application, by detecting whether the target file information is written in the target data node of the target storage system, and setting the target file information to be written to the target data node after the target application merges multiple shard files, effective detection of whether the target application merges to obtain a new file is achieved. By detecting that the target file information is written in the target data node, the integrity of the target file is detected based on the target file information and the file information of multiple shard files to obtain a detection result. When the target application merges to obtain a new file, effective verification of the integrity of the merged file is achieved. By determining the target processing method based on the detection result, it is achieved to determine whether the target application is allowed to send the merged file to the downstream based on the integrity of the merged file, thereby avoiding sending wrong files to the downstream and causing production problems, thereby effectively improving the accuracy of the file.

[0064] It can be seen that the solution provided by the present application achieves the purpose of performing integrity detection on the file obtained by merging the application, thereby realizing the technical effect of improving the accuracy of the file, and further solving the technical problem in the related technology that after the application merges the fragmented files, the merged file is directly sent to the downstream for further processing, resulting in poor file accuracy.

[0065] Optionally, in the data processing method provided in the embodiment of the present application, based on the target file information and the file information of multiple shard files, the integrity of the target file is detected to obtain a detection result, including: extracting the job name to which the target file belongs from the target file information; obtaining a first matching relationship between multiple job names and multiple file verification methods, and determining the target file verification method from multiple file verification methods based on the first matching relationship and the job name to which the target file belongs; according to the target file verification method, based on the target file information and the file information of multiple shard files, the integrity of the target file is detected to obtain a detection result.

[0066] Optionally, the monitoring module in the target processing system can obtain the target file information from the target data node when detecting that the target file information is written in the target data node, and then extract the job name to which the target file belongs from the target file information. Afterwards, the target monitoring module can send the job name to the configuration module in the target processing system.

[0067] Optionally, the configuration module stores a first matching relationship between multiple job names and multiple file verification methods. After receiving the job name in the target file information, the configuration module can determine the target file verification method that matches the job name from the multiple file verification methods based on the first matching relationship and the job name to which the target file belongs. The file verification method corresponding to each job name can be preset by the staff.

[0068] Optionally, after determining the target file verification method, the configuration module may determine the program identifier of the verification program adopted by the target file verification method, and then send the program identifier of the verification program corresponding to the target file verification method to the data processing module in the target processing system.

[0069] Optionally, the data processing module provides a unified data verification interface and nested methods, and a complete verification model is formed by multiple applications that implement the interface. The data processing module can execute the interface implementation programs in sequence according to the configuration order, and the output of the previous implementation program is the input of the next implementation program.

[0070] For example, the verification procedure includes:

[0071] The file size check corresponds to the processing program A1 of the interface I implemented;

[0072] The file line number check corresponds to the processing program A2 of the interface I implemented;

[0073] The file content verification corresponds to the processing procedure A3 of the interface I implemented.

[0074] Optionally, after receiving the program identifier of the verification program sent by the configuration module, the data processing module can assemble the corresponding verification program according to the program identifier, and then execute it in the order of assembly. For example, if the program identifier sent by the configuration module is A2A1, the data processing module assembles the A2 and A1 programs, and calls A2 first and then A1. Optionally, after the data processing system completes the program corresponding to the aforementioned program identifier according to the call, it determines that the verification is completed according to the target file verification method and obtains the detection result.

[0075] It should be noted that, through the above method, a file verification method based on job name matching is implemented to perform targeted detection on the integrity of the target file, thereby improving the accuracy of the detection result.

[0076] Optionally, in the data processing method provided in the embodiment of the present application, according to the target file verification method, based on the target file information and the file information of multiple shard files, the integrity of the target file is detected, including: when the target file verification method includes a row verification method, extracting the number of rows of the target file from the target file information, and extracting the number of rows of the shard file from the file information of each shard file; based on the number of rows of the target file and the number of rows of the multiple shard files, detecting the integrity of the target file; when the target file verification method includes a content verification method, extracting the storage address information of the target file from the target file information, and extracting the storage address information of the shard file from the file information of each shard file; obtaining the target file according to the storage address information of the target file, and obtaining multiple shard files according to the storage address information of the multiple shard files; detecting the integrity of the target file according to the content of the target file and the content of the multiple shard files.

[0077] In some embodiments, the target file verification method includes at least one of the following: a line verification method and a content verification method.

[0078] Optionally, when the target file verification method includes a row verification method, the number of rows of the target file is extracted from the target file information, and the number of rows of each shard file is extracted from the file information of the shard file. After that, the target processing system can calculate the sum of the number of rows of multiple shard files, and then determine whether the sum of the number of rows of multiple shard files is the same as the number of rows of the target file to detect the integrity of the target file.

[0079] Optionally, when the target file verification method includes a content verification method, the storage address information of the target file is extracted from the target file information, and the storage address information of each fragment file is extracted from the file information of each fragment file, wherein the storage address information is equivalent to the above-mentioned file path. Afterwards, the target processing system can determine whether the content of the target file includes the content of each fragment file to detect the integrity of the target file.

[0080] Optionally, when the target file verification method only includes a row verification method, if the sum of the number of rows of multiple shard files is the same as the number of rows of the target file, it is determined that the target file detection has passed; if the sum of the number of rows of multiple shard files is different from the number of rows of the target file, it is determined that the target file detection has failed.

[0081] Optionally, when the target file verification method only includes the content verification method, if the content of the target file includes the content of each segment file, it is determined that the target file detection has passed; if the content of the target file does not include the content of each segment file, it is determined that the target file detection has failed.

[0082] Optionally, when the target file verification method includes a line verification method and a content verification method, if the sum of the number of lines of multiple segment files is the same as the number of lines of the target file, and the content of the target file includes the content of each segment file, then it is determined that the target file detection has passed; if the sum of the number of lines of multiple segment files is different from the number of lines of the target file, or the content of the target file does not include the content of each segment file, then it is determined that the detection has failed.

[0083] In some embodiments, when the information obtained during detection is abnormal (eg, not garbled), it is determined that there is an abnormality in the detection; and when the detection information cannot be obtained, it is determined that the detection has failed.

[0084] It should be noted that, through the above method, the segment file and the target file are compared to achieve integrity detection, thereby improving the accuracy of the detection result.

[0085] Optionally, in the data processing method provided in the embodiment of the present application, the target processing mode is determined based on the detection result, including: extracting the job name to which the target file belongs from the target file information; obtaining a second matching relationship between multiple job names and multiple result processing configurations, and determining the target result processing configuration from the multiple result processing configurations based on the second matching relationship and the job name to which the target file belongs; determining the target processing mode based on the target result processing configuration and the detection result.

[0086] Optionally, the monitoring module in the target processing system can obtain the target file information from the target data node when detecting that the target file information is written in the target data node, and then extract the job name to which the target file belongs from the target file information. Afterwards, the target monitoring module can send the job name to the result processing module in the target processing system.

[0087] Optionally, the result processing module stores a second matching relationship between multiple job names and multiple result processing configurations. After receiving the job name in the target file information, the result processing module can determine the target result processing configuration that matches the job name from the multiple result processing configurations based on the second matching relationship and the job name to which the target file belongs. The target result processing configuration corresponding to each job name can be preset by the staff, and each result processing configuration includes a corresponding relationship between multiple detection results and multiple processing methods, and the processing methods in different result processing configurations are not completely the same.

[0088] Optionally, after obtaining the detection result, the data processing module may send the detection result to the result processing module, so that the result processing module determines the target processing method based on the target result processing configuration and the detection result. For example, the result processing module finds the processing method corresponding to the detection result from the target result processing configuration according to the detection result, and determines the processing method as the target processing method.

[0089] For example, the result processing module provides a unified data verification interface and nested methods, and multiple applications that implement the interface form a complete result processing model. The result processing module can execute interface implementation programs in sequence according to the configuration order, and the output of the previous implementation program is the input of the next implementation program.

[0090] For example, the handlers include:

[0091] Modify the information in the file merge node (i.e., the target data node) from pending verification to successful verification, corresponding to the processing procedure B1 of the implemented interface I;

[0092] Modify the information in the file merge node from pending verification to verification failure, corresponding to the processing procedure B2 of the implemented interface I;

[0093] Delete the information in the file merge node, corresponding to the processing procedure B3 of the implemented interface I;

[0094] Trigger an alarm, corresponding to the implemented interface I handler B4.

[0095] Optionally, an optional target result processing configuration is as follows:

[0096] The detection passes the flag 0 and is configured as B1;

[0097] The detection failed flag 1 is configured as B2B4;

[0098] Detection failure flag 2, configured as B4;

[0099] The detection has abnormal flag 3, configured as B3.

[0100] For example, if the detection result is that the detection fails, the result processing module determines that the corresponding target processing method is B2B4, and the result processing module can first call B2 and then call B4 to implement the execution of the target processing method.

[0101] It should be noted that, through the above-mentioned method, it is achieved that the target processing method is determined based on the target result processing configuration matching the job name, thereby improving the accuracy of the determined target processing method.

[0102] Optionally, in the data processing method provided in the embodiment of the present application, the integrity of the target file is detected based on the target file information and the file information of multiple shard files to obtain a detection result, including: determining the target executor to be subjected to integrity detection from multiple executors through a monitoring module in the target processing system, wherein the multiple executors belong to a data processing module in the target processing system; importing the executor information of the target executor into the first data node in the target storage system through the monitoring module; controlling the target executor through the data processing module according to the executor information in the first data node to detect the integrity of the target file based on the target file information and the file information of multiple shard files to obtain a detection result.

[0103] In some embodiments, the data processing module includes multiple executors, and the executors are used to perform the file integrity detection task. The monitoring module in the target processing system can specify which executor performs the integrity detection task of the current file (ie, the target file).

[0104] In some embodiments, Figure 4 is a schematic diagram of the operation of the monitoring module and the data processing module provided in the embodiment of the present application, such as Figure 4 As shown, when the monitoring module detects that the target file information is written in the target data node, it determines the target executor to be executed for integrity detection from multiple executors. For example, the monitoring module can determine the target executor according to the executor number. For example, if the executor numbered 02 was assigned to execute integrity detection last time, the executor numbered 03 is determined as the target executor this time, thereby evenly dividing the integrity detection task among multiple executors.

[0105] Optionally, after determining the target executor, the monitoring module may import the executor information of the target executor into the first data node in the target storage system. The executor information may be the IP information of the target executor, the first data node belongs to the above-mentioned multiple data nodes, and the first data node refers to a data node for storing the information of the target executor to be subjected to integrity detection.

[0106] Optional, such as Figure 4 As shown, the data processing module can monitor the first data node, and then obtain the executor information from the first data node when the executor information of the target executor is detected to be written in the first data node. After that, the data processing module can determine the target executor according to the obtained executor information, and then control the target executor to detect the integrity of the target file based on the target file information and the file information of the multiple shard files to obtain the detection result.

[0107] Optional, such as Figure 4As shown, after determining that the target executor has completed the integrity test, the data processing module can import the test completion information into the second data node in the target storage system, the second data node belongs to the above-mentioned multiple data nodes, and the second data node refers to the data node used to store the test completion information of the target executor. Optionally, after determining that the target executor has completed the integrity test, the data processing module can also send the test result to the result processing module and delete the information in the first data node.

[0108] Optional, such as Figure 4 As shown, the monitoring module can monitor the second data node to timely determine whether the integrity detection is completed. When the monitoring module detects that there is detection completion information in the second data node, it can monitor whether the target file information in the target data node is updated, so as to determine whether the integrity detection task needs to be re-executed when the target file information is updated. Wherein, when the monitoring module determines that the integrity detection task needs to be re-executed, the monitoring module can re-determine the target executor and delete the detection completion information in the second data node.

[0109] It should be noted that, through the above method, the monitoring module and the data processing module are decoupled, and the monitoring center only reads and forwards the data, which can support the processing of large amounts of data, reduce system performance risks, and improve processing efficiency. By using the data nodes in the target storage system as a bridge, auxiliary transmission of information can be achieved, thereby improving the communication efficiency between the monitoring module and the data processing module.

[0110] Optionally, in the data processing method provided in an embodiment of the present application, after determining the target processing method based on the detection results, the method also includes: updating the target file information in the target data node based on the target processing method, and sending the target information to the target application based on the target processing method, wherein the target information is used to characterize whether the target application is allowed to send the target file to the next processing system.

[0111] After determining the target processing mode, the result processing module can execute the target processing mode. The process of executing the target processing mode includes updating the target file information in the target data node and sending the target information to the target application.

[0112] Optionally, the result processing module updates the target file information in the target data node based on the target processing mode. Different target processing modes correspond to different update contents. For example, the target processing mode determined in this embodiment may include one of the following: "Modify the information in the file merge node (i.e., the target data node) from pending verification to successful verification", "Modify the information in the file merge node from pending verification to failed verification", "Delete the information in the file merge node", "Trigger an alarm (e.g., add warning information in the file merge node)", etc.

[0113] Optionally, the result processing module sends target information to the target application based on the target processing method. Different target processing methods correspond to different target information. For example, the target processing method determined in this embodiment may include one of the following: "sending a message of passing the test to the target processing system", "sending a message of failing the test to the target processing system", "sending a message of detecting anomaly to the target processing system", "sending a message of detecting failure to the target processing system". The aforementioned "message of passing the test", "message of failing the test", "message of detecting anomaly" and "message of detecting failure" are equivalent to the target information. Optionally, when the target application receives a message of passing the test, it determines that the target file can be sent to the next processing system. When receiving a message of failing the test, or detecting anomaly, or detecting failure, it determines that the target file is prohibited from being sent to the next processing system. And the target application can re-merge the files when receiving a message of detecting anomaly, and generate an early warning message when receiving a message of detecting failure to remind the administrator to intervene in the process.

[0114] It should be noted that, through the above process, on the one hand, the processing results for the target file can be effectively stored for easy record query, and on the other hand, effective indication of whether the target application can send the target file to the next processing system is achieved, thereby effectively avoiding the target application from sending the wrong target file to the next processing system, thereby effectively improving the accuracy of the target file sent by the target application, avoiding production problems and the human and material costs associated with solving production problems.

[0115] Optionally, in the data processing method provided in an embodiment of the present application, the target file information in the target data node is updated based on the target processing method, including: updating the to-be-verified flag in the target file information to a verification success flag, or updating the to-be-verified flag in the target file information to a verification failure flag, or deleting the target file information.

[0116] For example, when the test result indicates that the test is passed, the to-be-verified flag in the target file information is updated to a verification success flag; when the test result indicates that the test is not passed, the to-be-verified flag in the target file information is updated to a verification failure flag; when the test result indicates that the test is abnormal, the target file information is deleted so that the target application can merge the files again and the target processing system can re-perform integrity testing.

[0117] It should be noted that, through the above process, the target file information is effectively modified, thereby improving the accurate recording of the processing results of the target file.

[0118] In an optional embodiment, Figure 5 is a schematic diagram of a data processing method provided according to an embodiment of the present application, according to Figure 5 An optional data processing method in this embodiment is described. Figure 5 As shown, the target application first slices the original file to obtain multiple slice files, and then writes the file information of the slice files to the file generation node of the target storage system. After that, the target application performs file merging processing to obtain the target file, and then writes the file information of the target file to the file merging node (i.e., the target data node) of the target storage system. After that, the target processing system monitors the data changes in the file merging node, performs integrity detection on the file information with the same job name and the same session number in the file generation node and the file merging node, determines the target processing method according to the detection results, and executes the target processing method. The target application determines whether to send the target file to the next processing system based on the target processing method of the target processing system.

[0119] It can be seen that the solution provided by the present application achieves the purpose of performing integrity detection on the file obtained by merging the application, thereby realizing the technical effect of improving the accuracy of the file, and further solving the technical problem in the related technology that after the application merges the fragmented files, the merged file is directly sent to the downstream for further processing, resulting in poor file accuracy.

[0120] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0121] Example 2

[0122] The embodiment of the present application also provides a data processing device. It should be noted that the data processing device of the embodiment of the present application can be used to execute the data processing method provided by the embodiment of the present application. The data processing device provided by the embodiment of the present application is introduced below.

[0123] According to an embodiment of the present application, a device for implementing the above data processing method is also provided, such as Figure 6 As shown, the device comprises:

[0124] The detection module 601 is used to detect whether the target file information is written into the target data node of the target storage system, wherein the target storage system stores data based on multiple data nodes, the target data node belongs to multiple data nodes, and the target file information refers to the file information of the target file obtained after the target application merges multiple shard files, and the target file information is written into the target data node after the target application merges multiple shard files;

[0125] An acquisition module 602 is used to acquire target file information when it is detected that target file information is written in the target data node;

[0126] Determination module 603 is used to detect the integrity of the target file based on the target file information and the file information of multiple fragment files, obtain the detection result, and determine the target processing method based on the detection result, wherein the target processing method is used to guide the target application to send the target file to the next processing system, or to guide the target application to prohibit sending the target file to the next processing system.

[0127] In an embodiment of the present application, by detecting whether the target file information is written in the target data node of the target storage system, and setting the target file information to be written to the target data node after the target application merges multiple shard files, effective detection of whether the target application merges to obtain a new file is achieved. By detecting that the target file information is written in the target data node, the integrity of the target file is detected based on the target file information and the file information of multiple shard files to obtain a detection result. When the target application merges to obtain a new file, effective verification of the integrity of the merged file is achieved. By determining the target processing method based on the detection result, it is achieved to determine whether the target application is allowed to send the merged file to the downstream based on the integrity of the merged file, thereby avoiding sending wrong files to the downstream and causing production problems, thereby effectively improving the accuracy of the file.

[0128] It can be seen that the solution provided by the present application achieves the purpose of performing integrity detection on the file obtained by merging the application, thereby realizing the technical effect of improving the accuracy of the file, and further solving the technical problem in the related technology that after the application merges the fragmented files, the merged file is directly sent to the downstream for further processing, resulting in poor file accuracy.

[0129] Optionally, in the data processing device provided in the embodiment of the present application, the determination module also includes: an extraction sub-module, used to extract the job name to which the target file belongs from the target file information; an acquisition sub-module, used to obtain a first matching relationship between multiple job names and multiple file verification methods, and determine the target file verification method from multiple file verification methods based on the first matching relationship and the job name to which the target file belongs; a first detection sub-module, used to detect the integrity of the target file according to the target file verification method, based on the target file information and the file information of multiple segmented files, and obtain a detection result.

[0130] Optionally, in the data processing device provided in the embodiment of the present application, the first detection submodule also includes: a first extraction unit, used to extract the number of lines of the target file from the target file information, and extract the number of lines of the slice file from the file information of each slice file when the target file verification method includes a row verification method; a first detection unit, used to detect the integrity of the target file based on the number of lines of the target file and the number of lines of multiple slice files; a second extraction unit, used to extract the storage address information of the target file from the target file information, and extract the storage address information of the slice file from the file information of each slice file when the target file verification method includes a content verification method; a first acquisition unit, used to acquire the target file according to the storage address information of the target file, and acquire multiple slice files according to the storage address information of multiple slice files; a second detection unit, used to detect the integrity of the target file based on the content of the target file and the content of the multiple slice files.

[0131] Optionally, in the data processing device provided in the embodiment of the present application, the determination module also includes: a third extraction unit, used to extract the job name to which the target file belongs from the target file information; a first acquisition unit, used to obtain a second matching relationship between multiple job names and multiple result processing configurations, and determine the target result processing configuration from the multiple result processing configurations based on the second matching relationship and the job name to which the target file belongs; and a determination unit, used to determine the target processing method based on the target result processing configuration and the detection result.

[0132] Optionally, in the data processing device provided in the embodiment of the present application, the determination module also includes: a determination sub-module, used to determine the target executor to be subjected to integrity detection from multiple executors through the monitoring module in the target processing system, wherein the multiple executors belong to the data processing module in the target processing system; a processing sub-module, used to import the executor information of the target executor into the first data node in the target storage system through the monitoring module; and a second detection sub-module, used to control the target executor through the data processing module according to the executor information in the first data node, to detect the integrity of the target file based on the target file information and the file information of multiple shard files, and obtain a detection result.

[0133] Optionally, in the data processing device provided in the embodiment of the present application, the data processing device also includes: a processing module, used to update the target file information in the target data node based on the target processing method, and send the target information to the target application based on the target processing method, wherein the target information is used to indicate whether the target application is allowed to send the target file to the next processing system.

[0134] Optionally, in the data processing device provided in the embodiment of the present application, the processing module also includes: a first update module, used to update the to-be-verified flag in the target file information to a verification success flag, or, a second update module, used to update the to-be-verified flag in the target file information to a verification failure flag, or, a deletion module, used to delete the target file information.

[0135] It should be noted that the detection module 601, the acquisition module 602 and the determination module 603 correspond to steps S201 to S203 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

[0136] Example 3

[0137] An embodiment of the present application may provide an electronic device, Figure 7 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 7 As shown, the electronic device may include: one or more ( Figure 7 (only one is shown) processor 1002, memory 1004, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0138] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: detect whether the target file information is written in the target data node of the target storage system, wherein the target storage system stores data based on multiple data nodes, the target data node belongs to multiple data nodes, and the target file information refers to the file information of the target file obtained after the target application merges multiple shard files, and the target file information is written to the target data node after the target application merges multiple shard files; when it is detected that the target file information is written in the target data node, the target file information is obtained; based on the target file information and the file information of multiple shard files, the integrity of the target file is detected to obtain the detection result, and the target processing method is determined based on the detection result, wherein the target processing method is used to guide the target application to send the target file to the next processing system, or to guide the target application to prohibit sending the target file to the next processing system.

[0140] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: extract the job name to which the target file belongs from the target file information; obtain a first matching relationship between multiple job names and multiple file verification methods, and determine the target file verification method from multiple file verification methods based on the first matching relationship and the job name to which the target file belongs; according to the target file verification method, based on the target file information and the file information of multiple segmented files, detect the integrity of the target file to obtain a detection result.

[0141] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: when the target file verification method includes a row verification method, extract the number of rows of the target file from the target file information, and extract the number of rows of the slice file from the file information of each slice file; based on the number of rows of the target file and the number of rows of multiple slice files, detect the integrity of the target file; when the target file verification method includes a content verification method, extract the storage address information of the target file from the target file information, and extract the storage address information of the slice file from the file information of each slice file; obtain the target file according to the storage address information of the target file, and obtain multiple slice files according to the storage address information of multiple slice files; detect the integrity of the target file according to the content of the target file and the content of the multiple slice files.

[0142] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: extract the job name to which the target file belongs from the target file information; obtain a second matching relationship between multiple job names and multiple result processing configurations, and determine the target result processing configuration from the multiple result processing configurations based on the second matching relationship and the job name to which the target file belongs; determine the target processing method based on the target result processing configuration and the detection result.

[0143] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the target executor to be subjected to integrity detection from multiple executors through the monitoring module in the target processing system, wherein the multiple executors belong to the data processing module in the target processing system; import the executor information of the target executor into the first data node in the target storage system through the monitoring module; control the target executor through the data processing module according to the executor information in the first data node, detect the integrity of the target file based on the target file information and the file information of multiple shard files, and obtain the detection result.

[0144] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: after determining the target processing method based on the detection results, update the target file information in the target data node based on the target processing method, and send the target information to the target application based on the target processing method, wherein the target information is used to indicate whether the target application is allowed to send the target file to the next processing system.

[0145] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: update the to-be-verified flag in the target file information to a verification success flag, or update the to-be-verified flag in the target file information to a verification failure flag, or delete the target file information.

[0146] In an embodiment of the present application, by detecting whether the target file information is written in the target data node of the target storage system, and setting the target file information to be written to the target data node after the target application merges multiple shard files, effective detection of whether the target application merges to obtain a new file is achieved. By detecting that the target file information is written in the target data node, the integrity of the target file is detected based on the target file information and the file information of multiple shard files to obtain a detection result. When the target application merges to obtain a new file, effective verification of the integrity of the merged file is achieved. By determining the target processing method based on the detection result, it is achieved to determine whether the target application is allowed to send the merged file to the downstream based on the integrity of the merged file, thereby avoiding sending wrong files to the downstream and causing production problems, thereby effectively improving the accuracy of the file.

[0147] It can be seen that the solution provided by the present application achieves the purpose of performing integrity detection on the file obtained by merging the application, thereby realizing the technical effect of improving the accuracy of the file, and further solving the technical problem in the related technology that after the application merges the fragmented files, the merged file is directly sent to the downstream for further processing, resulting in poor file accuracy.

[0148] It can be understood by those skilled in the art that Figure 7 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 7 The structure of the electronic device is not limited. Figure 7 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 7 Different configurations shown.

[0149] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0150] Example 4

[0151] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0152] Optionally, in this embodiment, the above-mentioned storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0153] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program of the data processing method steps.

[0154] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0155] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0160] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: Applied to target processing systems, including: Detecting whether target file information is written into a target data node of a target storage system, wherein the target storage system stores data based on multiple data nodes, the target data node belongs to the multiple data nodes, and the target file information refers to file information of a target file obtained after a target application merges multiple shard files, and the target file information is written into the target data node after the target application merges the multiple shard files; In the case where it is detected that the target file information is written into the target data node, acquiring the target file information; Based on the target file information and the file information of the multiple shard files, the integrity of the target file is detected to obtain a detection result, and a target processing method is determined based on the detection result, wherein the target processing method is used to guide the target application to send the target file to the next processing system, or to guide the target application to prohibit sending the target file to the next processing system.

2. The method according to claim 1, characterized in that Based on the target file information and the file information of the multiple fragment files, the integrity of the target file is detected to obtain a detection result, including: Extracting the job name to which the target file belongs from the target file information; Acquire a first matching relationship between a plurality of job names and a plurality of file verification modes, and determine a target file verification mode from the plurality of file verification modes based on the first matching relationship and the job name to which the target file belongs; According to the target file verification method, based on the target file information and the file information of the multiple fragment files, the integrity of the target file is detected to obtain a detection result.

3. The method according to claim 2, characterized in that According to the target file verification method, based on the target file information and the file information of the multiple fragment files, detecting the integrity of the target file includes: In the case where the target file verification mode includes a row verification mode, extracting the number of rows of the target file from the target file information, and extracting the number of rows of each fragment file from the file information of each fragment file; Based on the number of lines of the target file and the number of lines of the plurality of fragment files, detecting the integrity of the target file; In the case where the target file verification method includes a content verification method, extracting the storage address information of the target file from the target file information, and extracting the storage address information of each fragment file from the file information of each fragment file; Acquire the target file according to the storage address information of the target file, and acquire the multiple fragment files according to the storage address information of the multiple fragment files; The integrity of the target file is detected according to the content of the target file and the contents of the multiple segment files.

4. The method according to claim 1, characterized in that: Determining a target processing method based on the detection result includes: Extracting the job name to which the target file belongs from the target file information; Acquire a second matching relationship between a plurality of job names and a plurality of result processing configurations, and determine a target result processing configuration from the plurality of result processing configurations based on the second matching relationship and the job name to which the target file belongs; The target processing mode is determined based on the target result processing configuration and the detection result.

5. The method according to claim 1, characterized in that Based on the target file information and the file information of the multiple fragment files, the integrity of the target file is detected to obtain a detection result, including: Determining a target executor to be subjected to integrity detection from a plurality of executors through a monitoring module in the target processing system, wherein the plurality of executors belong to a data processing module in the target processing system; Importing the executor information of the target executor into the first data node in the target storage system through the monitoring module; The data processing module controls the target executor according to the executor information in the first data node, detects the integrity of the target file based on the target file information and the file information of the multiple shard files, and obtains the detection result.

6. The method according to claim 1, characterized in that After determining the target processing method based on the detection result, the method further includes: The target file information in the target data node is updated based on the target processing mode, and the target information is sent to the target application based on the target processing mode, wherein the target information is used to indicate whether the target application is allowed to send the target file to the next processing system.

7. The method according to claim 6, characterized in that Updating target file information in the target data node based on the target processing mode includes: Update the pending verification mark in the target file information to a verification success mark, or, The to-be-verified flag in the target file information is updated to a verification failure flag, or, Delete the target file information.

8. A data processing device, characterized in that: Applied to target processing systems, including: a detection module, configured to detect whether target file information is written into a target data node of a target storage system, wherein the target storage system stores data based on multiple data nodes, the target data node belongs to the multiple data nodes, the target file information refers to file information of a target file obtained after a target application merges multiple shard files, and the target file information is written into the target data node after the target application merges the multiple shard files; An acquisition module, configured to acquire the target file information when detecting that the target file information is written into the target data node; A determination module is used to detect the integrity of the target file based on the target file information and the file information of the multiple segment files, obtain a detection result, and determine a target processing method based on the detection result, wherein the target processing method is used to guide the target application to send the target file to the next processing system, or to guide the target application to prohibit sending the target file to the next processing system.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein the program executes the data processing method according to any one of claims 1 to 7 when running.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images