File processing method and optimization method

Abstract

The embodiment of the invention provides a file processing method and an optimization method. The file processing method comprises the steps of obtaining a production file corresponding to a service system and a gray-scale file having a comparison relationship with the production file; adding file mark bytes for row data contained in the production file and the gray file respectively, and calculating hash values corresponding to the row data; selecting a data storage space corresponding to the row data according to the file mark bytes and the hash value, and performing data consistency detection on the data storage space according to the row data; and determining difference data between the production file and the gray file in the data storage space according to a detection result.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of file processing technology, and more particularly to file processing methods and optimization methods. Background Art

[0002] With the development of computer and Internet technology, the data interaction mode in the form of files has become one of the main data interaction modes between most business systems and external systems. Data interaction can be carried out in batches through files, which can effectively improve the service quality and efficiency of business systems. In the prior art, in order to improve the reliability of business systems, a grayscale reflux mechanism is usually introduced during the system upgrade process. On this basis, by comparing the files in the grayscale environment with the files in the production environment, the data related to the scheduled upgraded business system version can be backtested to optimize the business system according to the backtest results, thereby providing stable services. In the prior art, when performing file comparison processing, due to the large difference in file size, ranging from MB level to GB level, for smaller files, the data can be loaded into the memory for line-by-line comparison, while for larger files, the comparison can only be completed by the K-way merge sorting method of the file, but this processing method consumes a lot of computing resources, and the data comparison granularity cannot reach the row level. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, an embodiment of this specification provides a file processing method. One or more embodiments of this specification also relate to a file processing device, an optimization method, an optimization device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a file processing method is provided, including: Obtaining a production file corresponding to the business system and a grayscale file having a comparative relationship with the production file; Adding file mark bytes to the row data respectively contained in the production file and the grayscale file, and calculating the hash value corresponding to the row data; Selecting a data storage space corresponding to the row data according to the file mark byte and the hash value, and performing a data consistency check on the data storage space according to the row data; The difference data between the production file and the grayscale file is determined in the data storage space according to the detection result.

[0005] According to a second aspect of the embodiments of this specification, an optimization method is provided, including: Obtain the production files of the business system's associated production dimension and the grayscale files of the associated grayscale dimension; Adding file mark bytes to the row data respectively contained in the production file and the grayscale file, and calculating the hash value corresponding to the row data; Selecting a data storage space corresponding to the row data according to the file mark byte and the hash value, and performing a data consistency check on the data storage space according to the row data; The difference data between the production file and the grayscale file is determined in the data storage space according to the detection result, and the business system is optimized based on the difference data.

[0006] According to a third aspect of an embodiment of this specification, there is provided a file processing device, including: An acquisition module is configured to acquire a production file corresponding to the business system and a grayscale file having a comparative relationship with the production file; An adding module is configured to add a file mark byte to the row data respectively contained in the production file and the grayscale file, and calculate a hash value corresponding to the row data; a detection module configured to select a data storage space corresponding to the row data according to the file mark byte and the hash value, and perform a data consistency detection on the data storage space according to the row data; The determination module is configured to determine the difference data between the production file and the grayscale file in the data storage space according to the detection result.

[0007] According to a fourth aspect of the embodiments of this specification, an optimization device is provided, including: An acquisition module is configured to acquire a production file associated with a production dimension and a grayscale file associated with a grayscale dimension of a business system; An adding module is configured to add a file mark byte to the row data respectively contained in the production file and the grayscale file, and calculate a hash value corresponding to the row data; a detection module configured to select a data storage space corresponding to the row data according to the file mark byte and the hash value, and perform a data consistency detection on the data storage space according to the row data; The optimization module is configured to determine the difference data between the production file and the grayscale file in the data storage space according to the detection result, and optimize the business system based on the difference data.

[0008] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned file processing method or optimization method are implemented.

[0009] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned file processing method or optimization method are implemented.

[0010] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned file processing method or optimization method when executed by a processor.

[0011] The file processing method provided in this embodiment can first obtain the production file corresponding to the business system and the grayscale file having a comparison relationship with the production file in order to improve the efficiency of file comparison and reduce the consumption of computing resources; on this basis, in order to achieve row-level data comparison operations, file mark bytes can be added to the row data respectively contained in the production file and the grayscale file, and hash values ​​corresponding to the row data respectively contained in the production file and the grayscale file can be calculated; on this basis, the corresponding data storage space can be selected for each row of data according to the file mark bytes and the hash values, and the data storage space can be tested for data consistency according to each row of data; in this way, after the row data are stored in their respective corresponding data storage spaces, the row data stored in the data storage space can be eliminated according to the row data to be written, so as to determine the difference data between the production file and the grayscale file in the data storage space according to the elimination result; the file consistency test can be completed by streaming comparison of row data, thereby effectively improving the file comparison accuracy and consuming less computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of a file processing method provided by an embodiment of this specification; Figure 2 is a flowchart of a file processing method provided by an embodiment of this specification; Figure 3 It is a schematic diagram of a processing process of a file processing method provided by an embodiment of this specification; Figure 4 is a flow chart of an optimization method provided by an embodiment of this specification; Figure 5 is a processing flow chart of an optimization method provided by an embodiment of this specification; Figure 6is a structural schematic diagram of a file processing device provided by an embodiment of this specification; Figure 7 It is a structural schematic diagram of an optimization device provided by an embodiment of this specification; Figure 8 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0013] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0014] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0016] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0017] First, the terms involved in one or more embodiments of this specification are explained.

[0018] TA system: It is the abbreviation of the open-end fund registration and settlement system of the Clearing Corporation. It is used to register the fund shares of investors in their securities accounts.

[0019] Grayscale reflux mechanism: It is an important part of the grayscale release strategy. It involves the process of promptly rolling back the version to a stable version after problems are found during the grayscale testing phase.

[0020] K-way Merge: (K-way Merge) is an algorithm based on merge sort, which divides the data to be sorted into K subsequences, sorts each subsequence separately, and then merges these sorted subsequences into an ordered sequence.

[0021] In this specification, a file processing method is provided. One or more embodiments of this specification also relate to a file processing device, an optimization method, an optimization device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0022] In actual applications, when comparing files in the production environment and the grayscale environment, it is necessary to decide whether to enable K-way merge comparison based on the file size and the set threshold. When the threshold is set too large, it is easy to cause content overflow problems. When the threshold is set too small, the performance of file comparison will be reduced. At the same time, the K-way merge operation on the file needs to ensure that the file is in the path accessible to the current program, that is, the program does not have the elastic ability to expand horizontally without the help of a network file system or a distributed file system. In addition, the K-way merge comparison method is not accurate for line-by-line comparison of merged files. Under this solution, in extreme cases, there will be one more line in the first line of file A, and the second line will actually be consistent with the content of file B, and all lines of the two files will be considered to be mismatched. This increases the difficulty of troubleshooting; at the same time, since there may be files of different types in the business system, K-way merge can only operate on a certain type of file. Therefore, the concurrency granularity of this solution is at the file type level, and cannot reach the data level. Therefore, an effective solution is urgently needed to solve the above problems.

[0023] See also Figure 1As shown in the schematic diagram, the file processing method provided in this embodiment can first obtain the production file corresponding to the business system and the grayscale file having a comparison relationship with the production file in order to improve the file comparison efficiency and reduce the consumption of computing resources; on this basis, in order to achieve row-level data comparison operations, file mark bytes can be added to the row data respectively contained in the production file and the grayscale file, and the hash values ​​corresponding to the row data respectively contained in the production file and the grayscale file can be calculated; on this basis, the corresponding data storage space can be selected for each row of data according to the file mark bytes and the hash values, and the data storage space can be tested for data consistency according to each row of data; in this way, after the row data are stored in their respective corresponding data storage spaces, the row data stored in the data storage space can be eliminated according to the row data to be written, so as to determine the difference data between the production file and the grayscale file in the data storage space according to the elimination result; the file consistency test is completed by streaming comparison of row data, thereby effectively improving the file comparison accuracy and consuming less computing resources.

[0024] See also Figure 2 , Figure 2 A flowchart of a file processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0025] Step S202, obtaining a production file corresponding to the business system and a grayscale file having a comparison relationship with the production file.

[0026] The file processing method provided in this embodiment can be applied to the consistency detection of grayscale files and production files involved in a business system that provides arbitrary functional services to users, wherein the functional services provided by the business system include but are not limited to financial transaction services, transaction services, video browsing services, instant messaging services, information sharing services, smart device control services, etc.; it is used to implement consistency detection of production files of a production environment associated with the business system, and grayscale files of an associated grayscale environment, thereby achieving data-level file comparison processing, so that when optimizing the business system, the operation can be completed by referring to the compared difference data.

[0027] Specifically, the production file specifically refers to the data file corresponding to the business system in the production environment, wherein the production environment refers to the maintenance / development stage of the business system, and the data file stores the files involved in the business system in the production environment. For example, if the business system is an inter-integration trading system, it can store centralized reporting data, accounting details data, channel interaction data, or capital interaction data. Correspondingly, the grayscale file specifically refers to the data file corresponding to the business system in the grayscale environment, wherein the grayscale environment refers to the grayscale reflow test stage of the business system, and the data file stores the files involved in the business system in the grayscale environment. For example, if the business system is an inter-integration trading system, it can store centralized reporting data, accounting details data, channel interaction data, or capital interaction data. By performing consistency detection on production files and grayscale files, an alarm can be issued for the optimization of the business system, so as to provide guarantee for the optimization of the business system when the files are consistent.

[0028] It should be noted that the comparison relationship between grayscale files and production files means that the two belong to different environments but store the same data type. By comparing files with this relationship, the problem of the business system can be effectively determined to achieve the purpose of root cause location.

[0029] Based on this, in order to improve the efficiency of file comparison while reducing the consumption of computing resources, the production file corresponding to the business system and the grayscale file that has a comparison relationship with the production file can be obtained first; on this basis, in order to achieve row-level data comparison operations, file mark bytes can be added to the row data contained in the production file and the grayscale file respectively, and the hash values ​​corresponding to the row data contained in the production file and the grayscale file respectively can be calculated; on this basis, the corresponding data storage space can be selected for each row of data according to the file mark byte and the hash value, and the data storage space can be tested for data consistency according to each row of data; in this way, after the row data is stored in their respective corresponding data storage spaces, the row data stored in the data storage space can be eliminated according to the row data to be written, so as to determine the difference data between the production file and the grayscale file in the data storage space according to the elimination result; the file consistency test is completed by streaming comparison of row data, thereby effectively improving the file comparison accuracy and consuming less computing resources.

[0030] Furthermore, when obtaining grayscale files and production files, in order to ensure that the obtained files have a contrast relationship, this can be achieved by executing a file reading task. In this embodiment, the specific implementation method is as follows: Initialize a thread pool matching the business system, and call set resources in the thread pool according to the file comparison strategy to execute the file reading task corresponding to the business system; read the production file and the grayscale file having a comparison relationship with the production file in the preset system directory according to the task execution result.

[0031] Specifically, the thread pool matched by the business system specifically refers to the thread pool composed of available threads when the file corresponding to the business system performs file consistency detection. Correspondingly, the file comparison strategy specifically refers to the strategy that specifies the call resource size and the file reading path. Correspondingly, the file reading task specifically refers to the task of reading production files and grayscale files in the path specified by the file comparison strategy.

[0032] Based on this, when obtaining the production files associated with the business system and the grayscale files that have a comparison relationship with the production files, the thread pool matching the business system can be initialized first. On this basis, the set resources can be called in the thread pool according to the file comparison strategy to execute the file reading task corresponding to the business system; the file reading task can be used to determine the path of the file to be read, and then the production files and the grayscale files that have a comparison relationship with the production files can be read in the preset system directory according to the task execution results, so as to perform consistency detection on the production files and grayscale files that have a comparison relationship subsequently.

[0033] In practical applications, see Figure 3 As shown in the schematic diagram, after the thread pool is initialized, the file reading task can be executed according to fixed resources, so that each assigned process can read the production file and the grayscale file in the execution directory according to the received comparison file type, so that when each subsequent file is called by the thread pool, the data in the file can be read line by line and the subsequent file consistency detection processing can be completed.

[0034] In specific implementation, the files read by each process can be decided by the scheduling node. According to the decision result, the path list that the process needs to access can be determined. By putting the paths in the path list into the message queue, after the file reading service subscribes to the message queue, a file path can be obtained to complete the file reading operation, so as to further complete the file consistency detection processing of the business system.

[0035] For example, in a financial transaction scenario, business system H can provide financial transaction services to users. When business system H needs to be upgraded, in order to provide guarantee for the system upgrade, it is necessary to read the production files of business system H in the production environment and the grayscale files in the grayscale environment, and after determining that the read files have a comparison relationship, a consistency test is performed to optimize business system H according to the consistency test results. For example, for business system H, a centralized backup 92 file named 92-PRD is read in the production environment, and a centralized backup 92 file named 92-REFLUX is read in the grayscale environment. Each file size is 50GB. Subsequently, the 92-PRD file and the 92-REFLUX file need to be compared, and the difference items are finally output to optimize business system H.

[0036] In summary, by reading files according to the set strategy, it can be ensured that the read files are associated with the business system, and the grayscale files and production files have a comparative relationship, so that the subsequent file consistency detection results can optimize the business system.

[0037] Step S204, adding file mark bytes to the row data respectively contained in the production file and the grayscale file, and calculating the hash value corresponding to the row data.

[0038] Specifically, after the production file and the grayscale file have a comparison relationship, in order to achieve the file consistency detection processing at the data level, the file mark byte can be added to the row data contained in the production file and the grayscale file respectively, so as to reflect the file and environment to which each row of data belongs through the file mark byte. On this basis, the hash value of the row data contained in the production file and the grayscale file respectively can be further calculated, so that the data storage space can be selected according to the hash value in the future, thereby achieving the purpose of file consistency detection at the data level.

[0039] Among them, the file mark byte specifically refers to the mark byte appended after each row of data. The mark byte can be composed of one or more sub-bytes. Different positions and different byte values ​​can express different meanings. For example, in this embodiment, the file mark byte can represent the source file and environment of the row data through different byte values, so as to be used in data-level consistency detection processing.

[0040] Furthermore, when adding a file mark byte to the row data, in order to facilitate the subsequent completion of the data level file consistency detection, it can be achieved by adding a type mark byte and an environment mark byte. In this embodiment, adding a file mark byte to any row data included in the production file or any row data included in the grayscale file includes: Read the target row data and determine the source file of the target row data; determine the target file type mark byte and the target environment mark byte according to the file type of the source file; use the target file type mark byte and the target environment mark byte as the target file mark bytes corresponding to the target row data, and add them to the target row data.

[0041] Specifically, the target row data specifically refers to the row data that currently needs to append the mark byte, and the corresponding source file can be a production file or a grayscale file. Correspondingly, the target file type mark byte specifically refers to the mark byte that represents the file type to which the row data belongs, which can be expressed by 2 bytes. Correspondingly, the target environment mark byte specifically refers to the mark byte that represents the environment corresponding to the row data, which can be expressed by 1 byte.

[0042] Based on this, when adding a file mark byte to any row data contained in the production file or any row data contained in the grayscale file, the row data of the current byte to be appended can be obtained first, and then the source file of the target row data can be determined; since different source files represent row data corresponding to different environments, the target file type mark byte and the target environment mark byte can be determined according to the file type of the source file; thereafter, the target file type mark byte and the target environment mark byte can be used as the target file mark bytes corresponding to the target row data and appended to the target row data, and so on. After the file mark bytes are appended to the row data contained in the production file and the grayscale file, they can be used for subsequent data storage space selection and file consistency detection processing.

[0043] In actual applications, by appending a file type tag byte and an environment tag byte to each row of data, it is possible to reflect the file to which the row data belongs through the file type tag byte, and reflect whether the row data comes from a grayscale environment or a production environment through the environment tag byte. This makes it easier to determine the source of the difference data after subsequent file consistency detection and processing, so that it can be used for optimized use of the business system.

[0044] Following the above example, after obtaining the 92-PRD file corresponding to the production environment and the 92-REFLUX file corresponding to the grayscale environment, further, a line of data can be read from each of the two files, and then a mark byte expressing the file type and a mark byte expressing the environment corresponding to the file are appended to each line of data read. For example, the i-th line of data is read from the 92-PRD file, and a 2-bit byte is appended after the i-th line of data to indicate the file type of the corresponding 92-PRD file, and a 1-bit byte "0" is appended to indicate the corresponding production environment. Similarly, the j-th line of data is read from the 92-REFLUX file, and a 2-bit byte is appended after the j-th line of data to indicate the file type of the corresponding 92-REFLUX file, and a 1-bit byte "1" is appended to indicate the corresponding grayscale environment. After obtaining the line data after the additional mark byte, the line data after the additional mark byte can be used for subsequent file consistency detection, and the thread can continue to read line data from the file and repeat the above operation to achieve the purpose of streaming processing.

[0045] Furthermore, for the row data with the appended mark bytes, in order to facilitate the subsequent precise bucketing, the environment mark bytes and file type mark bytes appended in the row data can be stripped, and then the hash value can be calculated to obtain the hash value corresponding to each original row data, and then the file consistency detection processing can be performed based on the environment mark bytes and file type mark bytes appended to the row data.

[0046] In summary, by appending environment mark bytes and file type mark bytes to the row data read in the production file and the grayscale file, the file type and environment to which the row data belongs can be guaranteed, so that when performing subsequent file consistency detection, the situations where the data is the same but the data sources are different can be screened out to indicate data consistency.

[0047] Step S206: select a data storage space corresponding to the row data according to the file mark byte and the hash value, and perform a data consistency check on the data storage space according to the row data.

[0048] Specifically, after adding the file mark byte to each row of data in turn and calculating the hash value corresponding to each row of data, in order to achieve data-level consistency detection processing, the file mark byte and hash value corresponding to each row of data can be combined to select data storage space for each row of data, so as to realize streaming processing to store the row data in their respective corresponding data storage spaces. During the storage process, data consistency detection can also be performed on the data storage space according to the row data to be stored, thereby realizing data-level consistency detection. Finally, the difference data between the production file and the grayscale file can be determined by counting the non-empty data storage space for optimized use of the business system.

[0049] That is to say, when selecting the data storage space, the corresponding data storage space is actually selected for each row of data. At this time, considering that the same data should select the same data storage space, the row data of the production file and the same row data in the grayscale file can select the same data storage space. After the data storage space selection is completed, in order to complete the data level consistency detection processing, the row data to be stored in the data storage space can be used to compare the row data already stored in the data storage space to complete the consistency detection. When the data storage space corresponding to the row data does not contain the row data corresponding to the data, it means that the current row data to be stored is single, and it can be written into the data storage space first. Until a certain storage stage, when the row data to be stored is the same as the row data already in the data storage space, it can be determined that the two row data are respectively the production file and the grayscale file, and the two are consistent. Therefore, the existing row data can be deleted from the data storage space, and the row data to be stored does not need to be stored in the data storage space. The subsequent row data storage operations are analogous. Finally, the remaining row data in the data storage space is the difference data between the production file and the grayscale file. If the data storage space is an empty set, it means that the grayscale file and the production file are consistent, which can provide guarantee for the upgrade of the business system.

[0050] Furthermore, when selecting data storage space for row data, the selection can be completed after calculating the hash value and then combined with the file type tag. In this embodiment, the specific implementation method is as follows: A file type tag byte is extracted from the file tag byte, and a space selection value corresponding to the row data is calculated based on the file type tag byte and the hash value; a candidate data storage space corresponding to a space identifier matching the space selection value is selected from a data storage space cluster; and the candidate data storage space is used as the data storage space corresponding to the row data.

[0051] Specifically, the space selection value specifically refers to the value obtained by the modulo calculation based on the file type tag byte and the hash value, and its corresponding maximum value is the same as the number of data storage spaces contained in the data storage space cluster. Accordingly, the data storage space cluster specifically refers to a cluster composed of multiple data storage spaces, which can adopt a distributed storage architecture, so as to ensure that the consistency detection processing of multiple rows of data can be completed concurrently under the requirements of streaming data processing. Correspondingly, the candidate data storage space specifically refers to the data storage space selected for the currently processed row data.

[0052] Based on this, when selecting data storage space for the row data being processed in the current stage, the file type mark byte can be first extracted from the file mark byte corresponding to the row data. At this time, the space selection value corresponding to the row data can be calculated based on the file type mark byte and the hash value; since the space selection value has a corresponding relationship with the identifier of the data storage space in the cluster, a candidate data storage space corresponding to the space identifier that matches the space selection value can be selected in the data storage space cluster; and the candidate data storage space is used as the data storage space corresponding to the row data, and so on. The data storage space is selected according to the above processing in each processing cycle of the row data, thereby effectively improving the efficiency of subsequent file consistency detection.

[0053] In actual applications, when selecting data storage space in combination with the file type mark byte and the hash value, the space selection value can be obtained by calculating the remainder of the two, and then the data storage space corresponding to the space identifier equal to the space selection value can be selected in the cluster for subsequent processing. In addition to the above calculation of the space selection value, the space selection value can also be calculated by summing the file type mark byte and the hash value, or by subtracting the file type mark byte from the hash value. In specific implementation, the method for calculating the space selection value can be selected according to actual needs, and this embodiment does not make any restrictions here.

[0054] In summary, by combining the file type mark byte and the hash value to calculate the space selection value, it can be ensured that the data storage space selected for each row of data is more accurate, thereby effectively improving the detection efficiency when performing file consistency detection.

[0055] Furthermore, when performing data consistency check on the data storage space according to the row data, considering that the data in the file is stored in the data storage space according to the row level, multiple pre-storage check operations can be performed when storing row data in the data storage space. In this embodiment, the specific implementation method is as follows: Detect whether there is storage row data identical to the row data in the data storage space; if not, store the row data in the data storage space; if so, determine the storage file mark byte corresponding to the storage row data, and detect whether the file mark byte and the storage file mark byte are the same; if they are not the same, execute the step of storing the row data in the data storage space; if they are the same, delete the storage row data in the data storage space.

[0056] Specifically, storing row data specifically refers to row data that has been stored in the data storage space; correspondingly, storing file mark bytes specifically refers to the file mark bytes corresponding to the stored row data. For related descriptions, please refer to the description of the file mark bytes in the above embodiment, and this embodiment will not be elaborated in detail here.

[0057] Based on this, in the current row data processing stage, after selecting and completing the data storage space for the row data, the data-level file consistency detection process can be started at this time; during the detection process, it can be first detected whether there is storage row data identical to the row data in the data storage space; if not, it means that the row data to be stored in the current stage may be differential data, then the row data can be first stored in the data storage space, and the processing of the next row data can be continued. If it exists, it means that the row data to be stored in the current stage is the same as the storage row data in the data storage space. If the two are the same, it further indicates that the row data and the storage row data may come from the grayscale file and the production file respectively, and belong to the same data, and the two are consistent.

[0058] In order to ensure that the possibility is more accurate, the storage file mark byte corresponding to the storage row data can be determined, and the file mark byte and the storage file mark byte can be detected to be the same; if they are not the same, it means that the above possibility is not accurate, so the step of storing the row data in the data storage space can be performed; if they are the same, it means that the above possibility is accurate, the row data and the storage row data are consistent data, so the storage row data can be deleted in the data storage space. The same row data can be eliminated after the consistency detection process, and enter the next row data processing cycle, and so on, until the end, the remaining row data can be used as difference data.

[0059] In summary, by comparing the stored row data with the row data, it can be ensured that the consistent row data can be eliminated, and the remaining row data can reflect the difference between the production file and the grayscale file, thereby achieving data-level file consistency detection processing.

[0060] On this basis, considering that the grayscale file and the production file have a contrasting relationship, if the two are highly consistent, a large amount of data in the file is the same. In order to achieve a more fine-grained data consistency detection, the decision of storage and deletion can be completed in combination with the marker byte. In this embodiment, the specific implementation method is as follows: Extract the associated file type tag byte and the associated environment tag byte from the file tag byte, and extract the storage file type tag byte and the storage environment tag byte from the storage file tag byte; detect whether the file types of the row data and the storage row data are the same based on the associated file type tag byte and the storage file type tag byte; if they are not the same, execute the step of storing the row data in the data storage space; if they are the same, detect whether the source environments of the row data and the storage row data are consistent based on the associated environment tag byte and the storage environment tag byte; if they are consistent, execute the step of storing the row data in the data storage space; if they are inconsistent, execute the step of deleting the storage row data in the data storage space.

[0061] Specifically, the associated file type tag byte is the file type tag byte, and the associated environment tag byte is the environment tag byte. Correspondingly, the stored file type tag byte and the stored environment tag byte are the file type tag byte and the environment tag byte corresponding to the stored row data. For the above description, please refer to the same or corresponding description in the above embodiment, and this embodiment will not be repeated here.

[0062] Based on this, when it is determined that the row data is consistent with the stored row data, it means that the row data and the stored row data may come from the production file and the grayscale file respectively, and the two may be consistent. In order to detect this possibility, the file consistency detection process at the data level can be further completed. Therefore, the associated file type mark byte and the associated environment mark byte can be extracted from the file mark byte, and the storage file type mark byte and the storage environment mark byte can be extracted from the storage file mark byte.

[0063] On this basis, the file types of the row data and the stored row data can be detected according to the associated file type mark byte and the stored file type mark byte. If they are not the same, it means that the row data and the stored row data are only the same in data, but the file types they belong to are not the same, so the step of storing the row data in the data storage space can be performed. If they are the same, it means that the file types of the row data and the stored row data are also the same, so the source environment of the row data and the stored row data can be further detected according to the associated environment mark byte and the storage environment mark byte. If they are the same, it means that the row data and the stored row data are not only the same in data, but also the same in file type and environment, such as both from the production environment or from the grayscale environment, so it means that the row data has not completed the consistency detection with the corresponding row data in another file, so the step of storing the row data in the data storage space can be performed; if they are not the same, it means that the row data and the stored row data have the same data, the same in file type, but different in environment, so it can be determined that the row data and the stored row data are from the production file and the grayscale file respectively, and both belong to the same row data, so the step of deleting the stored row data in the data storage space can be performed. In order to achieve the purpose of eliminating the same row data.

[0064] In actual applications, when comparing the environment tag bytes of row data and storage row data, the operation can be completed by determining the XOR value corresponding to the environment tag byte. For example, if the XOR value of the environment tag bytes corresponding to the row data and the storage row data is 1, it means that the row data and the storage row data are from different environments; if the XOR value is 0, it means that the row data and the storage row data are from the same environment, so that the file consistency detection processing at the data level can be completed. When the data is consistent, the file type is consistent, and the XOR value is 1 (the row data and the storage row data come from different environments), the storage row data can be deleted from the data storage space to delete the consistent data. Otherwise, the row data can be stored in the data storage space and enter the next row data consistency detection processing cycle.

[0065] Continuing with the above example, after reading the i-th line of data from the 92-PRD file and the j-th line of data from the 92-REFLUX file, and completing the appending of the file type mark byte and the environment mark byte, the bucket number (space identifier of the data storage space) can be calculated based on the hash value hash_i corresponding to the i-th line of data and its corresponding file type mark byte. According to the calculation result, it is determined that the i-th line of data in the 92-PRD file corresponds to comparison bucket No. 8. Similarly, the bucket number (space identifier of the data storage space) is calculated based on the hash value hash_j corresponding to the j-th line of data and its corresponding file type mark byte. According to the calculation result, it is determined that the j-th line of data in the 92-REFLUX file corresponds to comparison bucket No. 120.

[0066] Furthermore, the above embodiment is described with the storage process of the i-th row of data. Specifically, for the i-th row of data in the 92-PRD file, it can be first detected whether there is an item identical to the i-th row of data in comparison bucket No. 8. If not, the i-th row of data with the additional environment mark byte and file type mark byte can be directly stored in comparison bucket No. 8. If it exists, the target row of data is obtained in comparison bucket No. 8, and the file type mark byte and environment mark byte corresponding to the row data are determined. When it is determined that the file types of the i-th row of data and the target row of data are inconsistent according to the file type mark byte, the i-th row of data with the additional environment mark byte and file type mark byte can be stored in comparison bucket No. 8. When the file types of the i-th row of data and the target row of data are consistent, it can be determined whether the environments of the i-th row of data and the target row of data are the same according to the environment mark byte. If they are the same, the i-th row of data with the additional environment mark byte and file type mark byte can be stored in comparison bucket No. 8. If they are not the same, it means that the target row data is the row data from the 92-REFLUX file, and it is the same row data as the i-th row data. Therefore, the compared target row data can be eliminated in comparison bucket No. 8 to delete the consistent row data and continue the subsequent row data consistency detection processing.

[0067] In summary, by adopting multi-level data consistency detection, data-level comparison can be achieved during file consistency detection processing, thereby meeting the comparison requirements in different scenarios and improving data comparison accuracy.

[0068] Step S208: determining the difference data between the production file and the grayscale file in the data storage space according to the detection result.

[0069] Specifically, after completing the data-level consistency detection process for the production file and the grayscale file in the streaming processing mode, the difference data between the production file and the grayscale file can be determined in the data storage space according to the detection result. In other words, through the above-mentioned data-level consistency detection, the consistent data can be eliminated from the data storage space, and the remaining data in the data storage space is the difference data between the production file and the grayscale file. According to the difference data, the upgrade defects of the business system and other problems can be reflected. The business party of the business system can optimize the business system according to the difference data, and then provide stable services to users.

[0070] Among them, differential data specifically refers to data that is different after comparing production files and grayscale files at the data level, and the difference between the two is determined. The differential data can reflect the differences between the business system in the production environment and the grayscale environment, and then the differential data can be used as a reference in the business system optimization phase, so that the business system can be optimized in a more stable direction.

[0071] In addition, in order to avoid consuming more computing resources in the file consistency detection stage, data statistics processing can be performed through distributed counters. In this embodiment, the specific implementation method is as follows: Calling a distributed counter to perform an incremental counting operation according to the data reading result; wherein, before executing the step of determining the difference data between the production file and the grayscale file in the data storage space according to the detection result, it also includes: calling a distributed counter to perform a decrement counting operation according to the detection result, and when the distributed counter ends counting, executing the step of determining the difference data between the production file and the grayscale file in the data storage space according to the detection result.

[0072] Specifically, the distributed counter refers to a counter used to count the processed row data when the row data is checked for consistency. The increment count operation is a +1 count operation, which is used to indicate that a row of data has been read for processing, and the decrement count operation is a -1 count operation, which is used to indicate that the consistency check of a row of data has been completed. The end of the count indicates that the distributed counter has returned to zero.

[0073] Based on this, considering that the production file and the grayscale file may contain a large amount of data, in order to avoid confusion in the data-level comparison method, each operation can be counted through a distributed counter. In other words, the distributed counter can be called to perform an incremental counting operation based on the data reading result. After the consistency detection process, the distributed counter can be called to perform a decrement counting operation based on the detection result until the distributed counter counts, and then the difference data determination processing operation can be performed.

[0074] For example, when reading the i-th row of data from the 92-PRD file, a +1 operation can be performed through the distributed counter, and the i-th row of data can be stored in comparison bucket No. 8, or when deleting the target row of data from comparison bucket No. 8, a -1 operation can be performed through the distributed counter. Since the data comparison process is a streaming comparison, the distributed counter will always have a non-zero value before the data comparison is completed. After the data comparison is completed, when it is determined that the distributed counter returns to 0, the file reading and comparison are completed, and the difference data between the 92-PRD file and the 92-REFLUX file can be determined by searching all comparison buckets.

[0075] In summary, by using distributed counters to count the reading and comparison of data, it is possible to ensure that file consistency detection at the data level is more stable.

[0076] Furthermore, in order to save memory resources and ensure data comparison efficiency, the comparison operation of the two files can be completed in a streaming data comparison manner. In this embodiment, the specific implementation is as follows: When it is determined according to the detection result that there are still unprocessed line data in the production file and / or the grayscale file, the unprocessed line data is used as the line data, and the step of adding file mark bytes to the line data respectively contained in the production file and the grayscale file is performed; until the line data respectively contained in the production file and the grayscale file are processed, the remaining line data in the data storage space is used as the difference data between the production file and the grayscale file.

[0077] Based on this, in order to achieve the purpose of streaming processing, iterative processing can be used. That is, when it is determined according to the detection results that there are still unprocessed line data in the production file and / or the grayscale file, the unprocessed line data can be used as the line data, and the step of adding file mark bytes to the line data respectively contained in the production file and the grayscale file is performed; and so on, after continuously comparing the read line data, until the line data respectively contained in the production file and the grayscale file are processed, it means that the line data contained in the production file and the grayscale file have been compared, so the remaining line data in the data storage space can be used as the difference data between the production file and the grayscale file.

[0078] In summary, by adopting an iterative processing method to implement streaming data, storage resources can be effectively utilized, excess or insufficient storage resources can be avoided, and resource utilization can be effectively improved.

[0079] On this basis, clustering can be used to determine the data with differences. In this embodiment, the specific implementation method is as follows: At least one non-empty data storage space is determined from a plurality of data storage spaces; the candidate row data stored in each non-empty data storage space is clustered according to the file type, and the difference data between the production file and the grayscale file is determined according to the clustering result.

[0080] Specifically, the non-empty data storage space specifically refers to the data storage space that still stores the remaining row data. Based on this, after the data consistency detection process, at least one non-empty data storage space can be determined from the multiple data storage spaces participating in this data consistency detection; on this basis, considering that the row data that has not been deleted from the non-empty data storage space may be the difference data between the production file and the grayscale file, and considering that there may be multiple non-empty data storage spaces, the candidate row data stored in each non-empty data storage space can be clustered according to the file type, so that the difference data between the production file and the grayscale file can be determined according to the clustering result.

[0081] In actual applications, for all non-empty data storage spaces, the row data stored in the non-empty data storage space can be classified and aggregated according to the file type, such as centralized backup of 92 files. Therefore, the data for which no matching items are found can be determined based on the aggregation results, that is, the difference data between the production environment and the grayscale environment, and then detailed error problems can be output based on the environment tag bytes of the data. For example, there are three data in the backup 92 files in the production environment that do not exist in the grayscale environment, so as to be used in business system optimization.

[0082] Continuing with the above example, after performing the above-mentioned data-level consistency comparison on the row data contained in the 92-PRD file and the 92-REFLUX file respectively, the number of comparison buckets can be retrieved, and the non-empty comparison buckets can be obtained. At this time, the data in the non-empty comparison buckets can be used as difference data and output according to the set format. The bytes appended to the data in the comparison bucket can reflect the environment to which the data belongs, so that in the financial transaction system, system optimization can be completed according to the difference data.

[0083] The file processing method provided in this embodiment can first obtain the production file corresponding to the business system and the grayscale file having a comparison relationship with the production file in order to improve the efficiency of file comparison and reduce the consumption of computing resources; on this basis, in order to achieve row-level data comparison operations, file mark bytes can be added to the row data respectively contained in the production file and the grayscale file, and hash values ​​corresponding to the row data respectively contained in the production file and the grayscale file can be calculated; on this basis, the corresponding data storage space can be selected for each row of data according to the file mark bytes and the hash values, and the data storage space can be tested for data consistency according to each row of data; in this way, after the row data are stored in their respective corresponding data storage spaces, the row data stored in the data storage space can be eliminated according to the row data to be written, so as to determine the difference data between the production file and the grayscale file in the data storage space according to the elimination result; the file consistency test can be completed by streaming comparison of row data, thereby effectively improving the file comparison accuracy and consuming less computing resources.

[0084] See also Figure 4 , Figure 4 A flow chart of an optimization method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0085] Step S402, obtaining a production file associated with a production dimension and a grayscale file associated with a grayscale dimension of the business system; Step S404, adding file mark bytes to the row data respectively contained in the production file and the grayscale file, and calculating the hash value corresponding to the row data; Step S406, selecting a data storage space corresponding to the row data according to the file mark byte and the hash value, and performing a data consistency check on the data storage space according to the row data; Step S408, determining the difference data between the production file and the grayscale file in the data storage space according to the detection result, and optimizing the business system based on the difference data.

[0086] The optimization method provided in this embodiment can refer to the description of the file processing method in the above embodiment, and this embodiment will not be described in detail here.

[0087] The following combination Figure 5 , taking the application of the optimization method provided in this specification in the software upgrade scenario as an example, the optimization method is further explained. Figure 5 A processing flow chart of an optimization method provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0088] Step S502 , initialize a thread pool matching the business system, and call set resources in the thread pool according to the file comparison strategy to execute the file reading task corresponding to the business system.

[0089] Step S504, reading the production file and the grayscale file having a comparison relationship with the production file in a preset system directory according to the task execution result.

[0090] Step S506, adding file mark bytes to the row data respectively contained in the production file and the grayscale file, and calculating the hash value corresponding to the row data.

[0091] Step S508, extracting the file type mark byte from the file mark byte, and calculating the space selection value corresponding to the row data according to the file type mark byte and the hash value.

[0092] Step S510 , selecting a candidate data storage space corresponding to a space identifier matching a space selection value in a data storage space cluster, and using the candidate data storage space as the data storage space corresponding to the row data.

[0093] Step S512: performing data consistency check on the data storage space according to the row data.

[0094] Among them, the data consistency check of the data storage space is performed according to the row data, including: checking whether there is storage row data identical to the row data in the data storage space; if not, storing the row data in the data storage space; if so, extracting the associated file type mark byte and the associated environment mark byte from the file mark byte, and extracting the storage file type mark byte and the storage environment mark byte from the storage file mark byte; based on the associated file type mark byte and the storage file type mark byte, checking whether the file types of the row data and the storage row data are the same; if not, executing the step of storing the row data in the data storage space; if they are the same, checking whether the source environments of the row data and the storage row data are consistent based on the associated environment mark byte and the storage environment mark byte; if they are consistent, executing the step of storing the row data in the data storage space; if not, executing the step of deleting the storage row data in the data storage space.

[0095] Step S514: determine at least one non-empty data storage space from the multiple data storage spaces according to the detection result, and cluster the candidate row data stored in each non-empty data storage space according to the file type.

[0096] Step S516, determining the difference data between the production file and the grayscale file according to the clustering result, and optimizing the business system based on the difference data.

[0097] To summarize, in order to improve the efficiency of file comparison and reduce the consumption of computing resources at the same time, the production file corresponding to the business system and the grayscale file that has a comparison relationship with the production file can be obtained first; on this basis, in order to achieve row-level data comparison operations, file mark bytes can be added to the row data contained in the production file and the grayscale file respectively, and the hash values ​​corresponding to the row data contained in the production file and the grayscale file respectively can be calculated; on this basis, the corresponding data storage space can be selected for each row of data according to the file mark byte and the hash value, and the data storage space can be tested for data consistency according to each row of data; in this way, after the row data is stored in their respective corresponding data storage spaces, the row data stored in the data storage space can be eliminated according to the row data to be written, so as to determine the difference data between the production file and the grayscale file in the data storage space according to the elimination result; the file consistency test is completed by streaming comparison of row data, thereby effectively improving the file comparison accuracy and consuming less computing resources.

[0098] Corresponding to the above method embodiment, this specification also provides a file processing device embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of a file processing device provided by an embodiment of the present specification. Figure 6 As shown, the device comprises: An acquisition module 602 is configured to acquire a production file corresponding to the business system and a grayscale file having a comparison relationship with the production file; An adding module 604 is configured to add a file mark byte to the row data respectively included in the production file and the grayscale file, and calculate a hash value corresponding to the row data; A detection module 606 is configured to select a data storage space corresponding to the row data according to the file mark byte and the hash value, and perform a data consistency detection on the data storage space according to the row data; The determination module 608 is configured to determine the difference data between the production file and the grayscale file in the data storage space according to the detection result.

[0099] In an optional embodiment, adding a file mark byte to any row of data included in the production file or any row of data included in the grayscale file includes: Read the target row data and determine the source file of the target row data; determine the target file type mark byte and the target environment mark byte according to the file type of the source file; use the target file type mark byte and the target environment mark byte as the target file mark bytes corresponding to the target row data, and add them to the target row data.

[0100] In an optional embodiment, the device further comprises: The counting module is configured to call the distributed counter to perform an incremental counting operation according to the data reading result; wherein, before the step of determining the difference data between the production file and the grayscale file in the data storage space according to the detection result is executed, it also includes: calling the distributed counter to perform a decrement counting operation according to the detection result, and when the distributed counter counts, executing the step of determining the difference data between the production file and the grayscale file in the data storage space according to the detection result.

[0101] In an optional embodiment, the detection module 606 is further configured to: A file type tag byte is extracted from the file tag byte, and a space selection value corresponding to the row data is calculated based on the file type tag byte and the hash value; a candidate data storage space corresponding to a space identifier matching the space selection value is selected from a data storage space cluster; and the candidate data storage space is used as the data storage space corresponding to the row data.

[0102] In an optional embodiment, the detection module 606 is further configured to: Detect whether there is storage row data identical to the row data in the data storage space; if not, store the row data in the data storage space; if so, determine the storage file mark byte corresponding to the storage row data, and detect whether the file mark byte and the storage file mark byte are the same; if they are not the same, execute the step of storing the row data in the data storage space; if they are the same, delete the storage row data in the data storage space.

[0103] In an optional embodiment, the detection module 606 is further configured to: Extract the associated file type tag byte and the associated environment tag byte from the file tag byte, and extract the storage file type tag byte and the storage environment tag byte from the storage file tag byte; detect whether the file types of the row data and the storage row data are the same based on the associated file type tag byte and the storage file type tag byte; if they are not the same, execute the step of storing the row data in the data storage space; if they are the same, detect whether the source environments of the row data and the storage row data are consistent based on the associated environment tag byte and the storage environment tag byte; if they are consistent, execute the step of storing the row data in the data storage space; if they are inconsistent, execute the step of deleting the storage row data in the data storage space.

[0104] In an optional embodiment, the determining module 608 is further configured to: When it is determined according to the detection result that there are still unprocessed line data in the production file and / or the grayscale file, the unprocessed line data is used as the line data, and the step of adding file mark bytes to the line data respectively contained in the production file and the grayscale file is performed; until the line data respectively contained in the production file and the grayscale file are processed, the remaining line data in the data storage space is used as the difference data between the production file and the grayscale file.

[0105] In an optional embodiment, when there are multiple data storage spaces, the determining module 608 is further configured to: At least one non-empty data storage space is determined from a plurality of data storage spaces; the candidate row data stored in each non-empty data storage space is clustered according to the file type, and the difference data between the production file and the grayscale file is determined according to the clustering result.

[0106] In an optional embodiment, the acquisition module 602 is further configured to: Initialize a thread pool matching the business system, and call set resources in the thread pool according to the file comparison strategy to execute the file reading task corresponding to the business system; read the production file and the grayscale file having a comparison relationship with the production file in the preset system directory according to the task execution result.

[0107] The file processing device provided in this embodiment can first obtain the production file corresponding to the business system and the grayscale file having a comparison relationship with the production file in order to improve the efficiency of file comparison and reduce the consumption of computing resources; on this basis, in order to achieve row-level data comparison operations, file mark bytes can be added to the row data respectively contained in the production file and the grayscale file, and hash values ​​corresponding to the row data respectively contained in the production file and the grayscale file can be calculated; on this basis, the corresponding data storage space can be selected for each row of data according to the file mark bytes and the hash values, and the data storage space can be checked for data consistency according to each row of data; in this way, after the row data are stored in their respective corresponding data storage spaces, the row data stored in the data storage space can be eliminated according to the row data to be written, so as to determine the difference data between the production file and the grayscale file in the data storage space according to the elimination result; the file consistency test can be completed by streaming comparison of row data, thereby effectively improving the file comparison accuracy and consuming less computing resources.

[0108] The above is a schematic scheme of a file processing device of this embodiment. It should be noted that the technical scheme of the file processing device and the technical scheme of the above-mentioned file processing method belong to the same concept, and the details not described in detail in the technical scheme of the file processing device can be referred to the description of the technical scheme of the above-mentioned file processing method.

[0109] Corresponding to the above method embodiment, this specification also provides an optimization device embodiment, Figure 7 FIG. 1 shows a schematic diagram of the structure of an optimization device provided by an embodiment of this specification. Figure 7 As shown, the device comprises: The acquisition module 702 is configured to acquire the production file associated with the production dimension and the grayscale file associated with the grayscale dimension of the business system; An adding module 704 is configured to add a file mark byte to the row data respectively included in the production file and the grayscale file, and calculate a hash value corresponding to the row data; A detection module 706 is configured to select a data storage space corresponding to the row data according to the file mark byte and the hash value, and perform a data consistency detection on the data storage space according to the row data; The optimization module 708 is configured to determine the difference data between the production file and the grayscale file in the data storage space according to the detection result, and optimize the business system based on the difference data.

[0110] The above is a schematic scheme of an optimization device of this embodiment. It should be noted that the technical scheme of the optimization device and the technical scheme of the above optimization method belong to the same concept, and the details not described in detail in the technical scheme of the optimization device can be referred to the description of the technical scheme of the above optimization method.

[0111] Figure 8 The block diagram of a computing device 800 according to an embodiment of the present specification is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0112] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0113] In one embodiment of the present specification, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0114] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 800 may also be a mobile or stationary server.

[0115] The processor 820 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned file processing method or optimization method.

[0116] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned file processing method or optimization method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned file processing method or optimization method.

[0117] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned file processing method or optimization method.

[0118] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned file processing method or optimization method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned file processing method or optimization method.

[0119] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned file processing method or optimization method when executed by a processor.

[0120] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned file processing method or optimization method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned file processing method or optimization method.

[0121] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0123] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0124] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0125] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A file processing method, characterized in that: include: Obtaining a production file corresponding to the business system and a grayscale file having a comparative relationship with the production file; Adding file mark bytes to the row data respectively contained in the production file and the grayscale file, and calculating the hash value corresponding to the row data; Selecting a data storage space corresponding to the row data according to the file mark byte and the hash value, and performing a data consistency check on the data storage space according to the row data; The difference data between the production file and the grayscale file is determined in the data storage space according to the detection result.

2. The file processing method according to claim 1, characterized in that: Adding a file mark byte to any row of data contained in the production file or any row of data contained in the grayscale file includes: Read the target row data and determine the source file of the target row data; Determine a target file type mark byte and a target environment mark byte according to the file type of the source file; The target file type mark byte and the target environment mark byte are used as target file mark bytes corresponding to the target row data, and are added to the target row data.

3. The file processing method according to claim 2, characterized in that: After the step of reading the target row data is executed, the method further includes: Call the distributed counter to perform incremental counting operations based on the data reading results; Wherein, before the step of determining the difference data between the production file and the grayscale file in the data storage space according to the detection result is performed, it also includes: The distributed counter is called to perform a decrement operation according to the detection result, and when the distributed counter finishes counting, a step of determining the difference data between the production file and the grayscale file in the data storage space according to the detection result is performed.

4. The file processing method according to claim 1, characterized in that: The selecting the data storage space corresponding to the row data according to the file mark byte and the hash value comprises: Extracting a file type mark byte from the file mark byte, and calculating a space selection value corresponding to the row data according to the file type mark byte and the hash value; Selecting a candidate data storage space corresponding to a space identifier matching the space selection value in the data storage space cluster; The candidate data storage space is used as the data storage space corresponding to the row data.

5. The file processing method according to claim 1, characterized in that: The performing data consistency detection on the data storage space according to the row data includes: Detecting whether there is storage row data identical to the row data in the data storage space; If not, storing the row data in the data storage space; If so, determine the storage file mark byte corresponding to the storage row data, and detect whether the file mark byte is the same as the storage file mark byte; If they are not the same, executing the step of storing the row data in the data storage space; If they are the same, the storage row data is deleted in the data storage space.

6. The file processing method according to claim 5, characterized in that: The detecting whether the file mark byte is the same as the stored file mark byte comprises: Extracting the associated file type mark byte and the associated environment mark byte from the file mark byte, and extracting the storage file type mark byte and the storage environment mark byte from the storage file mark byte; Detecting whether the file types of the row data and the stored row data are the same according to the associated file type mark byte and the stored file type mark byte; If they are not the same, executing the step of storing the row data in the data storage space; If they are the same, detecting whether the source environments of the row data and the stored row data are consistent according to the associated environment mark byte and the stored environment mark byte; If they are consistent, executing the step of storing the row data in the data storage space; If they are inconsistent, the step of deleting the storage row data in the data storage space is executed.

7. The file processing method according to claim 5 or 6, characterized in that: Determining the difference data between the production file and the grayscale file in the data storage space according to the detection result includes: When it is determined according to the detection result that there is still unprocessed line data in the production file and / or the grayscale file, the unprocessed line data is used as the line data, and a step of adding a file mark byte to the line data respectively contained in the production file and the grayscale file is performed; When the line data respectively included in the production file and the grayscale file are processed completely, the remaining line data in the data storage space is used as the difference data between the production file and the grayscale file.

8. The file processing method according to claim 1, characterized in that: In the case where there are multiple data storage spaces, determining the difference data between the production file and the grayscale file in the data storage space according to the detection result includes: Determine at least one non-empty data storage space among a plurality of data storage spaces; The candidate row data stored in each non-empty data storage space is clustered according to the file type, and the difference data between the production file and the grayscale file is determined according to the clustering result.

9. The file processing method according to any one of claims 1 to 6, characterized in that: The obtaining of the production file corresponding to the business system and the grayscale file having a comparative relationship with the production file includes: Initialize a thread pool matching the business system, and call set resources in the thread pool according to the file comparison strategy to execute the file reading task corresponding to the business system; The production file and the grayscale file having a comparative relationship with the production file are read in a preset system directory according to the task execution result.

10. An optimization method, characterized in that: include: Obtain the production files of the business system's associated production dimension and the grayscale files of the associated grayscale dimension; Adding file mark bytes to the row data respectively contained in the production file and the grayscale file, and calculating the hash value corresponding to the row data; Selecting a data storage space corresponding to the row data according to the file mark byte and the hash value, and performing a data consistency check on the data storage space according to the row data; The difference data between the production file and the grayscale file is determined in the data storage space according to the detection result, and the business system is optimized based on the difference data.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 10 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 10.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images