File reading method and device, computer equipment and storage medium
By obtaining and configuring the file and application parameters in the file reading method, decomposing the file into fragments and distributing it to multiple applications for processing, the problem of traditional file reading is solved and an efficient and convenient file reading process is achieved.
Patent Information
- Application Number
- CN202411990491.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-04-25
AI Technical Summary
Traditional file reading methods have the problem of low file reading efficiency, which is difficult to meet the needs of large-scale transaction data and multi-file content reading.
By obtaining the configuration parameters of the pending files and each to be connected applications, determining the execution content size and the minimum number of fragments, calculating the number of executable tasks, and distributing the file fragments to the corresponding to be connected applications for reading processing, realizing multi-application, multi-threading and sharding reading.
Improves the efficiency and speed of file reading, and segmented reading facilitates independent processing and monitoring of exceptions, improving convenience and reading quality.
Smart Images
Figure CN119884052A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of file processing, and in particular to a file reading method, device, computer equipment and storage medium. Background Art
[0002] The transaction data accumulated by the Internet system has led to an increasing amount of transaction data for these products, as well as an increasing number of transaction documents with third-party institutions and increasing file content. In order to ensure the integrity of the data, the stable operation of online transactions, and the accuracy and timeliness of transactions, we need to focus on timely, effective and complete reading of file data.
[0003] However, traditional file reading methods have problems such as low file reading efficiency. Summary of the invention
[0004] Based on this, it is necessary to provide a file reading method, device, computer equipment and storage medium that can improve reading efficiency in response to the above technical problems.
[0005] In a first aspect, a file reading method is provided, the method comprising:
[0006] Obtain the configuration parameters of the files to be processed and each application to be connected; the configuration parameters include the application name, the number of units of the minimum deployable unit, and the number of active threads of the node of the minimum deployable unit;
[0007] In response to the configuration operation for the execution content size of the minimum fragment, the execution content size is determined, the file size of the to-be-processed file is determined according to the to-be-processed file, and the total number of minimum fragments is obtained by calculation according to the file size and the execution content size;
[0008] Calculate the number of executable tasks of each application to be connected based on the number of units of each application to be connected and the number of active threads of the corresponding node of the application to be connected to obtain the number of executable tasks of the corresponding application to be connected;
[0009] The file to be processed is segmented according to the execution content size and the minimum total number of segments to obtain corresponding file segments, and each file segment is distributed to the corresponding application to be connected for reading and processing according to the number of executable tasks.
[0010] In one of the embodiments, obtaining configuration parameters of the files to be processed and each application to be connected includes: determining a transmission path in response to a configuration operation on the transmission path; determining a cloud storage server in response to a selection operation on a cloud storage server; obtaining and storing the files to be processed from the cloud storage server via the storage path; and / or storing the files to be processed in response to an upload operation on the files to be processed.
[0011] In one of the embodiments, obtaining configuration parameters of the file to be processed and each application to be connected includes: in response to a selection operation on the application to be connected, determining and connecting each application to be connected; outputting a corresponding configuration parameter feedback instruction to each application to be connected, so that each application to be connected feeds back the corresponding configuration parameter according to the corresponding configuration parameter feedback instruction; and receiving each configuration parameter.
[0012] In one of the embodiments, the method further includes: calculating based on the minimum total number of fragments and the corresponding number of executable tasks of the application to be connected to obtain the estimated number of executable fragments of the node active threads of the corresponding application to be connected; calculating based on the estimated number of executable fragments of the node active threads of each application to be connected and the corresponding number of node active threads of the application to be connected to obtain the estimated number of executable fragments of the minimum deployable unit of the corresponding application to be connected; and displaying the estimated number of executable fragments of each node active thread and the corresponding estimated number of executable fragments of the minimum deployable unit.
[0013] In one of the embodiments, the method further includes: receiving the processing progress percentage of all target file fragments periodically fed back by each application to be connected, and displaying the processing progress percentage of each target file fragment; the target file fragment is a file fragment that has been distributed; obtaining the overall processing progress percentage of the file after statistics based on the minimum total number of fragments and the processing progress percentage of each target file fragment, and displaying the overall processing progress percentage of the file; calculating based on the file size and the overall processing progress percentage of the file to obtain the remaining progress execution time of the file.
[0014] In one of the embodiments, the method further includes: obtaining execution exception information of the target file fragments of each application to be connected; the execution exception information includes the exception fragment identity, exception occurrence time, exception occurrence location, exception problem type and exception repair difficulty; the exception repair difficulty includes easy or difficult; and generating a file execution exception log based on the execution exception information of all target file fragments.
[0015] In one of the embodiments, the method further includes: in response to the difficulty of repairing the exception corresponding to the target file segment being difficult, displaying the execution exception information of the corresponding target file segment; in response to performing a data repair adjustment operation on the target file segment according to the execution exception information of the corresponding target file segment, determining the target file segment after the data repair adjustment, and randomly distributing the target file segment after the data repair adjustment to the corresponding application to be connected for compensatory reading processing.
[0016] In a second aspect, a file reading device is provided, which includes a data acquisition module, a data configuration module, a task calculation module and a shard processing module.
[0017] Among them, the data acquisition module is used to obtain the configuration parameters of the files to be processed and each application to be connected; the configuration parameters include the application name, the number of units of the minimum deployable unit, and the number of active threads of the node of the minimum deployable unit; the data configuration module is used to determine the execution content size in response to the configuration operation on the execution content size of the minimum fragment, determine the file size of the file to be processed according to the file to be processed, and obtain the minimum total number of fragments after calculation based on the file size and the execution content size; the task calculation module is used to calculate based on the number of units of each application to be connected and the number of active threads of the corresponding node of the application to be connected to obtain the corresponding number of executable tasks of the application to be connected; the fragmentation processing module is used to split the file to be processed into fragments according to the execution content size and the minimum total number of fragments to obtain the corresponding file fragments, and distribute each file fragment to the corresponding application to be connected for reading and processing in turn according to the number of executable tasks.
[0018] In a third aspect, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of any method in the above method embodiments are implemented.
[0019] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method in the above method embodiments are implemented.
[0020] The above-mentioned file reading method, device, computer equipment and storage medium obtain configuration parameters of the file to be processed and each application to be connected; the configuration parameters include the application name, the number of units of the minimum deployable unit, and the number of active threads of the node of the minimum deployable unit; then, in response to the configuration operation on the execution content size of the minimum fragment, the execution content size is determined, the file size of the file to be processed is determined according to the file to be processed, and the total number of minimum fragments is obtained after calculation according to the file size and the execution content size; then, according to the number of units of each application to be connected and the number of active threads of the node of the corresponding application to be connected, the number of executable tasks of the corresponding application to be connected is calculated; finally, according to the execution content size and the total number of minimum fragments, the file to be processed is segmented to obtain the corresponding file fragments, and according to the number of executable tasks, each file fragment is distributed to the corresponding application to be connected for reading and processing in turn, thereby realizing multi-application multi-threaded reading and segmented reading after segmenting the file to be processed, thereby improving reading efficiency and reading speed, and segmented reading facilitates independent processing and monitoring of abnormalities, thereby improving convenience. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 An application environment diagram of a file reading method in an embodiment;
[0022] Figure 2 A first flow chart of a file reading method in one embodiment;
[0023] Figure 3 A schematic diagram of a first process of acquiring configuration parameters of a file to be processed and each application to be connected in one embodiment;
[0024] Figure 4 A schematic diagram of a second process for obtaining configuration parameters of a file to be processed and each application to be connected in one embodiment;
[0025] Figure 5 A second flow chart of a file reading method in one embodiment;
[0026] Figure 6 A third flow chart of a file reading method in one embodiment;
[0027] Figure 7 A fourth flow chart of a file reading method in one embodiment;
[0028] Figure 8 is a structural block diagram of a file reading device in an embodiment;
[0029] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0031] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.
[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0033] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first resistor may be referred to as a second resistor, and similarly, a second resistor may be referred to as a first resistor. Both the first resistor and the second resistor are resistors, but they are not the same resistor.
[0034] It can be understood that the “connection” in the following embodiments should be understood as “electrical connection”, “communication connection”, etc. if the connected circuits, modules, units, etc. have electrical signals or data transmission between each other.
[0035] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" etc. specify the presence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof.
[0036] The file reading method provided by this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.
[0037] In one embodiment, Figure 2 As shown, a file reading method is provided, which is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps 201 to 204.
[0038] Step 201: Obtain the files to be processed and the configuration parameters of each application to be connected.
[0039] The configuration parameters include the application name, the number of units of the minimum deployable unit, and the number of active threads of the node of the minimum deployable unit. It can be understood that the minimum deployable unit (Pod) is the smallest basic unit for deployment and management in the kubernetes cluster. Specifically, the server 104 can obtain the configuration parameters of the files to be processed and each application to be connected.
[0040] In one embodiment, if Figure 3As shown, obtaining the files to be processed and the configuration parameters of each application to be connected includes steps 301 to 304.
[0041] Step 301, in response to performing a configuration operation on a transmission path, determining a transmission path;
[0042] Step 302, in response to the cloud storage server selection operation, determining the cloud storage server;
[0043] Step 303, obtaining and storing the file to be processed from the cloud storage server through the storage path;
[0044] And / or, step 304, in response to the upload operation on the file to be processed, storing the file to be processed.
[0045] Specifically, the server 104 determines the transmission path in response to a configuration operation on the transmission path; then, in response to a selection operation on the cloud storage server, it determines the cloud storage server; then, it obtains and stores the files to be processed from the cloud storage server through the storage path; at the same time, in response to an upload operation on the files to be processed, it stores the files to be processed, thereby improving the convenience and efficiency of obtaining the files to be processed.
[0046] In this embodiment, in response to a configuration operation on the transmission path, a transmission path is determined; then, in response to a selection operation on a cloud storage server, a cloud storage server is determined; then, the files to be processed are obtained and stored from the cloud storage server through the storage path; at the same time, in response to an upload operation on the files to be processed, the files to be processed are stored, thereby improving the convenience and efficiency of obtaining the files to be processed.
[0047] In one embodiment, if Figure 4 As shown, obtaining the files to be processed and the configuration parameters of each application to be connected includes steps 401 to 403.
[0048] Step 401, in response to selecting an application to be connected, determining and connecting each application to be connected;
[0049] Step 402, outputting a corresponding configuration parameter feedback instruction to each application to be connected, so that each application to be connected feeds back a corresponding configuration parameter according to the corresponding configuration parameter feedback instruction;
[0050] Step 403: receiving various configuration parameters.
[0051] Specifically, the server 104 determines and connects each application to be connected in response to a selection operation on the application to be connected; then, outputs a corresponding configuration parameter feedback instruction to each application to be connected, so that each application to be connected feeds back the corresponding configuration parameter according to the corresponding configuration parameter feedback instruction; finally, receives the configuration parameters of each application to be connected, thereby improving the efficiency and convenience of obtaining the configuration parameters of each application to be connected.
[0052] In a specific example, the method further includes:
[0053] In response to the minimum number of deployable units being greater than or equal to a default value of the number of units, outputting a minimum number of deployable units exceeded prompt message;
[0054] In response to the number of active threads of the node of the minimum deployable unit being greater than or equal to the default value of the number of active threads of the node, an active thread number exceeding prompt information is output. The above is only a specific example, and it is flexibly set according to user needs in actual applications and is not limited here.
[0055] In this embodiment, in response to a selection operation on an application to be connected, each application to be connected is determined and connected; then, a corresponding configuration parameter feedback instruction is output to each application to be connected, so that each application to be connected feeds back the corresponding configuration parameter according to the corresponding configuration parameter feedback instruction; finally, the configuration parameters of each application to be connected are received, thereby improving the efficiency and convenience of obtaining the configuration parameters of each application to be connected.
[0056] Step 202, in response to the configuration operation on the execution content size of the minimum fragment, the execution content size is determined, the file size of the to-be-processed file is determined according to the to-be-processed file, and the total number of minimum fragments is calculated according to the file size and the execution content size.
[0057] Specifically, the server 104 determines the execution content size in response to the configuration operation on the execution content size of the minimum fragment, determines the file size of the file to be processed according to the file to be processed, and obtains the total number of minimum fragments after calculation based on the file size and the execution content size.
[0058] In a specific example, the execution content size of the smallest segment can be configured as 500k. It is understandable that the total number of the smallest segments can be obtained by dividing the file size by the execution content size. The above is only a specific example, and it can be flexibly set according to user needs in actual applications and is not limited here.
[0059] Step 203 , calculating according to the number of units of each application to be connected and the number of active threads of the corresponding node of the application to be connected, to obtain the number of executable tasks of the corresponding application to be connected.
[0060] Specifically, the server 104 performs calculation according to the number of units of each application to be connected and the number of active threads of the corresponding node of the application to be connected, and thus obtains the number of executable tasks of the corresponding application to be connected.
[0061] In a specific example, the number of executable tasks of the corresponding application to be connected can be obtained by multiplying the number of units of each application to be connected and the number of active threads of the corresponding application to be connected. The above is only a specific example, and it is flexibly set according to user needs in actual applications and is not limited here.
[0062] In one embodiment, if Figure 5 As shown, the method further includes steps 501 to 503.
[0063] Step 501, calculating according to the minimum total number of fragments and the number of executable tasks of the corresponding application to be connected, to obtain the estimated number of executable fragments of the node active thread of the corresponding application to be connected;
[0064] Step 502, calculating the estimated number of executable segments of the node active threads of each application to be connected and the number of active threads of the corresponding node of the application to be connected to obtain the estimated number of executable segments of the minimum deployable unit of the corresponding application to be connected;
[0065] Step 503 : Display the estimated number of executable segments of active threads of each node and the estimated number of executable segments of the corresponding minimum deployable unit.
[0066] Specifically, the server 104 calculates the estimated number of executable fragments of the node active threads of the corresponding application to be connected based on the total number of minimum fragments and the number of executable tasks of the corresponding application to be connected; then, the server 104 calculates the estimated number of executable fragments of the node active threads of each application to be connected and the corresponding number of active threads of the node of the application to be connected to obtain the estimated number of executable fragments of the minimum deployable unit of the corresponding application to be connected; then, the estimated number of executable fragments of the active thread of each node and the estimated number of executable fragments of the corresponding minimum deployable unit are displayed, so as to facilitate the understanding of the estimated workload of the minimum deployable unit corresponding to each application to be connected and the estimated workload of the active threads of each node of the corresponding minimum deployable unit, and to facilitate the fragment allocation, workload and work efficiency statistics.
[0067] In a specific example, the total number of minimum fragments divided by the number of executable tasks of the corresponding application to be connected can obtain the estimated number of executable fragments of the node active threads of the corresponding application to be connected. The estimated number of executable fragments of the node active threads of each application to be connected multiplied by the number of node active threads of the corresponding application to be connected can obtain the estimated number of executable fragments of the minimum deployable unit of the corresponding application to be connected. The above is only a specific example. In actual application, it is flexibly set according to user needs and is not limited here.
[0068] In this embodiment, the estimated number of executable fragments of the node active threads of the corresponding application to be connected is calculated based on the minimum total number of fragments and the number of executable tasks of the corresponding application to be connected; then, the estimated number of executable fragments of the node active threads of each application to be connected and the corresponding number of active threads of the node to be connected are calculated to obtain the estimated number of executable fragments of the minimum deployable unit of the corresponding application to be connected; then, the estimated number of executable fragments of each node active thread and the corresponding estimated number of executable fragments of the minimum deployable unit are displayed, so as to facilitate understanding of the estimated workload of each minimum deployable unit corresponding to each application to be connected and the estimated workload of each node active thread of the corresponding minimum deployable unit, and to facilitate fragment allocation, workload and work efficiency statistics.
[0069] Step 204 , the file to be processed is segmented according to the execution content size and the minimum total number of segments to obtain corresponding file segments, and each file segment is distributed to the corresponding application to be connected for reading and processing in sequence according to the number of executable tasks.
[0070] Specifically, the server 104 splits the file to be processed into segments according to the execution content size and the minimum total number of segments to obtain corresponding file segments, and distributes each file segment to the corresponding application to be connected for reading and processing in turn according to the number of executable tasks. This simultaneously realizes multi-application multi-threaded reading and segmented reading after splitting the file to be processed, thereby improving reading efficiency and speed. Segmented reading facilitates independent processing and monitoring of exceptions, thereby improving convenience.
[0071] Based on this, the above-mentioned file reading method obtains the configuration parameters of the file to be processed and each application to be connected; the configuration parameters include the application name, the number of units of the minimum deployable unit, and the number of active threads of the node of the minimum deployable unit; then, in response to the configuration operation on the execution content size of the minimum fragment, the execution content size is determined, the file size of the file to be processed is determined according to the file to be processed, and the total number of minimum fragments is obtained after calculation according to the file size and the execution content size; then, according to the number of units of each application to be connected and the number of active threads of the node of the corresponding application to be connected, the number of executable tasks of the corresponding application to be connected is calculated; finally, according to the execution content size and the total number of minimum fragments, the file to be processed is segmented to obtain the corresponding file fragments, and according to the number of executable tasks, each file fragment is distributed to the corresponding application to be connected for reading and processing in turn, thereby realizing multi-application multi-threaded reading and segmented reading after segmenting the file to be processed, thereby improving reading efficiency and reading speed, and segmented reading facilitates independent processing and monitoring of exceptions, thereby improving convenience.
[0072] In one embodiment, if Figure 6 As shown, the method further includes steps 601 to 603.
[0073] Step 601, receiving the processing progress percentage of all target file segments regularly fed back by each application to be connected, and displaying the processing progress percentage of each target file segment;
[0074] Step 602, after statistics are collected based on the total number of minimum segments and the processing progress percentage of each target file segment, the overall processing progress percentage of the file is obtained, and the overall processing progress percentage of the file is displayed;
[0075] Step 603, calculate according to the file size and the percentage of the overall processing progress of the file to obtain the remaining progress execution time of the file.
[0076] Among them, the target file fragment is the file fragment that has been distributed. The server 104 receives the processing progress percentage of all target file fragments fed back by each application to be connected at regular intervals, and displays the processing progress percentage of each target file fragment; then, the overall processing progress percentage of the file is obtained after statistics are performed based on the minimum total number of fragments and the processing progress percentage of each target file fragment, and the overall processing progress percentage of the file is displayed; then, the remaining progress execution time of the file is calculated based on the file size and the overall processing progress percentage of the file, which is convenient for fine-grained monitoring and overall monitoring of the file reading progress, improves the convenience of file reading, and is also conducive to the staff to adjust the task according to the file reading progress.
[0077] In this embodiment, the processing progress percentage of all target file fragments periodically fed back by each application to be connected is received, and the processing progress percentage of each target file fragment is displayed; then, the overall processing progress percentage of the file is obtained by statistics based on the minimum total number of fragments and the processing progress percentage of each target file fragment, and the overall processing progress percentage of the file is displayed; then, the remaining progress execution time of the file is obtained based on the file size and the overall processing progress percentage of the file, which facilitates the refined monitoring and overall monitoring of the file reading progress, improves the convenience of file reading, and is also beneficial for staff to adjust tasks according to the file reading progress.
[0078] In a specific example, the method further includes:
[0079] According to the processing progress percentage of all target file fragments, the number of executed fragments of the minimum deployable unit of each application to be connected is counted;
[0080] In response to the first target minimum deployable unit, determine whether all file fragments have been distributed; wherein the first target minimum deployable unit is the minimum deployable unit whose number of executed fragments of the corresponding minimum deployable unit of the application to be connected is equal to the number of fragments expected to be executed by the corresponding minimum deployable unit of the application to be connected; the second target minimum deployable unit is the minimum deployable unit whose number of executed fragments of the corresponding minimum deployable unit of the application to be connected is less than the number of fragments expected to be executed by the corresponding minimum deployable unit of the application to be connected;
[0081] If all file segments are distributed, forward the second target file segments of the first preset ratio from the second target minimum deployable unit to the first target minimum deployable unit for reading and processing; the second target file segments are target file segments that have been distributed to the second target minimum deployable unit and have not completed reading and processing;
[0082] If all file fragments have not been distributed, the undistributed file fragments of the second preset ratio are distributed to the first target minimum deployable unit for reading and processing, which facilitates automatic task allocation during file reading. The above is only a specific example. In actual application, it is flexibly set according to user needs and is not limited here.
[0083] In one embodiment, if Figure 7 As shown, the method further includes step 701 to step 702.
[0084] Step 701, obtaining execution exception information of target file segments of each application to be connected;
[0085] Step 702: Generate a file execution exception log according to the execution exception information of all target file segments.
[0086] The execution exception information includes the abnormal fragment identity, abnormal occurrence time, abnormal occurrence location, abnormal problem type and abnormal repair difficulty; the abnormal repair difficulty includes easy or difficult. Specifically, the server 104 obtains the execution exception information of the target file fragments of each application to be connected; then, according to the execution exception information of all target file fragments, a file execution exception log is generated, which is convenient for monitoring the abnormal information during the file reading process, thereby improving the efficiency and convenience of monitoring.
[0087] In a specific example, the method also includes: monitoring the load of the minimum deployable unit of each application to be connected, determining that a load abnormality occurs in the minimum deployable unit of the application to be connected when the load of the minimum deployable unit of the application to be connected is greater than or equal to a load threshold, and outputting load adjustment prompt information; performing statistics on the load adjustment prompt information to generate a load abnormality statistical log. The above is only a specific example. In actual applications, it is flexibly set according to user needs and is not limited here.
[0088] In a specific example, when the exception flag position 1 is identified, the execution exception information of the target file segment of each application to be connected is obtained. Among them, the exception flag position 1 indicates that the target file segment of the application to be connected has an execution exception. The initial value of the exception flag position is 0. The above is only a specific example. In actual application, it is flexibly set according to user needs and is not limited here.
[0089] In this embodiment, the execution exception information of the target file fragments of each application to be connected is obtained; then, a file execution exception log is generated based on the execution exception information of all target file fragments, which facilitates the monitoring of exception information during the file reading process and improves the efficiency and convenience of monitoring.
[0090] In one embodiment, the method further comprises:
[0091] In response to the difficulty of repairing the exception corresponding to the target file segment being difficult, displaying execution exception information of the corresponding target file segment;
[0092] In response to performing a data repair adjustment operation on the target file segment according to the execution exception information of the corresponding target file segment, the target file segment after the data repair adjustment is determined, and the target file segment after the data repair adjustment is randomly distributed to the corresponding application to be connected for compensation reading processing.
[0093] Specifically, in response to the difficulty of repairing the exception corresponding to the target file segment being difficult, the server 104 displays the execution exception information of the corresponding target file segment; then, in response to performing a data repair adjustment operation on the target file segment according to the execution exception information of the corresponding target file segment, the target file segment after the data repair adjustment is determined, and the target file segment after the data repair adjustment is randomly distributed to the corresponding application to be connected for compensatory reading processing, thereby improving the convenience of the file reading process and also improving the file reading quality.
[0094] In this embodiment, in response to the difficulty of repairing the exception corresponding to the target file segment being difficult, the execution exception information of the corresponding target file segment is displayed; then, in response to performing a data repair adjustment operation on the target file segment according to the execution exception information of the corresponding target file segment, the target file segment after the data repair adjustment is determined, and the target file segment after the data repair adjustment is randomly distributed to the corresponding application to be connected for compensatory reading processing, thereby improving the convenience of the file reading process and also improving the file reading quality.
[0095] In a specific example, the method further includes:
[0096] In response to the exception repair difficulty corresponding to the target file segment being easy, the target file segment is automatically repaired to obtain the target file segment after automatic data repair, and the target file segment after automatic data repair is randomly distributed to the corresponding application to be connected for compensation reading processing.
[0097] Specifically, in response to the difficulty of the exception repair corresponding to the target file segment being easy, the server 104 performs automatic data repair processing on the target file segment to obtain the target file segment after the automatic data repair processing, and randomly distributes the target file segment after the automatic data repair processing to the corresponding application to be connected for compensatory reading processing, thereby improving the convenience of the file reading process and also improving the file reading quality.
[0098] In this embodiment, in response to the abnormal repair difficulty corresponding to the target file segment being easy, the target file segment is automatically repaired to obtain the target file segment after automatic data repair, and the target file segment after automatic data repair is randomly distributed to the corresponding application to be connected for compensatory reading processing, thereby improving the convenience of the file reading process and the quality of file reading.
[0099] It should be understood that although Figure 2-7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-7 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0100] Second, as Figure 8 As shown, a file reading device is provided, which includes a data acquisition module 810, a data configuration module 820, a task calculation module 830 and a shard processing module 840.
[0101] Among them, the data acquisition module 810 is used to obtain the configuration parameters of the files to be processed and each application to be connected; the configuration parameters include the application name, the number of units of the minimum deployable unit, and the number of active threads of the node of the minimum deployable unit; the data configuration module 820 is used to determine the execution content size in response to the configuration operation on the execution content size of the minimum fragment, determine the file size of the file to be processed based on the file to be processed, and obtain the total number of minimum fragments after calculation based on the file size and the execution content size; the task calculation module 830 is used to calculate based on the number of units of each application to be connected and the number of active threads of the corresponding node of the application to be connected to obtain the number of executable tasks of the corresponding application to be connected; the fragmentation processing module 840 is used to split the file to be processed into fragments according to the execution content size and the total number of minimum fragments to obtain the corresponding file fragments, and distribute each file fragment to the corresponding application to be connected for reading and processing in turn according to the number of executable tasks.
[0102] In one embodiment, the data acquisition module 810 includes a first data acquisition unit.
[0103] Among them, the first data acquisition unit is used to determine the transmission path in response to a configuration operation on the transmission path; the first data acquisition unit is used to determine the cloud storage server in response to a selection operation on the cloud storage server; the first data acquisition unit is used to obtain and store the files to be processed from the cloud storage server through the storage path; and / or, the first data acquisition unit is used to store the files to be processed in response to an upload operation on the files to be processed.
[0104] In one embodiment, the data acquisition module 810 includes a second data acquisition unit.
[0105] Among them, the second data acquisition unit is used to determine and connect each application to be connected in response to a selection operation on the application to be connected; the second data acquisition unit is used to output a corresponding configuration parameter feedback instruction to each application to be connected, so that each application to be connected can feedback the corresponding configuration parameter according to the corresponding configuration parameter feedback instruction; the second data acquisition unit is used to receive each configuration parameter.
[0106] In one of the embodiments, the device further comprises an executable fragment number calculation module.
[0107] Among them, the executable fragment number calculation module is used to calculate according to the minimum total number of fragments and the corresponding number of executable tasks of the application to be connected, and obtain the estimated number of executable fragments of the node active threads of the corresponding application to be connected; the executable fragment number calculation module is used to calculate according to the estimated number of executable fragments of the node active threads of each application to be connected and the corresponding number of node active threads of the application to be connected, and obtain the estimated number of executable fragments of the minimum deployable unit of the corresponding application to be connected; the executable fragment number calculation module is used to display the estimated number of executable fragments of each node active thread and the corresponding estimated number of executable fragments of the minimum deployable unit.
[0108] In one of the embodiments, the device further includes a progress monitoring module.
[0109] Among them, the progress monitoring module is used to receive the processing progress percentage of all target file fragments periodically fed back by each application to be connected, and display the processing progress percentage of each target file fragment; the target file fragment is the file fragment that has been distributed; the progress monitoring module is used to obtain the overall file processing progress percentage after statistics based on the minimum total number of fragments and the processing progress percentage of each target file fragment, and display the overall file processing progress percentage; the progress monitoring module is used to calculate based on the file size and the overall file processing progress percentage to obtain the remaining progress execution time of the file.
[0110] In one of the embodiments, the device further includes an exception log generating module.
[0111] Among them, the exception log generation module is used to obtain the execution exception information of the target file fragments of each application to be connected; the execution exception information includes the exception fragment identity, the exception occurrence time, the exception occurrence location, the exception problem type and the exception repair difficulty; the exception repair difficulty includes easy or difficult; the exception log generation module is used to generate a file execution exception log according to the execution exception information of all target file fragments.
[0112] In one of the embodiments, the device further includes a data compensation module.
[0113] Among them, the data compensation module is used to display the execution exception information of the corresponding target file segment in response to the difficulty of the exception repair corresponding to the target file segment being difficult; the data compensation module is used to perform a data repair adjustment operation on the target file segment in response to the execution exception information of the corresponding target file segment, determine the target file segment after the data repair adjustment, and randomly distribute the target file segment after the data repair adjustment to the corresponding application to be connected for compensation reading processing.
[0114] For the specific definition of the file reading device, please refer to the definition of the file reading method above, which will not be repeated here. Each module in the above-mentioned file reading device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0115] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store files to be processed and configuration parameters of each application to be connected. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a file reading method is implemented.
[0116] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0117] In a third aspect, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps of any method in the above method embodiments are implemented.
[0118] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method in the above method embodiments are implemented.
[0119] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0121] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A file reading method, the method comprising: Obtain the files to be processed and the configuration parameters of each application to be connected; The configuration parameters include the application name, the number of units of the minimum deployable unit, and the number of active threads of the node of the minimum deployable unit; In response to a configuration operation for an execution content size of a minimum segment, the execution content size is determined, a file size of the to-be-processed file is determined according to the to-be-processed file, and a total number of minimum segments is obtained by calculating according to the file size and the execution content size; Calculating according to the number of units of each of the applications to be connected and the number of active threads of the corresponding node of the application to be connected, to obtain the number of executable tasks of the corresponding application to be connected; The to-be-processed file is segmented according to the execution content size and the minimum total number of segments to obtain corresponding file segments, and each file segment is distributed to the corresponding to-be-connected application in turn for reading and processing according to the number of each executable task.
2. The method according to claim 1, characterized in that The obtaining of the configuration parameters of the files to be processed and the applications to be connected includes: In response to performing a configuration operation on a transmission path, determining the transmission path; In response to performing a selection operation on a cloud storage server, determining the cloud storage server; Acquire and store the to-be-processed file from the cloud storage server through the storage path; And / or, in response to performing an upload operation on the file to be processed, storing the file to be processed.
3. The method according to claim 1, characterized in that The obtaining of the configuration parameters of the files to be processed and the applications to be connected includes: In response to a selection operation on the applications to be connected, determining and connecting each of the applications to be connected; Outputting a corresponding configuration parameter feedback instruction to each of the applications to be connected, so that each of the applications to be connected feeds back the corresponding configuration parameter according to the corresponding configuration parameter feedback instruction; The configuration parameters are received.
4. The method according to claim 1, characterized in that The method further comprises: Calculate the number of executable fragments of the node active thread of the corresponding application to be connected according to the minimum total number of fragments and the number of executable tasks of the corresponding application to be connected to obtain the number of executable fragments expected by the node active thread of the corresponding application to be connected; Calculate the number of executable fragments of the node active thread of each application to be connected and the number of active threads of the node of the corresponding application to be connected to obtain the number of executable fragments of the minimum deployable unit of the corresponding application to be connected; The estimated number of executable segments of each active thread of the node and the corresponding estimated number of executable segments of the minimum deployable unit are displayed.
5. The method according to claim 1, characterized in that The method further comprises: Receive the processing progress percentage of all target file segments regularly fed back by each of the to-be-connected applications, and display the processing progress percentage of each of the target file segments; the target file segments are the file segments that have completed distribution; Obtaining the overall processing progress percentage of the file after statistics are taken based on the total number of the minimum fragments and the processing progress percentage of each of the target file fragments, and displaying the overall processing progress percentage of the file; The remaining progress execution time of the file is obtained by calculating according to the file size and the percentage of the overall processing progress of the file.
6. The method according to claim 5, characterized in that The method further comprises: Obtaining execution exception information of the target file segments of each of the applications to be connected; the execution exception information includes an abnormal segment identity, abnormal occurrence time, abnormal occurrence location, abnormal problem type, and abnormal repair difficulty; the abnormal repair difficulty includes easy or difficult; A file execution exception log is generated according to the execution exception information of all the target file segments.
7. The method according to claim 6, characterized in that The method further comprises: In response to the exception repair difficulty corresponding to the target file segment being the difficult level, displaying execution exception information of the corresponding target file segment; In response to performing a data repair adjustment operation on the target file segment according to the execution exception information of the corresponding target file segment, the target file segment after the data repair adjustment is determined, and the target file segment after the data repair adjustment is randomly distributed to the corresponding application to be connected for compensation reading processing.
8. A file reading device, characterized in that: The device comprises: A data acquisition module, used to acquire the configuration parameters of the files to be processed and each application to be connected; the configuration parameters include the application name, the number of units of the minimum deployable unit, and the number of active threads of the node of the minimum deployable unit; a data configuration module, configured to determine the execution content size in response to a configuration operation on the execution content size of the minimum fragment, determine the file size of the to-be-processed file according to the to-be-processed file, and obtain the total number of minimum fragments after calculation according to the file size and the execution content size; A task calculation module, used to calculate according to the number of units of each application to be connected and the number of active threads of the corresponding node of the application to be connected, to obtain the number of executable tasks of the corresponding application to be connected; The fragment processing module is used to fragment the to-be-processed file according to the execution content size and the minimum total number of fragments to obtain corresponding file fragments, and distribute each file fragment to the corresponding to-be-connected application for reading and processing in turn according to the number of executable tasks.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.