Data processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention relates to a data processing method and device, equipment, a storage medium and a program product. The method comprises the steps that in response to a received task request sent by front-end equipment, a target file is imported into a data queue according to a target address in the task request, and the target address is the address of the target file in a specified server obtained after the front-end equipment uploads the target file to the specified server; calling a business application to enable the business application to export the target file from the data queue; and dynamically adjusting data import configuration and data export configuration according to the residence time of the target file in the data queue. Therefore, the overall performance of the application can be fully utilized without being limited by the disk capacity of a local machine.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for data processing. Background Art

[0002] In current Internet businesses, importing data from files into software systems is a common scenario. Usually, the technical implementation includes steps such as file uploading, temporary storage, reading, parsing, and importing. In this process, the front-end page transfers the file to the back-end application at one time, and the back-end application temporarily stores it on disk, and then loads it into memory to execute the import logic. However, when the file is large and multiple tasks are performed in parallel, the file uploaded to the back-end at one time is very likely to exceed the memory. Even if it can be saved to disk normally, the disk capacity will become the bottleneck of the entire import function. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for data processing is provided. The method includes: in response to receiving a task request sent by a front-end device, importing a target file into a data queue according to a target address in the task request, where the target address is the address of the target file in the designated server obtained after the front-end device uploads the target file to the designated server; by calling a business application, the business application is enabled to export the target file from the data queue; and dynamically adjusting the data import configuration and the data export configuration according to the residence time of the target file in the data queue.

[0004] In a second aspect of the present disclosure, a device for data processing is provided. The device comprises:

[0005] The import module is configured to import the target file into the data queue according to the target address in the task request in response to receiving the task request sent by the front-end device; the export module is configured to enable the business application to export the target file from the data queue by calling the business application; the adjustment module is configured to dynamically adjust the data import configuration and the data export configuration according to the residence time of the target file in the data queue.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the electronic device executes the method of the first aspect.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided, which includes a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the following, in conjunction with the accompanying drawings and with reference to the following detailed description, the above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 A schematic diagram of a system for data processing according to some embodiments of the present disclosure is shown;

[0013] Figure 3A A schematic diagram showing an intelligent control system for a system for controlling data processing according to some embodiments of the present disclosure is shown;

[0014] Figure 3B A schematic diagram of the control process of the intelligent control system according to some embodiments of the present disclosure is shown.

[0015] Figure 4 A schematic diagram showing an adjustment process for a service interface according to some embodiments of the present disclosure is shown;

[0016] Figure 5 A schematic diagram showing a data processing flow according to some embodiments of the present disclosure;

[0017] Figure 6 A schematic structural block diagram showing an apparatus for data processing according to some embodiments of the present disclosure; and

[0018] Figure 7 A block diagram of a computing device is shown in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0021] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation to the embodiments of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the embodiments of the present disclosure.

[0026] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0027] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0028] In some scenarios, a scenario of processing data into a software system can be provided. Usually, the technical implementation of this scenario includes steps such as file uploading, temporary storage, reading, parsing, and importing. In this scenario, the front-end device sends the file directly to the back-end device, which is then processed by the application in the back-end device. After receiving the file, the application in the back-end device needs to temporarily store the file to the local disk first, and then load it into the memory to execute the import logic.

[0029] In such data processing scenarios, if the file is large and multiple tasks are performed in parallel, OOM (Out of Memory) failures are likely to occur. Even if the data can be saved to the local disk normally, the capacity of the local disk will become the bottleneck of the entire import function. In addition, the files on the local disk can only be imported by the local service node, and the low load of other nodes cannot be utilized, making it difficult to take advantage of the distributed system.

[0030] The embodiment of the present disclosure provides a data processing scheme. According to the scheme, in response to receiving a task request sent by a front-end device, the target file is imported into a data queue according to the target address in the task request. The target address is the address of the target file in the specified server obtained after the front-end device uploads the target file to the specified server. The business application is called to enable the business application to export the target file from the data queue. The residence time of the target file in the data queue is recorded. And according to the residence time, the data import configuration and the data export configuration are dynamically adjusted, and the data import configuration and the data export configuration are associated with the residence time of the target file in the data queue.

[0031] Therefore, the embodiments of the present disclosure can provide a solution that can send files to a server for storage, which are then imported by the server into system software in a back-end device, and can perform real-time deployment during the process of importing files into the software system.

[0032] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, the example environment 100 may include a backend device 110. The backend device 110 may run an application 120 that supports data processing. The application 120 may be any suitable type of application that supports data processing, and the embodiments of the present disclosure are not limited in this respect. The front-end device 140 may interact with the application 120 via the backend device 110 and / or its attached devices.

[0033] In some embodiments, the backend device 110 communicates with the server 130 to provide services for the application 120. The backend device 110 can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a handheld computer, a portable game terminal, a VR / AR device, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio receiver, an e-book device, a game device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the backend device 110 can also support any type of interface for the user (such as a "wearable" circuit, etc.).

[0034] In the example environment 100 , if the application 120 is active, the backend device 110 may communicate with the cloud server to obtain the target file from the cloud server 130 .

[0035] The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 130 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, etc. The server 130 may provide background services for the application 120 that supports content presentation in the backend device 110.

[0036] A communication connection may be established between the server 130 and the backend device 110. The communication connection may be established in a wired manner or a wireless manner. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In the embodiments of the present disclosure, the server 130 and the backend device 110 may implement signaling interaction through the communication connection between the two.

[0037] The front-end device 140 may be any type of mobile terminal or physical server, including but not limited to the devices exemplified for the back-end device 110 and the server 130. The front-end device is used as the starting end for file transmission. In some embodiments, after the front-end device 140 selects the target file to be transmitted, it pulls the configuration information when the file is imported from the server 130 to the server 130. Then the front-end device 140 connects with the server 130 and uploads the target file. When the front-end device 140 uploads the target file, it can also obtain the target address corresponding to the target file in the server 130.

[0038] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0039] Figure 2 1 shows a schematic diagram of a system 200 for data processing according to some embodiments of the present disclosure. The system 200 for data processing can be run at the backend device 110 to implement the data processing function of the application 120. For ease of discussion, reference will be made to Figure 1 The system 200 is described with reference to the environment 100 of FIG.

[0040] like Figure 2As shown, the system 200 includes a producer 210, a data queue 220, and a consumer 230. The producer 210 is used to stream read files from the server 130 according to the target address, and import the read target files into the data queue 220 in the memory. The consumer 230 remotely calls the business interface to execute the export logic, so that the business application exports the target file from the data queue 220 and further imports it into the application system.

[0041] In the data processing task, multiple producers 210 concurrently stream the target file from the server 130. For example, when the target file is a table file, the producer 210 uses a streaming processing tool to read the table file content line by line to ensure that the data stays in the memory for the shortest time. The records of the read target file are parsed and immediately stored in the data queue 220 for export by the consumer 230. Multiple consumers 230 export the target file from the data queue 220 in parallel.

[0042] In some embodiments, the server 130 uses object storage service (OSS) technology to store and manage target files. OSS provides high availability, high reliability and elastic expansion capabilities. Using OSS can reduce the demand and cost of local storage and provide persistent storage and backup functions. Its elastic expansion characteristics are suitable for processing large files and high concurrent access. OSS is used to store uploaded table files (such as CSV, Excel) to ensure that data is safe and reliable during the import process.

[0043] In the above system 200, due to the use of streaming reading, the system 200 does not need to load the entire target file into the memory, thereby avoiding the bottleneck of the upper limit of the memory. The consumer 230 obtains the records of the target file from the data queue 220 and requests the business application to process it. This method not only effectively utilizes the memory, but also ensures the stability and efficiency of the system when processing large files.

[0044] Figure 3A A schematic diagram of an intelligent monitoring system 300A for regulating a data processing system according to some embodiments of the present disclosure is shown.

[0045] Reference Figure 3A The monitoring objects of the intelligent decision-making component 260 include JVM memory 250, target file 240, producer 210, data queue 220, consumer 230 and business interface 270.

[0046] In some embodiments, the collection indicators of the JVM memory 250 by the intelligent decision component 260 include the maximum memory, used memory, available memory, garbage collection times and time. The collection indicators of the producer 210 by the intelligent decision component 260 include the number of concurrently read files, the average reading speed, and the time taken to read the specified number of rows. The collection indicators of the consumer 230 by the intelligent decision component 260 include the number of concurrent file exports, the average processing speed, and the consumption time of a batch of data (the time from entering the data queue 220 to the complete export). The collection indicators of the data queue 220 by the intelligent decision component 260 include the maximum capacity of the queue, the current used capacity, and the average residence time of the target file in the queue. The collection indicators of the business interface 270 by the intelligent decision component 260 include the time taken for the application node to call the business interface 270, the percentage of the execution time of the business logic in the overall request, the amount of data transmitted in one call, the success rate, the response time, and the data integrity.

[0047] Additionally, the intelligent decision component 260 is also used to track the exception type, number of occurrences, and error stack.

[0048] Furthermore, the purpose of the intelligent decision-making component 260 collecting the above indicators is to adjust the number of producers, the size of the data queue, the number of consumers, and the number of transmissions and concurrency of the business interface to the optimal values.

[0049] Figure 3B The schematic diagram of the control process 300B of the intelligent control system according to the present disclosure is shown. The logic of how the system obtains the optimal value when executing a file import task is shown. Figure 3B In the process 300B shown, the intelligent decision component 260 uses the residence time of the data in the data queue 220 as an indicator to measure the performance of the import service. A target value (i.e., a preset threshold) can be given based on historical experience, and the intelligent decision component 260 adjusts the data processing system around the target value. The intelligent decision component 260 expands the production capacity by increasing the size of the data queue, increasing the number of producer concurrency, and increasing the number of queues each time. It expands the consumption capacity by increasing the number of consumer concurrency and increasing the amount of data transmitted at one time. The intelligent decision component 260 determines the relationship between the current residence time and the target value to decide whether to expand the production capacity or the consumption capacity. If the queue residence time cannot be reduced by expanding the consumption capacity for multiple times, it means that the current consumption capacity has reached a bottleneck and the production capacity needs to be reduced to achieve system balance.

[0050] like Figure 3B As shown, in block 310, the intelligent decision component 260 obtains the residence time of the target file in the data queue. The intelligent decision component 260 obtains the import time and export time of the target file into the data queue respectively, and calculates the residence time according to the import time and export time.

[0051] In box 320, the intelligent decision component 260 determines the size of the residence time and the preset threshold. In box 340, if the residence time is less than the preset threshold, it means that the production capacity is low and the consumption capacity is high, and the data import configuration needs to be adjusted to increase at least one of the data queue memory, the number of concurrent reads of the target file, and the number of batches of the target file into the queue. In box 330, if the residence time is greater than the preset threshold, it means that the production capacity is high and the consumption capacity is low, and the export configuration needs to be adjusted positively, that is, to increase at least one of the number of concurrent business application requests and the number of target files exported each time.

[0052] In box 350, the intelligent decision component determines whether the residence time is reduced after the data export configuration is adjusted forwardly, and further determines the number of times the residence time remains unchanged after the data export configuration is adjusted. When the residence time is reduced after the data export configuration is adjusted forwardly, it means that the consumer's consumption capacity can be further improved, so the data export configuration can continue to be adjusted forwardly. When the residence time remains unchanged, and the number of times the residence time remains unchanged reaches the preset number of cycles, it means that the consumer's consumption capacity has reached a bottleneck, and the producer's generation capacity needs to be reduced. The intelligent decision component needs to reversely adjust the data import configuration, that is, reduce at least one of the data queue memory, the number of concurrent reads of the target file, and the number of batches of the target file entering the queue.

[0053] In the embodiments of the present disclosure, the specific processes of the forward and reverse adjustment data import configuration and the forward and reverse adjustment data export configuration are not further disclosed, but the disclosure scope of the present disclosure is not implied.

[0054] Figure 4 A schematic diagram of an adjustment process 400 for a service interface according to some embodiments of the present disclosure is shown.

[0055] like Figure 4 As shown, in block 410, the intelligent decision-making component monitors and collects the total amount of data imported into the software system from the data queue by the business interface and the time of importing the data.

[0056] In block 420 , the intelligent decision-making component calculates the ratio of the total amount of data imported into each application node corresponding to the business interface to the import time.

[0057] At block 430 , the intelligent decision component increases at least one of the number of concurrency of consumers to the application nodes and the number of batch imports per application node.

[0058] In block 440, the intelligent decision component determines the relationship between the newly obtained ratio and the ratio before adjustment. If the ratio increases, it means that the performance of the application node has not reached a bottleneck, and it can further return to block 430 to allow the increase of at least one of the number of concurrent applications of the consumer and the number of batch imports of each application node. If the ratio decreases, it means that the performance of the application node has reached a bottleneck, and the last adjustment needs to be withdrawn, so jump to block 450.

[0059] At block 450 , the intelligent decision making component enables reducing at least one of a number of concurrent consumers to the application node and a number of batch imports per application node.

[0060] In the embodiments of the present disclosure, how to adjust the concurrent number of application nodes and the batch import number is not further disclosed, and the disclosure scope of the present disclosure is not implied.

[0061] Figure 5 FIG. 5 is a schematic diagram of a data processing process 500 according to some embodiments of the present disclosure. The process 500 may be performed at least partially in Figure 1 The process is executed at the backend device 110 shown.

[0062] In block 510, the backend device 110 imports the target file into the data queue according to the target address in the task request in response to receiving the task request sent by the frontend device, wherein the target address is the address of the target file in the designated server obtained after the frontend device uploads the target file to the designated server.

[0063] In block 520 , the backend device 110 calls the business application so that the business application exports the target file from the data queue.

[0064] In block 530 , the backend device 110 dynamically adjusts the data import configuration and the data export configuration according to the residence time of the target file in the data queue.

[0065] In some embodiments, data import configuration and data export configuration are dynamically adjusted according to the residence time of the target file in the data queue, including: obtaining a task request and creating a new import task process; parsing the task request in the import task process to obtain the target address; streaming reading of the target file from the target address; and importing the target file into the data queue.

[0066] In some embodiments, exporting the target file from the data queue includes: creating an export task process corresponding to the business application; and exporting the target file from the data queue in the export task process.

[0067] In some embodiments, process 500 further includes: recording the import time of the target file into the data queue; recording the export time of the target file from the data queue; and obtaining the residence time of the target file in the data queue based on the import time and the export time.

[0068] In some embodiments, dynamically adjusting the data import configuration and the data export configuration according to the residence time includes: determining whether the residence time is greater than a preset threshold; if it is determined that the residence time is less than the preset threshold, increasing at least one of the following in the data import configuration: data queue memory, the number of concurrent reads of the target file, and the number of batches of the target file being queued. Or if it is determined that the residence time is greater than the preset threshold, increasing at least one of the following in the data export configuration: the number of concurrent business application requests, and the number of target files exported each time.

[0069] In some embodiments, process 500 further includes: determining whether the dwell time changes. If it is determined that the dwell time remains unchanged after adjusting the data receiving configuration for a preset number of times, reducing at least one of the following items in the data reading configuration: data queue memory, concurrent reading of target files, and batch queue number of target files.

[0070] In some embodiments, process 500 also includes: obtaining the import configuration of the application node of the business application; calculating the ratio of the amount of imported data corresponding to the application node to the import time; and determining the change in the ratio after increasing at least one of the number of concurrent business application requests and the number of target files exported each time.

[0071] In some embodiments, process 500 also includes: if it is determined that the ratio has increased, allowing at least one of the number of concurrent business application requests and the number of target files exported each time to continue to increase; or if it is determined that the ratio has decreased, allowing at least one of the number of concurrent business application requests and the number of target files exported each time to decrease.

[0072] In some embodiments, process 500 further includes: saving the adjusted import configuration of the business application node as an initial strategy for the next data processing task.

[0073] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 A schematic structural block diagram of an apparatus 600 for data processing according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in a backend device 110. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0074] like Figure 6As shown, the device 600 includes an import module 610, an export module 620 and an adjustment module 630. The import module 610 is configured to import the target file into the data queue according to the target address in the task request in response to receiving the task request sent by the front-end device. The export module is configured to enable the business application to export the target file from the data queue by calling the business application. The adjustment module is configured to dynamically adjust the data import configuration and the data export configuration according to the residence time of the target file in the data queue.

[0075] In some embodiments, the device 600 is further configured to: obtain a task request and create a new import task process; parse the task request in the import task process to obtain a target address; stream read the target file from the target address; and import the target file into the data queue.

[0076] In some embodiments, the device 600 is further configured to: create a new export task process corresponding to the business application; and export the target file from the data queue in the export task process.

[0077] In some embodiments, the device 600 is also configured to: record the import time of the target file entering the data queue; record the export time of the target file exporting the data queue; and obtain the residence time of the target file in the data queue based on the import time and the export time.

[0078] In some embodiments, the device 600 is further configured to: dynamically adjust the data import configuration and the data export configuration according to the residence time, including: determining whether the residence time is greater than a preset threshold; if it is determined that the residence time is less than the preset threshold, then increase at least one of the following items in the data import configuration: data queue memory, the number of concurrent reads of the target file, and the number of batches of the target file to be queued. Or if it is determined that the residence time is greater than the preset threshold, then increase at least one of the following items in the data export configuration: the number of concurrent business application requests, and the number of target files exported each time.

[0079] In some embodiments, the device 600 is further configured to: determine whether the dwell time changes. If it is determined that the dwell time remains unchanged after adjusting the data receiving configuration for a preset number of times, then reduce at least one of the following items in the data reading configuration: data queue memory, number of concurrent reads of the target file, and number of batches of the target file to be queued.

[0080] In some embodiments, the device 600 is also configured to: obtain the import configuration of the application node of the business application; calculate the ratio of the amount of imported data corresponding to the application node to the import time; and determine the change in the ratio after increasing at least one of the concurrent number of business application requests and the number of target files exported each time.

[0081] In some embodiments, the device 600 is further configured to: if it is determined that the ratio has increased, allow at least one of the number of concurrent business application requests and the number of target files exported each time to continue to increase; or if it is determined that the ratio has decreased, allow at least one of the number of concurrent business application requests and the number of target files exported each time to decrease.

[0082] In some embodiments, the apparatus 600 is further configured to: save the adjusted import configuration of the business application node as the initial strategy for the next data processing task.

[0083] The units and / or modules included in the device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 600 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0084] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 1 The backend device 110 in.

[0085] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to implement Figure 1 Backend device 110 or Figure 6 Device 600.

[0086] like Figure 7As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0087] The electronic device 700 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 can be a volatile memory (e.g., register, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. The storage device 730 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 700.

[0088] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0089] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0090] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0091] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0092] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0094] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0095] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0096] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A data processing method, comprising: In response to receiving a task request sent by a front-end device, importing a target file into a data queue according to a target address in the task request, wherein the target address is an address of the target file in the designated server obtained after the front-end device uploads the target file to the designated server; By calling a business application, the business application is enabled to export the target file from the data queue; as well as The data import configuration and the data export configuration are dynamically adjusted according to the residence time of the target file in the data queue.

2. The method according to claim 1, wherein importing the target file into the data queue comprises: Obtain the task request and create a new import task process; Parsing the task request in the import task process to obtain the target address; Streaming read the target file from the target address; as well as Import the target file into the data queue.

3. The method according to claim 1, wherein exporting the target file from the data queue comprises: Create a new export task process corresponding to the business application; as well as The target file is exported from the data queue during the export task process.

4. The method according to claim 1, further comprising: Recording the import time of the target file into the data queue; Recording the export time of the target file exporting the data queue; as well as The residence time of the target file in the data queue is obtained according to the import time and the export time.

5. The method according to claim 1, wherein dynamically adjusting the data import configuration and the data export configuration according to the residence time comprises: Determining whether the dwell time is greater than a preset threshold; If it is determined that the residence time is less than the preset threshold, at least one of the following is increased in the data import configuration: Data queue memory, The number of concurrent reads of the target file. The number of batches of target files to be enqueued; or If it is determined that the residence time is greater than the preset threshold, at least one of the following is increased in the data export configuration: The number of concurrent business application requests, The number of target files to export each time.

6. The method according to claim 5, further comprising: determining whether the dwell time changes; as well as If it is determined that the dwell time remains unchanged after adjusting the data receiving configuration for a preset number of times, at least one of the following items is reduced in the data reading configuration: The data queue memory, The number of concurrent reads of the target file, The number of batches of target files to be enqueued.

7. The method according to claim 5, further comprising: Get the import configuration of the application node of the business application; Calculating the ratio of the amount of imported data corresponding to the application node to the import time; as well as After increasing at least one of the number of concurrent business application requests and the number of target files exported each time, determining a change in the ratio.

8. The method according to claim 7, further comprising: If it is determined that the ratio increases, at least one of the number of concurrent business application requests and the number of target files exported each time is allowed to continue to increase; or If it is determined that the ratio is reduced, it is allowed to reduce at least one of the number of concurrent business application requests and the number of target files exported each time.

9. The method according to claim 7, further comprising: The adjusted import configuration of the business application node is saved as the initial strategy for the next data processing task.

10. A device for data processing, comprising: An import module, configured to, in response to receiving a task request sent by a front-end device, import the target file into a data queue according to a target address in the task request; An export module is configured to call a business application so that the business application exports the target file from the data queue; The adjustment module is configured to dynamically adjust the data import configuration and the data export configuration according to the residence time of the target file in the data queue.

11. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the method according to any one of claims 1 to 9.

Images