Data processing method and device and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a data processing method and device and a storage medium, the method is applied to a cloud storage server, and the method comprises the following steps: in response to at least one data processing request, learning a network transmission state and a data file corresponding to each data processing request through an artificial intelligence learning model, the data processing request comprises a data uploading request and / or a data downloading request, and the network transmission state comprises a network resource state and a data file which is being transmitted in a current network, creating a plurality of coroutines in one thread based on the characteristics of the file to be transmitted, and processing each data file through each coroutine, through the arrangement of the artificial intelligence learning model, the network condition can be accurately predicted, more intelligent scheduling is realized, uploading or downloading of each data file is not affected by each other through the arrangement of each coroutine, and efficient, stable and flexible task scheduling is realized.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and provides a data processing method, apparatus and storage medium. Background Technology

[0002] With the development of the Internet, cloud storage has become a key technology for data management and access. How to efficiently upload and download files to cloud storage is an important challenge.

[0003] To achieve more efficient file uploads and downloads to cloud storage, thread pools, callback mechanisms, and RxJava are commonly used. However, thread creation and destruction incur performance overhead, concurrency control is not flexible enough, and improper thread pool size configuration can lead to resource waste or delayed task processing. Furthermore, handling multiple concurrent and synchronized tasks requires mechanisms such as locks and semaphores, increasing system complexity. In callback mechanisms, multiple nested asynchronous operations can result in excessively deep callback functions, significantly reducing system maintainability. Error handling requires additional logic and is prone to incomplete processing. In RxJava-based implementations, RxJava operators incur performance overhead, potentially impacting performance when processing large amounts of data, especially in complex scenarios with high processing complexity.

[0004] In summary, there is currently no efficient method for uploading files to cloud storage or downloading files from cloud storage. Summary of the Invention

[0005] This application provides a data processing method, apparatus, and storage medium to efficiently and flexibly realize the uploading and downloading of data files.

[0006] The specific technical solution provided in this application is as follows:

[0007] In a first aspect, embodiments of this application provide a data processing method, applied to a cloud storage server, comprising:

[0008] In response to at least one data processing request, the network transmission status and the data files corresponding to each data processing request are learned through an artificial intelligence learning model to obtain the characteristics of the file to be transmitted. The data processing request includes a data upload request and / or a data download request, and the network transmission status includes the network resource status and the data files currently being transmitted in the network.

[0009] Multiple coroutines are created in a single thread based on the characteristics of the files to be transferred, and each coroutine processes the data files.

[0010] Optionally, in response to at least one data processing request, an artificial intelligence learning model is used to learn the network transmission status and the data files corresponding to each data processing request to obtain the characteristics of the file to be transmitted, including:

[0011] Receive and respond to at least one data processing request from each computing device, and collect network resource status and data files currently being transmitted in the network through an artificial intelligence learning model;

[0012] Artificial intelligence learning models are used to learn the network resource status, the data files currently being transmitted in the network, and the data files corresponding to each data processing request, so as to obtain the number of data files that the network is allowed to transmit and the priority of each data file. The number of data files that the network is allowed to transmit represents the number of data files that the network resource status allows to transmit in addition to the data files that are currently being transmitted in the network.

[0013] The number of licenses is determined based on the number of data files allowed to be transmitted on the network, the data files currently being transmitted on the network, and a preset transmission threshold.

[0014] Each license is matched with each data processing request based on the priority of each data file;

[0015] The one-to-one matching relationship between licenses and data processing requests is determined as the characteristics of the file to be transferred.

[0016] Optionally, the number of licenses is determined based on the number of data files allowed to be transmitted on the network, the data files currently being transmitted on the network, and a preset transmission threshold, including:

[0017] Calculate the sum of the number of data files that the network is allowed to transmit and the number of data files that are currently being transmitted on the network;

[0018] Compare whether the sum is less than a preset transmission quantity threshold;

[0019] If so, the number of licenses is determined based on the difference between the transmission quantity threshold and the sum value;

[0020] Otherwise, the processing of each data processing request will be suspended.

[0021] Optionally, after pausing the processing of each data processing request, the process also includes:

[0022] Monitor the processing progress of each coroutine on each data file;

[0023] If the processing progress indicates that at least one data file has been processed, then the sum is re-compared to see if it is less than the preset transmission quantity threshold.

[0024] Optionally, each license is matched one-to-one with each data processing request based on the priority of each data file, including:

[0025] Sort the data processing requests according to the priority of each data file from high to low.

[0026] Each sorted data processing request is matched with a corresponding license until the number of remaining licenses or the number of remaining data processing requests is zero.

[0027] Optionally, multiple coroutines are created in a single thread based on the characteristics of the files to be transferred, and each coroutine processes each data file, including:

[0028] The number of coroutines to be created is determined based on the number of matching relationships included in the characteristics of the file to be transmitted;

[0029] In the thread corresponding to the data file currently being transmitted on the network, multiple coroutines are created using a coroutine container, with the number of coroutines being the same. Each coroutine is configured independently.

[0030] Different coroutines are used to process the data files corresponding to the data processing requests in the matching relationship.

[0031] Optionally, the artificial intelligence learning model is trained in the following way:

[0032] The historical network transmission status and historical data processing requests are provided as input parameters to the original artificial intelligence learning model. The original artificial intelligence learning model is trained in an iterative manner. The historical network transmission status is the network transmission status within the most recent sampling period obtained according to the preset sampling period, and the historical data processing requests are those received within the most recent sampling period.

[0033] In each iteration, the convergence condition is determined based on the loss value of that round. If the convergence condition is met, the original artificial intelligence learning model is considered converged. The process of each iteration is as follows:

[0034] The original artificial intelligence learning model learns from the input parameters to obtain the characteristics of historically transmitted files;

[0035] The loss value for this round is determined based on the characteristics of historically transmitted files, and the current model parameters of the original artificial intelligence learning model are adjusted based on this loss value.

[0036] Secondly, embodiments of this application also provide a data processing apparatus, including:

[0037] The response unit is used to respond to at least one data processing request by learning the network transmission status and the data files corresponding to each data processing request through an artificial intelligence learning model to obtain the characteristics of the file to be transmitted. The data processing request includes a data upload request and / or a data download request, and the network transmission status includes the network resource status and the data files currently being transmitted in the network.

[0038] The processing unit is used to create multiple coroutines in a thread based on the characteristics of the files to be transferred, and to process each data file through each coroutine.

[0039] Thirdly, a server includes:

[0040] Memory, used to store executable instructions;

[0041] A processor for reading and executing executable instructions stored in memory to implement the method as described in any of the first aspects.

[0042] Fourthly, a computer-readable storage medium, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any of the first aspects above.

[0043] The beneficial effects of this application are as follows:

[0044] In summary, the data processing method, apparatus, and storage medium provided in this application embodiment are applied to a cloud storage server. The method includes: responding to at least one data processing request, learning the network transmission status and the data files corresponding to each data processing request through an artificial intelligence learning model to obtain the characteristics of the file to be transmitted. The data processing request includes a data upload request and / or a data download request. The network transmission status includes the network resource status and the data files currently being transmitted on the network. Based on the characteristics of the file to be transmitted, multiple coroutines are created in a thread, and each coroutine processes each data file. The aforementioned artificial intelligence learning model can accurately predict network conditions and analyze data processing requests. Based on achieving more intelligent scheduling and optimization, the aforementioned coroutines achieve asynchronous processing in a synchronous manner within the same thread, thus ensuring that the upload or download of each data file does not affect each other, achieving efficient, stable, and flexible task scheduling.

[0045] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a schematic diagram of the system architecture for data processing in the embodiments of this application;

[0048] Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of a process for determining the characteristics of a file to be transmitted in an embodiment of this application;

[0050] Figure 4 This is a schematic diagram illustrating a process for determining the number of licenses in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of a process for processing data files according to their processing progress in an embodiment of this application.

[0052] Figure 6 This is a schematic diagram illustrating a process for matching licenses based on the priority of data files in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram illustrating a process of creating a coroutine and processing a data file using a coroutine, as described in an embodiment of this application.

[0054] Figure 8 This is a schematic diagram of the logical architecture of a data processing device according to an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the physical architecture of a server according to an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0058] The preferred embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0059] See Figure 1 As shown in the embodiments of this application, the system includes at least one computing device and a cloud storage server. Figure 1 In the diagram, computing device 1, computing device 2, ..., computing device n represent devices that upload data files to or download data files from the cloud storage server. In the following description, the process of uploading or downloading will be referred to as data processing.

[0060] In this embodiment of the application, the data processing method is mainly implemented on the cloud storage server side, which will be described in detail below.

[0061] See Figure 2 As shown in the embodiments of this application, a specific data processing flow is as follows:

[0062] Step 201: In response to at least one data processing request, the network transmission status and the data files corresponding to each data processing request are learned through an artificial intelligence learning model to obtain the characteristics of the file to be transmitted. The data processing request includes data upload request and / or data download request, and the network transmission status includes network resource status and the data files currently being transmitted in the network.

[0063] During implementation, when a computing device needs to upload a data file to the cloud storage server, or when a computing device needs to download a data file already stored on the cloud storage server, the aforementioned computing device will generate a data processing request, namely a data upload request and / or a data download request. Considering the large number of computing devices simultaneously interacting with the cloud storage server, the aforementioned data processing request will also carry the data file that the computing device wants to upload or download, for example, the identifier and storage address of the data file to be uploaded or downloaded. It should be noted that the term "data file" used subsequently is a general term, encompassing both the data file corresponding to the data processing request and the data files currently being processed (including uploads and downloads) on the network.

[0064] Unlike related technologies where computing devices directly interact with cloud storage servers after generating data processing requests, this embodiment first evaluates the current network transmission environment and related data files using a pre-trained artificial intelligence learning model. The following section explains how the artificial intelligence learning model is trained.

[0065] See Figure 2 As shown, the artificial intelligence learning model is trained in the following way:

[0066] (1) The historical network transmission status and historical data processing requests are provided as input parameters to the original artificial intelligence learning model, and the original artificial intelligence learning model is trained in an iterative manner. The historical network transmission status is the network transmission status in the most recent sampling period obtained according to the preset sampling period, and the historical data processing requests are those received in the most recent sampling period.

[0067] In this embodiment, the data used to train the artificial intelligence learning model comes from historical data, namely historical network transmission status and historical data processing requests. It should be noted that, in order to ensure that the trained artificial intelligence learning model can accurately reflect the latest network conditions and data file transmission status, the aforementioned historical network transmission status is the network transmission status within the most recent sampling period obtained according to a preset sampling period, and the historical data processing requests are received within the most recent sampling period. That is, with the sampling period as the interval, the data within the most recent sampling period is used to train the artificial intelligence learning model.

[0068] Furthermore, the historical network transmission status and historical data processing requests within the most recent sampling period are provided to the original artificial intelligence learning model to initiate training of the original artificial intelligence learning model.

[0069] In each iteration, the convergence condition is determined based on the loss value of that round. If the convergence condition is met, the original artificial intelligence learning model is considered to have converged. The process of each iteration is as follows:

[0070] (2) Enable the original artificial intelligence learning model to learn from the input parameters and obtain the characteristics of historically transmitted files;

[0071] Each time the original artificial intelligence learning model receives historical network transmission status and historical data processing requests within a sampling period, it uses the structure and parameters in the model to learn the input parameters, thereby obtaining the characteristics of historically transmitted files. The aforementioned original artificial intelligence learning model can be one of the following learning models: linear regression, decision tree, neural network, etc., without specific limitations here.

[0072] (3) Determine the current loss value based on the characteristics of historical transmitted files, and adjust the current model parameters of the original artificial intelligence learning model based on the current loss value.

[0073] After obtaining the characteristics of historical transmitted files, the characteristics of historical transmitted files are compared with the pre-set standard transmitted file characteristics to obtain the current loss value. The magnitude of the current loss value determines the method and range of adjustment for the current model parameters of the original artificial intelligence learning model.

[0074] The above process is repeated in each training round until the loss value of the current round meets the convergence condition. That is, the convergence condition is met when the difference between the latest obtained historical transmission file features and the pre-set standard transmission file features is very small. At this point, the current model parameters of the original artificial intelligence learning model are no longer adjusted, and the original artificial intelligence learning model obtained after training is determined as the artificial intelligence learning model.

[0075] See Figure 3 As shown, the method described above for obtaining the characteristics of the file to be transmitted through an artificial intelligence learning model specifically includes:

[0076] Step 2011: Receive and respond to at least one data processing request from each computing device, and collect network resource status and data files currently being transmitted in the network through an artificial intelligence learning model;

[0077] After the artificial intelligence learning model is trained, the cloud storage server receives and responds to multiple data processing requests from various computing devices. Because, in the current network environment, in addition to the data processing requests from these computing devices, there are other data files being processed between other computing devices. For example, besides the data processing requests mentioned above, there are also data files currently being transmitted over the network, i.e., data files being downloaded or uploaded by the other computing devices.

[0078] Based on this, during implementation, the aforementioned artificial intelligence learning model is used to collect network resource status, such as network bandwidth, latency, packet loss rate, etc., and the aforementioned artificial intelligence learning model is also used to collect data files currently being transmitted in the network, such as data file A being downloaded by computing device a and data file B being uploaded by computing device b.

[0079] Step 2012: Use an artificial intelligence learning model to learn the network resource status, the data files currently being transmitted in the network, and the data files corresponding to each data processing request, to obtain the number of data files that the network allows to be transmitted and the priority of each data file. The number of data files that the network allows to be transmitted represents the number of data files that the network resource status allows to be transmitted in addition to the data files that are currently being transmitted in the network.

[0080] After collecting the network resource status and the data files currently being transmitted in the network, an artificial intelligence learning model is used to learn the network resource status, the data files currently being transmitted in the network, and the data files corresponding to each data processing request. This allows us to obtain the number of data files that the network resource status allows to be transmitted, in addition to the data files currently being transmitted in the network, based on factors such as network bandwidth and packet loss rate.

[0081] Meanwhile, considering the varying importance of different data files, after determining the number of data files that the network is allowed to transmit, the AI ​​learning model will also prioritize each data file corresponding to each current data processing request. Given the limited number of data files that the network is allowed to transmit, this will allow higher-priority data files to be processed first.

[0082] Step 2013: Determine the number of licenses based on the number of data files allowed to be transmitted on the network, the data files currently being transmitted on the network, and a preset transmission quantity threshold;

[0083] After using an artificial intelligence learning model to determine the number of data files that the network is allowed to transmit, the system will further determine the number of licenses based on a preset transmission threshold, thereby making more efficient use of network resources.

[0084] See Figure 4 As shown, the number of licenses is determined based on the number of data files allowed to be transmitted on the network, the data files currently being transmitted on the network, and a preset transmission threshold, including:

[0085] Step 20131: Calculate the sum of the number of data files allowed to be transmitted on the network and the number of data files currently being transmitted on the network;

[0086] The aforementioned preset transmission threshold typically represents the maximum number of upload or download tasks that the network resources are allowed to handle. During implementation, this preset transmission threshold is used to further limit the actual number of data processing requests that can be processed.

[0087] During implementation, the sum of the number of data files that the network is allowed to transmit and the number of data files that are currently being transmitted on the network is first calculated. This sum is used to measure the total network resources used for all data processing.

[0088] Step 20132: Compare whether the sum is less than the preset transmission quantity threshold; if yes, proceed to step 20133; otherwise, proceed to step 20134.

[0089] After determining the above sum, the sum is further compared with the preset transmission quantity threshold. Since the current network can efficiently process data processing requests only when the sum is less than the preset transmission quantity threshold, the implementation process directly compares whether the sum is less than the preset transmission quantity threshold.

[0090] Step 20133: Determine the number of licenses based on the difference between the transmission quantity threshold and the sum value;

[0091] If the sum is less than the preset transmission quantity threshold, the sum is further subtracted from the transmission quantity threshold to obtain the difference. In this case, the number of licenses is set to be the same as the number of differences.

[0092] Step 20134: Pause the processing of each data processing request.

[0093] If the sum of the above values ​​is not less than the preset transmission quantity threshold, it means that the current network conditions are not able to effectively process the data processing requests. In this case, the processing of each data processing request is suspended, and the network conditions are waited for other data files to be processed or for the network conditions to improve.

[0094] See Figure 5 As shown, after pausing the processing of each data processing request, the following also applies:

[0095] Step 20135: Monitor the processing progress of each coroutine on each data file;

[0096] To improve the efficiency of processing data processing requests, after pausing the processing of each data processing request, it is also necessary to monitor the processing progress of other data files in real time. Considering that each data file in this application is uploaded or downloaded through a corresponding coroutine, during implementation, each coroutine will be monitored in real time to obtain the processing progress of the corresponding data file.

[0097] Step 20136: If the processing progress indicates that at least one data file has been processed, then re-compare whether the sum value is less than the preset transmission quantity threshold.

[0098] During implementation, if the monitored processing progress indicates that one or more data files have been processed, for example, when a data file is uploaded in multiple data blocks, after each data block is uploaded, the percentage of data sent relative to the total size of the data file is calculated to obtain the upload progress percentage, which is then displayed. In this case, the cloud storage server will again compare the above sum with a preset transmission quantity threshold, and then redetermine the number of licenses based on the comparison result. The process here will not be described in detail.

[0099] Step 2014: Match each license with each data processing request one by one based on the priority of each data file.

[0100] During implementation, after determining the priority of each data file and the number of licenses, the licenses are issued to the matching data processing requests, thereby processing the data files corresponding to the data processing requests.

[0101] See Figure 6 As shown, each license is matched with each data processing request one by one based on the priority of each data file, including:

[0102] Step 20141: Sort the data processing requests according to the priority of each data file from high to low;

[0103] During implementation, after determining the priority of each data file, the data files are sorted in descending order of priority. Then, the data processing requests are also sorted in descending order of priority to determine the order in which the data processing requests are processed.

[0104] Step 20142: Match each sorted data processing request with a corresponding license until the number of remaining licenses is zero or the number of remaining data processing requests is zero.

[0105] During implementation, after determining the order in which the data processing requests are processed, licenses are matched one-to-one with each sorted data processing request in chronological order. During the matching process, if the number of data processing requests is too large, the matching process stops when the number of remaining licenses reaches zero; similarly, if the number of licenses is too large, the matching process stops when the number of remaining data processing requests reaches zero.

[0106] Step 2015: Determine the one-to-one matching relationship between the license and the data processing request as the characteristics of the file to be transferred.

[0107] In this embodiment of the application, the one-to-one matching relationship between the license and the data processing request determined above represents the data file to be uploaded or downloaded, and the matching relationship is determined as the characteristics of the file to be transmitted.

[0108] Step 202: Create multiple coroutines in a thread based on the characteristics of the files to be transferred, and process each data file through each coroutine.

[0109] Unlike related technologies that use multiple threads to handle multiple data processing requests, this embodiment uses only one thread, thereby greatly reducing the overhead of task creation and destruction. When multiple data processing requests are received, multiple coroutines are created within the aforementioned thread for processing, thus enabling more efficient utilization of system and network resources when handling a large number of concurrent upload and download tasks.

[0110] For details, please refer to Figure 7 As shown, multiple coroutines are created in a single thread based on the characteristics of the files to be transferred, and each coroutine processes the data files, including:

[0111] Step 2021: Determine the number of coroutines to be created based on the number of matching relationships included in the characteristics of the file to be transferred;

[0112] After determining the characteristics of the file to be transmitted, the characteristics are parsed to obtain the number of matching relationships included in the characteristics of the file to be transmitted, and then the number of coroutines to be created is determined based on the number of matching relationships.

[0113] Step 2022: In the thread corresponding to the data file currently being transmitted on the network, create multiple coroutines with the same number of coroutines using a coroutine container, where each coroutine is configured independently;

[0114] During implementation, to achieve effective management of coroutines, a coroutine container, CoroutineScope, is created. CoroutineScope provides unified management of the coroutine lifecycle. For a data processing request, a new coroutine is started within CoroutineScope to execute the data processing task, thus creating multiple coroutines corresponding to the number of matching relationships mentioned above.

[0115] Furthermore, each coroutine is configured independently, utilizing SupervisorJob to prevent the failure of one data processing request from affecting other tasks. This ensures that the failure of each data processing request does not propagate to other data processing requests, thereby guaranteeing the stability of the entire system. For example, if the upload of a data file corresponding to a data processing request fails due to network fluctuations, it will not affect other data files that are being uploaded or downloaded in progress.

[0116] It should also be noted that the aforementioned coroutine container is created in the thread corresponding to the data file currently being transmitted on the network, thereby maximizing the saving of system overhead.

[0117] Step 2023: Process the data files corresponding to the data processing requests in the matching relationship using different coroutines.

[0118] After creating a coroutine for the data processing request, the coroutine can be used to process the data file corresponding to the data processing request in the above matching relationship, that is, to upload or download the data file corresponding to the data processing request.

[0119] It should be noted that, in order to improve processing efficiency, the above data files can be processed in blocks. For example, the data files can be divided into multiple small blocks in advance, and then the data can be sent to the cloud storage server in blocks sequentially over the network.

[0120] In addition, to facilitate the uploading or downloading of the aforementioned data files, a network module is installed between the computing device and the cloud storage server. This network module is primarily responsible for the following tasks: 1. HTTPS Request Sending: For data upload requests, MultipartBody is used to encapsulate the data file content. MultipartBody encapsulates the data file along with other form data into a request body conforming to the HTTPS protocol, which is then sent to the cloud storage server. For data download requests, the network module sets request headers to obtain metadata about the data file, such as file size and file type. For example, by setting the Range request header, the function of resuming interrupted data file downloads can be implemented. 2. Response Parsing: Upon receiving the response from the cloud storage server, the module parses the response based on its status code and content. If the status code indicates a successful request (e.g., 200), the data processing request is processed. For data upload requests, the module may check if the uploaded result is correct; for data download requests, the downloaded data file is written to the local storage of the computing device. If the status code indicates a failed request (e.g., 404, 500), the error information is passed to the relevant error handling module for processing. 3. Network Connection Management: Configure connection timeout, read timeout, and write timeout parameters in the OkHttp computing device to ensure network connection stability. Connection timeout parameters prevent prolonged waiting times for connection establishment; read timeout parameters prevent prolonged periods of unresponsiveness when reading data; and write timeout parameters ensure that writing data is not blocked due to network issues. Upon task completion or cancellation, the network connection is closed, releasing related resources and preventing resource leaks.

[0121] The error handling module is described below. Its function is to capture and handle various exceptions that occur during data file upload and download, ensuring system robustness. Specifically, it includes: 1. Error Capture: During data file upload and download, try-catch blocks are used to capture potential exceptions. Examples include network exceptions (such as network interruption, connection timeout) and file read / write exceptions (such as file not found, insufficient permissions). By capturing these exceptions, problems can be identified and addressed promptly. 2. Error Classification: Errors are classified according to their type. For example, network exceptions are categorized into connection timeout and request failure; file read / write exceptions are categorized into file not found and insufficient permissions. Different types of errors can be handled differently, improving the targeted nature of error handling. 3. Error Feedback: Error information is encapsulated into a unified error object and passed to the computing device through a callback function. The error object contains the error type, error information, etc., allowing the computing device to perform appropriate processing. For example, if it's a network connection timeout error, the computing device can prompt the user to check the network connection; if it's a file permission insufficient error, it can prompt the user to modify the file permissions.

[0122] In addition, a task management module is included. This module manages the entire lifecycle of uploaded and downloaded data files, providing flexible task operation functions. Specifically, it includes: 1. Task Creation: Based on the data files provided by the computing device and the configuration information of the cloud storage server, upload or download task objects are created. The task object contains basic task information, such as task ID, file path, cloud storage service address, and task status. Each task has a unique task (one task is one coroutine) ID for easy subsequent task management and tracking. 2. Task Status Management: Enumeration types are used to represent task status, such as RUNNING, PAUSED, CANCELLED, and COMPLETED. During task execution, the task status is updated based on different operations. For example, when a task starts execution, the status is set to RUNNING; when the user pauses the task, the status is set to PAUSED; and when the task completes, the status is set to COMPLETED. 3. Task Operations: Interfaces for pausing, resuming, and canceling tasks are provided. When the computing device invokes the pause operation, the current progress information is saved, and then the task execution is paused. For example, during file upload, the positions of uploaded data blocks are recorded so that uploads can resume from those positions during subsequent recovery. When the computing device invokes a recovery operation, it will continue executing the task based on the saved progress information. When the user invokes a cancel operation, the task execution will be stopped immediately, and related resources will be released, such as closing network connections and releasing file handles.

[0123] This application embodiment also includes an API module, which provides a simple and easy-to-use interface with the following functions: 1. Upload API: Provides interfaces for uploading single data files and batch uploading multiple data files. The API module can be used to pass in the data file path, cloud storage service configuration information, and a progress callback function. Calling this interface will create an upload task. 2. Download API: Provides interfaces for uploading single data files and batch downloading multiple data files. During implementation, the remote address of the data file, the local save path, and a progress callback function can be passed in. Similarly, calling this interface will create a download task. 3. Task Management API: Provides interfaces for pausing, resuming, and canceling tasks. Operations can be performed on specified tasks based on their IDs.

[0124] Based on the same inventive concept, see [reference] Figure 8 As shown in the figure, this application provides a data processing apparatus, including:

[0125] The response unit 801 is used to respond to at least one data processing request by learning the network transmission status and the data files corresponding to each data processing request through an artificial intelligence learning model to obtain the characteristics of the file to be transmitted. The data processing request includes a data upload request and / or a data download request, and the network transmission status includes the network resource status and the data files currently being transmitted in the network.

[0126] The processing unit 802 is used to create multiple coroutines in a thread based on the characteristics of the files to be transferred, and to process each data file through each coroutine.

[0127] Based on the same inventive concept, see [reference] Figure 9 As shown, this application embodiment provides a server, including: a memory 901 for storing executable instructions; and a processor 902 for reading and executing the executable instructions stored in the memory, and executing any of the methods described in the first aspect above.

[0128] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when the instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any of the first aspects above.

[0129] In summary, the data processing method, apparatus, and storage medium provided in this application embodiment are applied to a cloud storage server. The method includes: responding to at least one data processing request, learning the network transmission status and the data files corresponding to each data processing request through an artificial intelligence learning model to obtain the characteristics of the file to be transmitted. The data processing request includes a data upload request and / or a data download request. The network transmission status includes the network resource status and the data files currently being transmitted on the network. Based on the characteristics of the file to be transmitted, multiple coroutines are created in a thread, and each coroutine processes each data file. The aforementioned artificial intelligence learning model can accurately predict network conditions and analyze data processing requests. Based on achieving more intelligent scheduling and optimization, the aforementioned coroutines achieve asynchronous processing in a synchronous manner within the same thread, thus ensuring that the upload or download of each data file does not affect each other, achieving efficient, stable, and flexible task scheduling.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program product systems. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product system implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program product systems according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method is applied to cloud storage servers and includes: In response to at least one data processing request, an artificial intelligence learning model is used to learn the network transmission status and the data files corresponding to each data processing request to obtain the characteristics of the file to be transmitted. The data processing request includes a data upload request and / or a data download request, and the network transmission status includes the network resource status and the data files currently being transmitted in the network. Based on the characteristics of the file to be transferred, multiple coroutines are created in a thread, and each coroutine processes the data file.

2. The method as described in claim 1, characterized in that, In response to at least one data processing request, the system learns the network transmission status and the data files corresponding to each data processing request using an artificial intelligence learning model to obtain the characteristics of the file to be transmitted, including: Receive and respond to at least one data processing request from each computing device, and collect the network resource status and the data files currently being transmitted in the network through an artificial intelligence learning model; The artificial intelligence learning model is used to learn the network resource status, the data files currently being transmitted in the network, and the data files corresponding to each data processing request to obtain the number of data files allowed to be transmitted by the network and the priority of each data file. The number of data files allowed to be transmitted by the network represents the number of data files that the network resource status allows to be transmitted in addition to the data files currently being transmitted by the network. The number of licenses is determined based on the number of data files allowed to be transmitted on the network, the data files currently being transmitted on the network, and a preset transmission quantity threshold. Each license is matched with each data processing request one by one based on the priority of each data file; The one-to-one matching relationship between the license and the data processing request is determined as the characteristic of the file to be transmitted.

3. The method as described in claim 2, characterized in that, The determination of the number of licenses based on the number of data files allowed to be transmitted by the network, the data files currently being transmitted in the network, and a preset transmission quantity threshold includes: Calculate the sum of the number of data files allowed to be transmitted by the network and the number of data files currently being transmitted by the network; Compare whether the sum is less than the preset transmission quantity threshold; If so, the number of licenses is determined based on the difference between the transmission quantity threshold and the sum value; Otherwise, the processing of each of the aforementioned data processing requests will be suspended.

4. The method as described in claim 3, characterized in that, After suspending the processing of each of the data processing requests, the method further includes: Monitor the processing progress of each of the aforementioned coroutines on each of the aforementioned data files; If the processing progress indicates that at least one of the data files has been processed, then the sum value is re-compared to see if it is less than the preset transmission quantity threshold.

5. The method as described in claim 2, characterized in that, The step of matching each license with each data processing request one-to-one based on the priority of each data file includes: The data processing requests are sorted in descending order of priority for each of the data files; Each of the sorted data processing requests is matched with a corresponding license until the number of remaining licenses is zero or the number of remaining data processing requests is zero.

6. The method as described in claim 2, characterized in that, The process of creating multiple coroutines in a thread based on the characteristics of the file to be transmitted, and processing each of the data files through each coroutine, includes: The number of coroutines to be created is determined based on the number of matching relationships included in the features of the file to be transmitted; In the thread corresponding to the data file being transmitted in the current network, multiple coroutines are created using a coroutine container, with the number of coroutines being the same as the number of coroutines. Each coroutine is configured independently. The data files corresponding to the data processing requests in the matching relationship are processed by different coroutines.

7. The method according to any one of claims 1 to 6, characterized in that, The artificial intelligence learning model is trained in the following way: The historical network transmission status and historical data processing requests are provided as input parameters to the original artificial intelligence learning model, and the original artificial intelligence learning model is trained in an iterative manner. The historical network transmission status is the network transmission status within the most recent sampling period obtained according to a preset sampling period, and the historical data processing requests are those received within the most recent sampling period. During each iteration, the convergence condition is determined based on the loss value of that round. If the convergence condition is met, the original artificial intelligence learning model is considered to have converged. The iteration process is as follows: The original artificial intelligence learning model learns from the input parameters to obtain the characteristics of historically transmitted files; The loss value for this round is determined based on the characteristics of the historical transmitted files, and the current model parameters of the original artificial intelligence learning model are adjusted based on the loss value for this round.

8. A data processing apparatus, characterized in that, include: A response unit is configured to respond to at least one data processing request by learning the network transmission status and the data files corresponding to each data processing request through an artificial intelligence learning model to obtain the characteristics of the file to be transmitted. The data processing request includes a data upload request and / or a data download request, and the network transmission status includes the network resource status and the data files currently being transmitted in the network. The processing unit is configured to create multiple coroutines in a thread based on the characteristics of the file to be transmitted, and process each of the data files through each of the coroutines.

9. A server, characterized in that, include: Memory, used to store executable instructions; A processor for reading and executing executable instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor, the processor is able to perform the method as described in any one of claims 1 to 7.

Images