Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring a plurality of pieces of first to-be-processed data; performing legality verification on the plurality of first to-be-processed data to obtain a plurality of second to-be-processed data passing verification; determining the current capacity of idle resources of a data processing unit, and screening target data to be processed from the plurality of pieces of second data to be processed according to the current capacity; and packaging the target to-be-processed data to obtain packaged data, and sending the packaged data to the data processing unit for processing. By adopting the method, the data processing efficiency of the heterogeneous system can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and program product. Background Art

[0002] Generally, for fields with extremely large amounts of data, such as big data, artificial intelligence, and weather forecasts, the amount of computing generated is also very huge. In order to improve computing efficiency, a heterogeneous system consisting of a central processing unit and a data processing unit can be used for processing. Among them, the central processing unit is responsible for processing logical relationships, and the data processing unit is responsible for processing data.

[0003] However, when faced with data processing of a large number of lightweight kernel functions, the above-mentioned data processing method based on heterogeneous systems has the problem of low processing efficiency. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment, storage medium and program product that can improve the data processing efficiency of heterogeneous systems in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method. The method comprises:

[0006] Acquire a plurality of first data to be processed;

[0007] Performing a validity check on the plurality of first data to be processed to obtain a plurality of second data to be processed that pass the check;

[0008] Determining a current capacity of idle resources of the data processing unit, and screening target data to be processed from the plurality of second data to be processed according to the current capacity;

[0009] After the target data to be processed is packaged and processed, packaged data is obtained, and the packaged data is sent to the data processing unit for processing.

[0010] In one embodiment, the determining the current capacity of the idle resources of the data processing unit and filtering the target data to be processed from the plurality of second data to be processed according to the current capacity includes:

[0011] If the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by the plurality of second data to be processed, the plurality of second data to be processed are used as the target data to be processed;

[0012] If the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by the plurality of second data to be processed, second data to be processed that matches the current capacity of the idle resources is selected from the plurality of second data to be processed as the target data to be processed.

[0013] In one embodiment, when the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by the plurality of second data to be processed, and after the target data to be processed is packaged and sent to the data processing unit for processing, the method further includes:

[0014] Re-determine the new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, return to the step of filtering out the target data to be processed from the multiple second data to be processed based on the current capacity; the new second data to be processed is the second data to be processed in the multiple second data to be processed except the target data to be processed.

[0015] In one embodiment, before determining the current capacity of the idle resources of the data processing unit, the method further includes:

[0016] Based on the principle of continuous storage, the plurality of second data to be processed are sorted according to storage addresses to obtain sorted second data to be processed.

[0017] In one embodiment, the method further comprises:

[0018] Determining capacity information of the storage space according to the capacity of the space occupied by the plurality of second data to be processed;

[0019] The capacity information of the storage space is sent to the data processing unit; the capacity information is used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space.

[0020] In a second aspect, the present application further provides a data processing method. The method comprises: receiving packaged data sent by a central processing unit; the packaged data is obtained according to any method described in the first aspect above;

[0021] Parsing the packaged data to obtain a plurality of data to be processed;

[0022] The plurality of data to be processed are stored in a processing queue and processed in sequence.

[0023] In a third aspect, the present application further provides a data processing device. The device comprises:

[0024] A first acquisition module, configured to acquire a plurality of first data to be processed;

[0025] A second acquisition module; configured to perform a validity check on the plurality of first data to be processed, and obtain a plurality of second data to be processed that pass the check;

[0026] a first screening module, configured to determine a current capacity of idle resources of the data processing unit, and screen target data to be processed from the plurality of second data to be processed according to the current capacity;

[0027] The first sending module is used to package the target data to be processed to obtain packaged data, and send the packaged data to the data processing unit for processing.

[0028] In a fourth aspect, the present application further provides a data processing device. The device comprises:

[0029] A receiving module, configured to receive packaged data sent by a central processing unit; the packaged data is obtained according to the method described in any one of the first aspects above;

[0030] A third acquisition module is used to parse the packaged data to obtain a plurality of data to be processed;

[0031] The processing module is used to store the plurality of data to be processed in a processing queue for sequential processing.

[0032] In a fifth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the methods described in the first and second aspects are implemented.

[0033] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first and second aspects above.

[0034] In a seventh aspect, the present application further provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the methods described in the first and second aspects above.

[0035] The above-mentioned data processing method, device, computer equipment, storage medium and program product, by obtaining multiple first data to be processed and performing a validity check on the multiple first data to be processed, can obtain multiple second data to be processed that have passed the check, thereby determining the current capacity of the idle resources of the data processing unit, and can filter out target data to be processed from the multiple second data to be processed according to the current capacity, and then package the target data to be processed to obtain packaged data, and send the packaged data to the data processing unit for processing. Compared with traditional technologies, since the data processing method provided by the present application can filter out target data to be processed according to the current capacity of the idle resources of the data processing unit, the target data to be processed can match the current capacity of the idle resources of the data processing unit, thereby enabling the data processing unit to be in a full-load state, thereby improving the processing efficiency of data processing; in addition, since the target processing data is packaged and sent to the data processing unit in a unified manner, the resources of the central processing unit can be released in time, the occupancy rate of the central processing unit can be reduced, and the sending efficiency of the data to be processed can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A diagram of an application environment of a data processing method in one embodiment;

[0037] Figure 2 1 is a flow chart of a data processing method in one embodiment;

[0038] Figure 3 A schematic diagram of sending target data to be processed in one embodiment;

[0039] Figure 4 is a flow chart of a data processing method in another embodiment;

[0040] Figure 5 is a flow chart of a data processing method in another embodiment;

[0041] Figure 6 is a flow chart of a data processing method in another embodiment;

[0042] Figure 7 is a structural block diagram of a data processing device in one embodiment;

[0043] Figure 8 is a structural block diagram of a data processing device in another embodiment;

[0044] Figure 9 is a structural block diagram of a data processing device in another embodiment;

[0045] Figure 10 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] At present, data processing in heterogeneous systems can be completed through the collaboration of a central processing unit and a data processing unit. When the central processing unit receives an execution instruction, it can send the data corresponding to the execution instruction to the data processing unit, and then send the kernel function to be executed to the data processing unit for processing. However, for lightweight kernel functions, since the execution code in the data processing unit is relatively simple, but the time-consuming period of non-calculation is very short, the waiting time of the kernel function in the central processing unit is much longer than the processing time in the data processing unit, resulting in a low utilization rate of the data processing unit; in addition, when the order of magnitude of the lightweight kernel function is relatively high, it will cause the resources of the central processing unit to be occupied for a long time, resulting in an excessively high occupancy rate of the central processing unit. Therefore, due to the excessively high occupancy rate of the central processing unit and the low utilization rate of the data processing unit, the heterogeneous system has the problem of low processing efficiency in processing data. Based on this, the present application proposes a data processing method that can improve the data processing efficiency of heterogeneous systems.

[0048] The data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the heterogeneous system includes a central processing unit 102 and a data processing unit 104. The central processing unit 102 is responsible for processing the logical relationship between the data to be processed, and the data processing unit 104 is responsible for processing the data to be processed. The central processing unit 102 transmits data to the data processing unit 104 via a system bus.

[0049] In one embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 Taking the central processing unit in the example as an example, the following steps are included:

[0050] S201: Acquire a plurality of first data to be processed.

[0051] It should be noted that the central processing unit can receive an execution instruction input from an external device through an input interface and obtain data to be processed based on the execution instruction. In this embodiment, the data to be processed obtained based on the execution instruction can be determined as the first data to be processed. Optionally, the first data to be processed corresponding to the execution instruction can be obtained from an external device, or the first data to be processed corresponding to the execution instruction can be obtained from a memory. Exemplarily, the data to be processed can be a computing function applied to fields such as big data, artificial intelligence, and weather forecasting, and the computing function can be a lightweight kernel function executed on a heterogeneous system.

[0052] S202: Perform a validity check on the plurality of first data to be processed to obtain a plurality of second data to be processed that pass the check.

[0053] Among them, the legality check refers to checking whether the first data to be processed complies with the processing rules. For example, if the first data to be processed is a kernel function, the legality of the data type and data structure of the parameters of the kernel function can be checked.

[0054] In this embodiment, the first data to be processed can be verified for legitimacy according to preset processing rules. Optionally, the legitimacy of each first data to be processed can be verified separately, and the first data to be processed that passes the verification can be determined as the second data to be processed. Alternatively, the legitimacy of all first data to be processed can be verified uniformly, and the first data to be processed that passes the verification can be determined as the second data to be processed.

[0055] S203: Determine the current capacity of the idle resources of the data processing unit, and select target data to be processed from the plurality of second data to be processed according to the current capacity.

[0056] Among them, the data processing unit refers to an integrated circuit that processes a data set according to the data and instructions sent by the central processing unit under the control of the central processing unit. The data processing unit includes multiple registers, memories, computing units and other resources. The unoccupied resources in the data processing unit are called idle resources. In this embodiment, the capacity of the idle resources in the data processing unit at the current moment can be determined as the current capacity of the idle resources of the data processing unit.

[0057] It can be understood that if the processing queue of the data processing unit is in a full-load state, the idle time of the data processing unit can be reduced and the data processing efficiency can be improved. Therefore, in this embodiment, if the order of magnitude of the second units to be processed is large, the central processing unit can filter out the target data to be processed from multiple second data to be processed based on the current capacity of the idle resources of the data processing unit, so that the capacity of the filtered target data to be processed matches the current capacity of the idle resources of the data processing unit, thereby ensuring that the data processing unit is in a full-load state.

[0058] In this embodiment, the central processing unit can obtain the current capacity of the idle resources in the data processing unit through the driver interface. As an optional implementation, the capacity of the data processing unit can be divided into several processing units, and the processing unit of unprocessed data can be determined as the current capacity of the idle resources of the data processing unit. Based on the current number of idle resources, the second to-be-processed data that matches the current number of idle resources can be screened out from the second to-be-processed data as the target to-be-processed data. Optionally, the target to-be-processed data can include one second to-be-processed data, or the target to-be-processed data can also include multiple second to-be-processed data.

[0059] S204: After the target data to be processed is packaged and processed, packaged data is obtained, and the packaged data is sent to the data processing unit for processing.

[0060] Packaging refers to a data processing method of encapsulating one or more data into a data packet, and the data after the packaging process is called packaged data.

[0061] In this embodiment, the central processing unit can package the target data to be processed according to the data transmission protocol between the central processing unit and the data processing unit to obtain packaged data, and then send the packaged data to the data processing unit through the system bus, and the data processing unit can process the received packaged data.

[0062] It is understandable that if the target processing unit includes multiple second data to be processed, the time consumed by the central processing unit to send the multiple target data to be processed to the data processing unit is less than the time consumed by the central processing unit to send each target data to be processed to the data processing unit separately in sequence. For example, Figure 3 The figures shown are schematic diagrams of sending multiple target data to be processed separately and sending multiple target data to be processed uniformly.

[0063] In the above-mentioned data processing method, by obtaining multiple first data to be processed and performing a validity check on the multiple first data to be processed, multiple second data to be processed that pass the check can be obtained, thereby determining the current capacity of the idle resources of the data processing unit, and being able to filter out the target data to be processed from the multiple second data to be processed according to the current capacity, and then packaging the target data to be processed to obtain packaged data, and then sending the packaged data to the data processing unit for processing. Compared with traditional technologies, the data processing method provided by the present application can filter out the target data to be processed according to the current capacity of the idle resources of the data processing unit, so that the target data to be processed can match the current capacity of the idle resources of the data processing unit, thereby enabling the data processing unit to be in a full-load state, thereby improving the processing efficiency of data processing; in addition, since the target processing data is packaged and sent to the data processing unit in a unified manner, the resources of the central processing unit can be released in time, the occupancy rate of the central processing unit can be reduced, and the sending efficiency of the data to be processed can be improved.

[0064] In the above scenario of determining the current capacity of the idle resources of the data processing unit and filtering out the target data to be processed from the plurality of second data to be processed according to the current capacity, the target data to be processed can be filtered out according to the result of comparing the current capacity of the idle resources of the data processing unit with the capacity of the space occupied by the second data to be processed. Figure 4 As shown, the above S203 includes:

[0065] S301: If the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by the plurality of second data to be processed, the plurality of second data to be processed are used as target data to be processed.

[0066] Here, "not less than" may include "equal to" or "greater than," i.e., the current capacity of the idle resources of the data processing unit may be equal to the capacity of the space occupied by the plurality of second data to be processed, or the current capacity of the idle resources of the data processing unit may be greater than the capacity of the space occupied by the plurality of second data to be processed. It is understood that if the current capacity of the idle resources of the data processing unit can meet the capacity of the space occupied by the plurality of second data to be processed, the central processing unit may simultaneously send the plurality of second data to be processed to the data processing unit, so that the data processing unit can simultaneously receive the plurality of second data to be processed.

[0067] In this embodiment, the current capacity of the idle resources of the data processing unit can be compared with the capacity of the space occupied by multiple second data to be processed, and based on the comparison result, it can be determined whether the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by multiple second data to be processed. If the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by multiple second data to be processed, then the multiple second data to be processed can be used as target data to be processed.

[0068] As an optional implementation, the difference between the current capacity of the idle resources of the data processing unit and the capacity of the space occupied by multiple second data to be processed can be used as a comparison result. If the comparison result is greater than 0 or equal to 0, it can be considered that the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by multiple second data to be processed, and the multiple second data to be processed are used as target data to be processed.

[0069] As an optional implementation, the capacity of the resources of the data processing unit can be divided into several processing units, and the number of unoccupied processing units can be determined as the current capacity of the idle resources of the data processing unit, so that the difference between the number of unoccupied processing units and the number of multiple second data to be processed can be used as a comparison result. If the comparison result is greater than 0 or equal to 0, it can be determined that the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by multiple second data to be processed, and the second data to be processed can be determined as the target data to be processed.

[0070] S302: If the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by the plurality of second data to be processed, select the second data to be processed that matches the current capacity of the idle resources from the plurality of second data to be processed as the target data to be processed.

[0071] In this embodiment, the current capacity of the idle resources of the data processing unit can be compared with the capacity of the space occupied by multiple second data to be processed, and it can be determined based on the comparison result whether the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by multiple second data to be processed. If the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by multiple second data to be processed, the second data to be processed that matches the current capacity of the idle resources can be screened out from the multiple second data to be processed as the target data to be processed.

[0072] As an optional implementation, the capacity of the resources of the data processing unit can be divided into several processing units, and the number of unoccupied processing units can be determined as the current capacity of the idle resources of the data processing unit, so that the difference between the number of unoccupied processing units and the number of multiple second data to be processed can be used as a comparison result. If the comparison result is less than 0, it can be determined that the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by the multiple second data to be processed, and the target data to be processed is filtered out from the multiple second-generation processing data according to the number of unoccupied processing units.

[0073] For example, if the number of unoccupied processing units in the data processing unit is 2048 and the number of second units to be processed is 3000, then the 2048 second data to be processed may be determined as the target data to be processed.

[0074] In this embodiment, when the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by multiple second data to be processed, multiple second data to be processed can be used as target data to be processed. When the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by multiple second data to be processed, second data to be processed that matches the current capacity of the idle resources are screened out from the multiple second data to be processed as target data to be processed. Since the target data to be processed can be accurately determined according to the current capacity of the idle resources, the data processing unit can be in the maximum processing state, reducing the idle rate of the data processing unit, and thus improving the data processing efficiency.

[0075] In a scenario where the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by the plurality of second data to be processed, and second data to be processed that matches the current capacity of the idle resources is selected from the plurality of second data to be processed as the target data to be processed, there may still be second data to be processed in the second data to be processed that has not been sent to the data processing unit. In one embodiment, when the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by the plurality of second data to be processed, and after the target data to be processed is packaged and sent to the data processing unit for processing, the method further includes:

[0076] Re-determine the new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, return to the step of filtering out the target data to be processed from the multiple second data to be processed based on the current capacity; the new second data to be processed is the second data to be processed other than the target data to be processed in the multiple second data to be processed.

[0077] It should be noted that, when the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by multiple second data to be processed, and the central processing unit sends the target data to be processed to the data processing unit, there is still unsent second data to be processed in the central processing unit. The target data to be processed that matches the new current capacity can be re-determined from the second data to be processed based on the new current capacity of the idle resources of the data processing unit. The new current capacity refers to the capacity of the idle resources in the data processing unit after the central processing unit sends the target data to be processed to the data processing unit, and the new second data to be processed refers to the second data to be processed other than the target data to be processed in the second data to be processed. Optionally, the new second data to be processed may be one, or the new second data to be processed may be multiple.

[0078] In this embodiment, the central processing unit can re-obtain the new current capacity of the idle resources of the data processing unit through the driver interface. Optionally, if there are no idle resources in the data processing unit, the central processing unit can periodically obtain the new current capacity of the idle resources of the data processing unit until there are idle resources in the data processing and obtain the new current capacity of the idle resources of the data processing unit.

[0079] When the new current capacity of the idle resources of the data processing unit is obtained, the difference between the new current capacity and the capacity of the space occupied by the new second data to be processed can be used as a comparison result between the two, and based on the comparison result, it is determined whether the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by multiple new second data to be processed. If so, the target data to be processed that matches the new current capacity is re-screened from the new second data to be processed; if not, the new second data to be processed is determined as the target data to be processed.

[0080] In this embodiment, by re-determining the new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, returning to execute the step of filtering out the target data to be processed from multiple second data to be processed based on the current capacity, it is possible to re-filter out the target data to be processed when there is unsent second data to be processed in the second data to be processed, thereby avoiding the need to process the data to be processed multiple times, so that the second data to be processed can be quickly sent to the data processing unit, the resources of the central processing unit can be released in time, and the occupancy rate of the central processing unit can be reduced.

[0081] It should be noted that when processing data to be processed in the data processing unit, it is necessary to first obtain the data to be processed from the storage space and store the processing results in the storage space. In order to improve the efficiency of data processing, each piece of data to be processed can be sorted and processed in order, thereby reducing the query time of the storage space when obtaining the data to be processed and storing the processing results. In one embodiment, before determining the current capacity of the idle resources of the data processing unit, the above method further includes:

[0082] Based on the principle of continuous storage, the plurality of second data to be processed are sorted according to the storage addresses to obtain sorted second data to be processed.

[0083] Continuous storage refers to a storage method in which each piece of data to be processed is stored continuously in a storage space. In this embodiment, the second-generation processed data can be stored in a plurality of storage spaces with consecutive storage addresses in the order in which the second data to be processed are acquired, thereby determining the second data to be processed corresponding to each storage address as the sorted second data to be processed.

[0084] As an optional implementation, each second data to be processed can be reallocated to a storage space with a continuous storage address in the order of the capacity of the space occupied by each second data to be processed, so that the second data to be processed corresponding to each storage address can be determined as the sorted second data to be processed.

[0085] In this embodiment, by taking continuous storage as the principle, multiple second data to be processed are sorted according to the storage address of each second data to be processed to obtain the sorted second data to be processed, so that the central processing unit can send the second data to be processed with continuous storage addresses to the data processing unit, thereby enabling the data processing unit to reduce the time of obtaining each second data to be processed, thereby improving data processing efficiency.

[0086] Before sending the target data to be processed to the data processing unit, the central processing unit can instruct the data processing unit to create a data processing space that matches the capacity information based on the capacity application of the space occupied by the data to be processed. Figure 5 As shown, the above method also includes:

[0087] S401: Determine capacity information of a storage space according to the capacity of a space occupied by a plurality of second data to be processed.

[0088] The capacity information of the storage space refers to information related to the amount of information that can be stored in the memory. Optionally, the capacity information may include the size of the storage space or the storage address of the storage space. The capacity of the space occupied by the second data to be processed may include the size of the storage space occupied by each piece of second data to be processed and the size of the storage space occupied by the processing result corresponding to each piece of data to be processed.

[0089] In this embodiment, the central processing unit can determine the capacity of the space occupied by all second data to be processed based on the capacity of the space occupied by each second data to be processed, thereby determining the size of the storage space required for multiple second data to be processed, and the corresponding storage address in the storage space. It can be understood that the storage address can be a continuous address, or the storage address can also be a discontinuous address.

[0090] S402, sending the capacity information of the storage space to the data processing unit; the capacity information is used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space.

[0091] It should be noted that, in a heterogeneous system, the data processing unit is responsible for processing data. Therefore, the central processing unit needs to instruct the data processing unit to allocate a data processing space for storing a plurality of second data to be processed.

[0092] In this embodiment, the central processing unit can send the determined capacity information of the storage space to the data processing unit, so that the data processing unit allocates data processing space matching the capacity of the storage space according to the capacity information for storing multiple second data to be processed.

[0093] In this embodiment, the capacity information of the storage space is determined based on the capacity of the space occupied by multiple second data to be processed, so that the capacity information used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space can be sent to the data processing unit, thereby avoiding multiple creations of the data processing space and improving the processing efficiency of the system; in addition, since the data processing unit only needs to create the data processing space once, compared with traditional technologies, it can avoid the problem of memory fragmentation caused by multiple creations of the data processing space.

[0094] In one embodiment, Figure 6 As shown, a data processing method is also provided, in which the method is applied Figure 1 The data processing unit in FIG. 1 is taken as an example to illustrate the method, which includes the following steps:

[0095] S501, receiving packaged data sent by a central processing unit; the packaged data is obtained according to the data processing method in any of the above embodiments.

[0096] The packaged data refers to a data packet obtained by the central processing unit encapsulating the target data to be processed according to the data transmission protocol. In this embodiment, the data processing unit can receive the packaged data sent by the central processing unit through the system bus.

[0097] S502: Parse the packaged data to obtain a plurality of data to be processed.

[0098] It is understandable that the packaged data received by the data processing unit is a data packet, so the packaged data needs to be parsed to obtain a plurality of data to be processed.

[0099] In this embodiment, the data processing unit may parse and process the packaged data according to the data transmission protocol to obtain a plurality of data to be processed in the packaged data.

[0100] S503: Store the plurality of data to be processed in a processing queue for sequential processing.

[0101] Wherein, the processing queue refers to a linear list used to process the data to be processed in the data processing unit. Optionally, the processing queue can be a queue or a stack. In this embodiment, the processing queue in the data processing unit can be a circular queue with a size of 4096, and the data processing unit can allocate multiple data to be processed to the free space of the processing queue respectively.

[0102] In this embodiment, the data processing unit may sequentially store the plurality of data to be processed obtained by parsing into a processing queue to process each data to be processed. Optionally, if the processing queue is a queue, the data processing unit may process each data to be processed according to a first-in-first-out principle, or if the processing queue is a stack, the data processing unit may also process each data to be processed according to a first-in-last-out principle.

[0103] In this embodiment, the data processing unit can parse the packaged data sent by the central processing unit to obtain multiple data to be processed, so that the multiple data to be processed can be stored in the processing queue for processing in sequence, avoiding the problem of random processing of the data to be processed causing confusion in the processing results, so that the data processing unit can process each data to be processed in sequence, thereby improving the accuracy of the obtained processing results.

[0104] To facilitate understanding by those skilled in the art, the data processing method provided by this application is described in detail below. The method may include:

[0105] S1, the central processing unit obtains a plurality of first data to be processed.

[0106] S2, performing a validity check on the plurality of first data to be processed to obtain a plurality of second data to be processed that pass the check.

[0107] S3. Determine the capacity information of the storage space according to the capacity of the space occupied by the plurality of second data to be processed.

[0108] S4, sending the capacity information of the storage space to the data processing unit; the capacity information is used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space.

[0109] S5 , based on the principle of continuous storage, sorting the plurality of second data to be processed according to the storage addresses to obtain sorted second data to be processed.

[0110] S6, determining the current capacity of the idle resources of the data processing unit, and filtering out target data to be processed from the sorted second data to be processed according to the current capacity.

[0111] S7: If the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by the sorted second data to be processed, the sorted second data to be processed is used as the target data to be processed.

[0112] S8. If the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by the sorted second data to be processed, select the second data to be processed that matches the current capacity of the idle resources from the sorted second data to be processed as the target data to be processed.

[0113] S9, after the target data to be processed is packaged and processed, packaged data is obtained, and the packaged data is sent to the data processing unit for processing.

[0114] S10, re-determine the new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, return to execute the step of filtering out the target data to be processed from the multiple second data to be processed according to the current capacity; the new second data to be processed is the second data to be processed other than the target data to be processed in the multiple second data to be processed.

[0115] It should be noted that for the descriptions in S1-S10 above, reference can be made to the relevant descriptions in the above embodiments, and the effects are similar, so this embodiment will not be repeated here.

[0116] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0117] Based on the same inventive concept, the present application also provides a data processing device for implementing the aforementioned data processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more data processing device embodiments provided below can be found in the above-mentioned limitations on the data processing method and will not be repeated here.

[0118] In one embodiment, Figure 7 As shown, a data processing device is provided, comprising: a first acquisition module 11, a second acquisition module 12, a first screening module 13 and a first sending module 14, wherein:

[0119] The first acquisition module 11 is configured to acquire a plurality of first data to be processed.

[0120] The second acquisition module 12 is configured to perform a validity check on the plurality of first data to be processed, and obtain a plurality of second data to be processed that pass the check.

[0121] The first screening module 13 is configured to determine the current capacity of the idle resources of the data processing unit, and screen target data to be processed from the plurality of second data to be processed according to the current capacity.

[0122] The first sending module 14 is configured to package the target data to be processed to obtain packaged data, and send the packaged data to the data processing unit for processing.

[0123] The data processing device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0124] In one embodiment, Figure 8 As shown, the first screening module 13 includes: a first determining unit 131 and a second determining unit 132, wherein:

[0125] The first determining unit 131 is configured to use the plurality of second to-be-processed data as target to-be-processed data if the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by the plurality of second to-be-processed data.

[0126] The second determining unit 132 is configured to select second to-be-processed data that matches the current capacity of the idle resources from the multiple second to-be-processed data as target to-be-processed data if the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by the multiple second to-be-processed data.

[0127] The data processing device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0128] In one embodiment, when the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by the plurality of second data to be processed, and after the target data to be processed is packaged and sent to the data processing unit for processing, please continue to refer to Figure 8 The above device further includes: a second screening module 15, wherein:

[0129] The second screening module 15 is used to redetermine the new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, return to execute the step of filtering out the target data to be processed from the multiple second data to be processed according to the current capacity; the new second data to be processed is the second data to be processed other than the target data to be processed in the multiple second data to be processed.

[0130] The data processing device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0131] In one embodiment, before determining the current capacity of the idle resources of the data processing unit, please continue to refer to Figure 8 The above device further includes: a sorting module 16, wherein:

[0132] The sorting module 16 is configured to sort the plurality of second data to be processed according to storage addresses based on a principle of continuous storage to obtain sorted second data to be processed.

[0133] The data processing device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0134] In one embodiment, please refer to Figure 8 The above device further includes: a determination module 17 and a second sending module 18, wherein:

[0135] The determination module 17 is configured to determine the capacity information of the storage space according to the capacity of the space occupied by the plurality of second data to be processed.

[0136] The second sending module 18 is used to send the capacity information of the storage space to the data processing unit; the capacity information is used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space.

[0137] The data processing device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0138] In one embodiment, another data processing apparatus is provided, such as Figure 9 As shown, the above device includes: a receiving module 19, an analyzing module 20 and a processing module 21, wherein:

[0139] The receiving module 19 is used to receive the packaged data sent by the central processing unit; the packaged data is obtained according to any of the above-mentioned data processing devices.

[0140] The parsing module 20 is used to parse the packaged data to obtain a plurality of data to be processed.

[0141] The processing module 21 is used to store a plurality of data to be processed into a processing queue for sequential processing.

[0142] The data processing device provided in this embodiment can execute the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0143] Each module in the above-mentioned data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0144] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store kernel function data to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0145] Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0146] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0147] Acquire a plurality of first data to be processed;

[0148] Performing a validity check on the plurality of first data to be processed to obtain a plurality of second data to be processed that pass the check;

[0149] Determining the current capacity of the idle resources of the data processing unit, and selecting target data to be processed from the plurality of second data to be processed according to the current capacity;

[0150] After the target data to be processed is packaged and processed, packaged data is obtained, and the packaged data is sent to the data processing unit for processing.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] If the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by the plurality of second data to be processed, the plurality of second data to be processed are used as target data to be processed;

[0153] If the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by the plurality of second data to be processed, the second data to be processed that matches the current capacity of the idle resources is selected from the plurality of second data to be processed as the target data to be processed.

[0154] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0155] Re-determine the new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, return to the step of filtering out the target data to be processed from the multiple second data to be processed based on the current capacity; the new second data to be processed is the second data to be processed other than the target data to be processed in the multiple second data to be processed.

[0156] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0157] Based on the principle of continuous storage, the plurality of second data to be processed are sorted according to the storage addresses to obtain sorted second data to be processed.

[0158] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0159] Determining capacity information of the storage space according to the capacity of the space occupied by the plurality of second data to be processed;

[0160] The capacity information of the storage space is sent to the data processing unit; the capacity information is used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] Receiving packaged data sent by the central processing unit; the packaged data is obtained according to any of the steps above;

[0163] Parse the packaged data to obtain multiple data to be processed;

[0164] Store multiple data to be processed in the processing queue and process them in sequence.

[0165] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0166] Acquire a plurality of first data to be processed;

[0167] Performing a validity check on the plurality of first data to be processed to obtain a plurality of second data to be processed that pass the check;

[0168] Determining the current capacity of the idle resources of the data processing unit, and selecting target data to be processed from the plurality of second data to be processed according to the current capacity;

[0169] After the target data to be processed is packaged and processed, packaged data is obtained, and the packaged data is sent to the data processing unit for processing.

[0170] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0171] If the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by the plurality of second data to be processed, the plurality of second data to be processed are used as target data to be processed;

[0172] If the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by the plurality of second data to be processed, the second data to be processed that matches the current capacity of the idle resources is selected from the plurality of second data to be processed as the target data to be processed.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0174] Re-determine the new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, return to the step of filtering out the target data to be processed from the multiple second data to be processed based on the current capacity; the new second data to be processed is the second data to be processed other than the target data to be processed in the multiple second data to be processed.

[0175] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0176] Based on the principle of continuous storage, the plurality of second data to be processed are sorted according to the storage addresses to obtain sorted second data to be processed.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0178] Determining capacity information of the storage space according to the capacity of the space occupied by the plurality of second data to be processed;

[0179] The capacity information of the storage space is sent to the data processing unit; the capacity information is used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0181] Receiving packaged data sent by the central processing unit; the packaged data is obtained according to any of the steps above;

[0182] Parse the packaged data to obtain multiple data to be processed;

[0183] Store multiple data to be processed in the processing queue and process them in sequence.

[0184] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0185] Acquire a plurality of first data to be processed;

[0186] Performing a validity check on the plurality of first data to be processed to obtain a plurality of second data to be processed that pass the check;

[0187] Determining the current capacity of the idle resources of the data processing unit, and selecting target data to be processed from the plurality of second data to be processed according to the current capacity;

[0188] After the target data to be processed is packaged and processed, packaged data is obtained, and the packaged data is sent to the data processing unit for processing.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0190] If the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by the plurality of second data to be processed, the plurality of second data to be processed are used as target data to be processed;

[0191] If the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by the plurality of second data to be processed, the second data to be processed that matches the current capacity of the idle resources is selected from the plurality of second data to be processed as the target data to be processed.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0193] Re-determine the new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, return to the step of filtering out the target data to be processed from the multiple second data to be processed based on the current capacity; the new second data to be processed is the second data to be processed other than the target data to be processed in the multiple second data to be processed.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0195] Based on the principle of continuous storage, the plurality of second data to be processed are sorted according to the storage addresses to obtain sorted second data to be processed.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0197] Determining capacity information of the storage space according to the capacity of the space occupied by the plurality of second data to be processed;

[0198] The capacity information of the storage space is sent to the data processing unit; the capacity information is used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0200] Receiving packaged data sent by the central processing unit; the packaged data is obtained according to any of the steps above;

[0201] Parse the packaged data to obtain multiple data to be processed;

[0202] Store multiple data to be processed in the processing queue and process them in sequence.

[0203] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0204] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0205] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire a plurality of first data to be processed; Performing a validity check on the plurality of first data to be processed to obtain a plurality of second data to be processed that pass the check; Determine the current capacity of the idle resources of the data processing unit, and select the target data to be processed from the plurality of second data to be processed according to the current capacity; After the target data to be processed is packaged, packaged data is obtained, and the packaged data is sent to the data processing unit for processing.

2. The method according to claim 1, characterized in that The determining the current capacity of the idle resources of the data processing unit, and selecting the target data to be processed from the plurality of second data to be processed according to the current capacity, comprises: If the current capacity of the idle resources of the data processing unit is not less than the capacity of the space occupied by the plurality of second data to be processed, the plurality of second data to be processed are used as the target data to be processed; If the current capacity of the idle resources of the data processing unit is smaller than the capacity of the space occupied by the multiple second data to be processed, the second data to be processed that matches the current capacity of the idle resources is selected from the multiple second data to be processed as the target data to be processed.

3. The method according to claim 2, characterized in that In a case where the current capacity of the idle resources of the data processing unit is less than the capacity of the space occupied by the plurality of second data to be processed, and after the target data to be processed is packaged and sent to the data processing unit for processing, the method further includes: Re-determine a new current capacity of the idle resources of the data processing unit, and based on the new current capacity and the new second data to be processed, return to execute the step of filtering out the target data to be processed from the multiple second data to be processed based on the current capacity; the new second data to be processed is the second data to be processed in the multiple second data to be processed except the target data to be processed.

4. The method according to any one of claims 1 to 3, characterized in that: Before determining the current capacity of the idle resources of the data processing unit, the method further comprises: Based on the principle of continuous storage, the plurality of second data to be processed are sorted according to storage addresses to obtain sorted second data to be processed.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Determining capacity information of the storage space according to the capacity of the space occupied by the plurality of second data to be processed; The capacity information of the storage space is sent to the data processing unit; the capacity information is used to instruct the data processing unit to create a data processing space that matches the capacity of the storage space.

6. A data processing method, characterized in that: The method comprises: Receiving packaged data sent by a central processing unit; the packaged data is obtained according to the method according to any one of claims 1 to 5; Parsing the packaged data to obtain a plurality of data to be processed; The plurality of data to be processed are stored in a processing queue and processed in sequence.

7. A data processing device, characterized in that: The device comprises: A first acquisition module, used for acquiring a plurality of first data to be processed; A second acquisition module; used for performing a legality check on the plurality of first data to be processed, and obtaining a plurality of second data to be processed that pass the check; A first screening module, used to determine the current capacity of the idle resources of the data processing unit, and screen the target data to be processed from the plurality of second data to be processed according to the current capacity; The first sending module is used to package the target data to be processed to obtain packaged data, and send the packaged data to the data processing unit for processing.

8. A data processing device, characterized in that: The device comprises: A receiving module, configured to receive packaged data sent by a central processing unit; the packaged data is obtained according to the method described in any one of claims 1 to 5; A third acquisition module is used to parse the packaged data to obtain a plurality of data to be processed; The processing module is used to store the plurality of data to be processed in a processing queue for sequential processing.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images