A data processing method and related device

Abstract

The embodiment of the present application relates to the field of computer technology, and provides a data processing method and related devices thereof, wherein the data processing method comprises: obtaining the amount of pre-transmission data and the pre-transmission path based on input data; obtaining the total transmission duration based on the amount of pre-transmission data and the pre-transmission path; sending the amount of pre-transmission data to a second chip, so that the second chip can obtain the calculation duration based on the amount of pre-transmission data and the calculation speed; sending the total transmission duration to the second chip; calculating the output data based on the input data; sending the output data to the second chip, so that the second chip can obtain the current transmission duration based on the output data. The present application reduces the waiting time during data transmission by planning the transmission path of data in advance, so that computing resources and data transmission resources can work together more efficiently, thereby significantly improving the processing speed and efficiency of subsequent data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and more specifically to a data processing method and related devices. Background Art

[0002] One of the core challenges in promoting high-performance computing and building large-scale machine learning models (e.g., bloom models) is to handle large and complex data sets. This requires frequent data transfer between high-bandwidth memory and various storage devices (e.g., shared memory and virtual memory) to support efficient training and accurate reasoning of the model. In the face of such a large data flow, direct memory access (DMA) technology is indispensable.

[0003] Existing DMA technologies, such as Figure 2 As shown in the figure, in each round of calculation, data transmission can only be started after the previous round of data transmission and all pre-computation tasks are completely completed. This sequential execution mode greatly restricts the overall throughput and response speed of the system, making it difficult to meet the needs of efficient calculation of large-scale machine learning models. Since data transmission in existing DMA technology is limited by the rigid serial processing flow, the time consumption of data transmission often becomes the main factor limiting the response speed of large-scale machine learning models. Summary of the invention

[0004] The present application provides a technical solution for a data processing method and related devices, which can reduce the waiting time during data transmission, enable computing resources and data transmission resources to work together more efficiently, thereby significantly improving the processing speed and efficiency of subsequent data.

[0005] In a first aspect, the present application provides a data processing method, which is applied to a first chip; the method comprises:

[0006] Based on the input data, obtaining the pre-transmission data amount and the pre-transmission path; the input data is obtained in advance;

[0007] Based on the pre-transmission data amount and the pre-transmission path, obtaining a total transmission duration;

[0008] sending the pre-transmitted data amount to the second chip, so that the second chip can obtain the calculation duration based on the pre-transmitted data amount and the calculation speed; the calculation speed is pre-acquired by the second chip;

[0009] Sending the total transmission duration to the second chip;

[0010] Based on the input data, output data is calculated;

[0011] The output data is sent to the second chip, so that the second chip can obtain the current transmission duration based on the output data.

[0012] As a specific solution in the technical solution of the present application, the step of sending the output data to the second chip includes:

[0013] Acquire a transmission signal; the transmission signal is obtained by the second chip based on the remaining storage space of the second chip;

[0014] The output data is sent to the second chip based on the transmission signal.

[0015] As a specific solution in the technical solution of the present application, the transmission signal at least includes the remaining storage space of the second chip; and sending the output data to the second chip based on the transmission signal includes:

[0016] Based on the output data, a plurality of time series data segments are acquired; the storage space occupied by each time series data segment is smaller than the remaining storage space;

[0017] Each time sequence data segment is sent to the second chip in time sequence.

[0018] In a second aspect, the present application provides a data processing method, which is applied to a second chip; the method comprises:

[0019] receiving a pre-transmission data amount sent by a first chip; the pre-transmission data amount is acquired by the first chip based on input data; the input data is pre-acquired by the first chip;

[0020] Based on the pre-transmitted data volume and the calculation speed, obtaining the calculation duration; the calculation speed is obtained in advance by the second chip;

[0021] receiving a total transmission duration sent by the first chip; the total transmission duration is acquired by the first chip based on the amount of pre-transmission data and a pre-transmission path; the pre-transmission path is acquired by the first chip based on input data;

[0022] receiving output data sent by the first chip; the output data is calculated by the first chip based on the input data;

[0023] Based on the output data, obtain the current transmission duration;

[0024] Based on the total transmission duration and the current transmission duration, obtaining a remaining transmission duration;

[0025] If the remaining transmission duration is less than the calculation duration, the calculation task is started.

[0026] As a specific solution in the technical solution of the present application, before receiving the output data sent by the first chip, the method includes:

[0027] Get the remaining storage space;

[0028] If the remaining storage space is greater than a preset threshold, a transmission signal is sent to the first chip, so that the first chip can send output data to the second chip based on the transmission signal.

[0029] In a third aspect, the present application provides a chip, including:

[0030] A first processing module is used to obtain a pre-transmission data amount and a pre-transmission path based on input data; the input data is pre-acquired;

[0031] And, based on the pre-transmission data amount and the pre-transmission path, obtaining a total transmission duration;

[0032] A first transceiver module is used to send the pre-transmission data amount to the second chip, so that the second chip can obtain the calculation time based on the pre-transmission data amount and the calculation speed; the calculation speed is pre-acquired by the second chip;

[0033] And, sending the total transmission duration to the second chip;

[0034] The first processing module is further used to calculate output data based on the input data;

[0035] The first transceiver module is further used to send the output data to the second chip, so that the second chip can obtain the current transmission duration based on the output data.

[0036] As a specific solution in the technical solution of the present application, the first transceiver module is further used to obtain a transmission signal; the transmission signal is obtained by the second chip based on the remaining storage space of the second chip;

[0037] And, sending the output data to the second chip based on the transmission signal.

[0038] As a specific solution in the technical solution of the present application, the transmission signal at least includes the remaining storage space of the second chip; the first processing module is further used to obtain multiple time series data segments based on the output data; the storage space occupied by each time series data segment is smaller than the remaining storage space;

[0039] The first transceiver module is also used to send each timing data segment to the second chip in a timing sequence.

[0040] In a fourth aspect, the present application provides a chip, including:

[0041] A second transceiver module is used to receive a pre-transmission data volume sent by the first chip; the pre-transmission data volume is acquired by the first chip based on input data; the input data is pre-acquired by the first chip;

[0042] A second processing module, configured to obtain a calculation duration based on the pre-transmitted data volume and a calculation speed; the calculation speed is pre-acquired by the chip;

[0043] The second transceiver module is further used to receive the total transmission time sent by the first chip; the total transmission time is acquired by the first chip based on the pre-transmission data volume and the pre-transmission path; the pre-transmission path is acquired by the first chip based on the input data;

[0044] and, receiving output data sent by the first chip; the output data is calculated by the first chip based on the input data;

[0045] The second processing module is further used to obtain the current transmission duration based on the output data;

[0046] And, based on the total transmission duration and the current transmission duration, obtaining a remaining transmission duration;

[0047] And, if the remaining transmission duration is less than the calculation duration, the calculation task is started.

[0048] As a specific solution in the technical solution of the present application, the second processing module is further used to obtain the remaining storage space;

[0049] The second transceiver module is further configured to send a transmission signal to the first chip if the remaining storage space is greater than a preset threshold, so that the first chip can send output data to the chip based on the transmission signal.

[0050] In a fifth aspect, the present application provides a data processing system, including a first chip and a second chip;

[0051] The first chip is used to obtain the pre-transmission data amount and the pre-transmission path based on the input data; the input data is pre-acquired;

[0052] And, based on the pre-transmission data amount and the pre-transmission path, obtaining a total transmission duration;

[0053] and, sending the pre-transmission data amount to a second chip;

[0054] The second chip is used to obtain the calculation duration based on the pre-transmitted data amount and the calculation speed; the calculation speed is pre-acquired by the second chip;

[0055] The first chip is further configured to send the total transmission duration to the second chip;

[0056] And, based on the input data, calculating output data;

[0057] and, sending the output data to the second chip;

[0058] The second chip is further used to obtain the current transmission duration based on the output data;

[0059] And, based on the total transmission duration and the current transmission duration, obtaining a remaining transmission duration;

[0060] And, if the remaining transmission duration is less than the calculation duration, the calculation task is started.

[0061] As a specific solution in the technical solution of the present application, the second chip is also used to obtain the remaining storage space;

[0062] and, if the remaining storage space is greater than a preset threshold, sending a transmission signal to the first chip;

[0063] The first chip is further configured to send the output data to the second chip based on the transmission signal.

[0064] As a specific solution in the technical solution of the present application, the first chip is further used to obtain multiple time series data segments based on the output data; the storage space occupied by each time series data segment is smaller than the remaining storage space;

[0065] And, each timing data segment is sent to the second chip according to the timing.

[0066] In a sixth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the data processing method as described in any one of the first aspect or the second aspect.

[0067] In a seventh aspect, the present application provides a computing device comprising a memory and at least one chip, wherein the memory stores a computer program, wherein when the at least one chip executes the computer program, the data processing method as described in any one of the first aspect or the second aspect is implemented.

[0068] Compared with the prior art, this application reduces the waiting time during data transmission by planning the data transmission path in advance, so that computing resources and data transmission resources can work together more efficiently, thereby significantly improving the processing speed and efficiency of subsequent data. Through a sophisticated synchronization control mechanism, the close coupling and efficient collaboration between data transmission and computing tasks are ensured, avoiding the waste of computing resources during data transmission. At the same time, it also makes more full use of hardware resources and improves the efficiency of hardware resource use. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] By reading the detailed description of the embodiments of the present application with reference to the accompanying drawings, the objectives, features and advantages of the embodiments of the present application will become easy to understand. Among them:

[0070] Figure 1 A flowchart of a data processing method proposed in an embodiment of the present application;

[0071] Figure 2 A flow chart of the stages of processing data for existing DMA technology;

[0072] Figure 3 A flow chart showing the stages of data processing by the DMA technology proposed in the embodiment of the present application;

[0073] Figure 4 A schematic diagram of the structure of a first chip proposed in an embodiment of the present application;

[0074] Figure 5 A schematic diagram of the structure of a second chip proposed in an embodiment of the present application;

[0075] Figure 6 A schematic diagram of the structure of a data processing system proposed in an embodiment of the present application;

[0076] Figure 7 A hardware structure block diagram of a server of a data processing method proposed in an embodiment of the present application.

[0077] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0078] The terms "first", "second", etc. in the description and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects (for example, the first chip and the second chip are respectively represented as different chips, and the others are similar), and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules does not have to be limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules that appear in the embodiments of the present application is only a logical division. There may be other division methods when implemented in actual applications, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling between modules, and the communication connection can be electrical or other similar forms, which are not limited in the embodiments of the present application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0079] In order to solve the technical problem that data transmission in the existing DMA technology mentioned in the background art is limited by a rigid serial processing flow, the present application proposes an embodiment of a data processing method. Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present invention. This specification provides method or process operation steps as shown in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one of many execution orders and does not represent the only execution order. In actual execution, the method or process sequence shown in the embodiment or the drawings may be executed or executed in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 1 As shown, the execution subject of this exemplary process can be any chip on the market, for example, it can be interactively executed by the first chip 10 and the second chip 20, and the method includes steps S01 to S13.

[0080] Step S01: the first chip 10 obtains the pre-transmission data volume and the pre-transmission path based on input data.

[0081] It should be noted that in this embodiment, the input data is pre-acquired. It should be noted that when a large-scale machine learning model processes data, it will iteratively process the input initial data multiple times to obtain output data. In other words, in the application scenario of data iterative processing, Figure 2 As shown, multiple rounds of calculations are performed, and the input data of each round of calculations is the output data of the previous round of calculations. Since this is a mature technology, it will not be described in detail here. In other words, in an embodiment of the present application, the input data can be the initial data input into a large-scale machine learning model, or it can be the output data of the previous round of iterative calculations of a large-scale machine learning model.

[0082] It should be clear that the method for each large-scale machine learning model to obtain output data based on input data is similar, that is, under the premise that the data processing method remains unchanged, if the amount of input data is fixed, the amount of output data obtained is also fixed. For example, taking matrix multiplication as an example, if a large-scale machine learning model obtains output data from input data based on matrix multiplication, its basic steps are to obtain an input matrix based on the input data, then obtain a weight matrix based on the input matrix, and then obtain an output matrix (that is, output data) based on the product of the input matrix and the weight matrix. Since matrix multiplication is a mature technology, it will not be described here. It is easy to understand that if the amount of input data is fixed, the size of the input matrix is ​​also fixed, and the size of the obtained output matrix is ​​also fixed, that is, the amount of output data is also fixed. In other words, in this embodiment, if the input data is obtained, the amount of output data after the calculation is completed (that is, the amount of pre-transmission data mentioned above) can be estimated even if no calculation is performed.

[0083] It should be clear that in this embodiment, the pre-transmission path is a path for transmitting the output data below from the first chip 10 to the second chip 20. The transmission path between chips is a mature technology and will not be described here. In an embodiment of the present application, the transmission path of the two chips can be set manually, or the transmission path between the two chips can be set by dynamic planning. For example, in an embodiment of the present application, the method disclosed in the patent document with application number CN202110400833.7 and titled A method for determining a data transmission path, a chip, and a storage medium can be used to dynamically plan the transmission path between the first chip 10 and the second chip 20.

[0084] Step S02: the first chip 10 obtains a total transmission duration based on the pre-transmission data volume and the pre-transmission path.

[0085] It is easy to understand that if the pre-transmission path is determined, the bandwidth of the pre-transmission path is determined. It should be noted that the data transmission speed depends on the bandwidth of the transmission path. In other words, if the pre-transmission path is determined, the data transmission speed is also determined. If the transmission speed and the amount of output data are determined, the total transmission time during the transmission process can be obtained. For example, in a specific embodiment of the present application, assuming that the amount of output data is 1000MB and the transmission speed is 20MB / s, the total transmission time = 1000MB / (20MB / s) = 50s.

[0086] Step S03 : the first chip 10 sends the pre-transmission data volume to the second chip 20 .

[0087] It should be noted that the technology of one chip (ie, the first chip 10 ) sending data to another chip (ie, the second chip 20 ) is a mature technology and will not be described in detail here.

[0088] Step S04 : the second chip 20 receives the pre-transmission data volume sent by the first chip 10 .

[0089] It should be noted that the technology of one chip (ie, the second chip 20 ) receiving data sent by another chip (ie, the first chip 10 ) is a mature technology and will not be described in detail here.

[0090] Step S05: the second chip 20 obtains the calculation duration based on the pre-transmission data volume and the calculation speed.

[0091] It should be clear that the computing speed of a chip is an important indicator of its performance, which is usually measured by the number of operations that can be performed per second, including million operations per second, floating-point operations per second, trillion operations per second and other indicator parameters. These indicators reflect the speed and ability of the chip in processing data. In an embodiment of the present application, the computing speed is pre-acquired by the second chip. Based on the amount of data and computing speed of the operation, obtaining the computing time is also a mature technology, which will not be elaborated here.

[0092] Step S06 : the first chip 10 sends the total transmission duration to the second chip 20 .

[0093] It should be noted that the technology of one chip (ie, the first chip 10 ) sending data to another chip (ie, the second chip 20 ) is a mature technology and will not be described in detail here.

[0094] It should be noted that in the embodiments of the present application, the sequence numbers of the steps do not represent the order in which the steps are executed, but are only used to distinguish different steps. For example, in the embodiments of the present application, step S03 may be executed first, and then step S06; step S06 may be executed first, and then step S03; or step S03 and step S06 may be executed simultaneously. In the embodiments of the present application, the sequence numbers of other steps are also the same, and will not be repeated here.

[0095] Step S07 : the second chip 20 receives the total transmission duration sent by the first chip 10 .

[0096] It should be noted that the technology of one chip (ie, the second chip 20 ) receiving data sent by another chip (ie, the first chip 10 ) is a mature technology and will not be described in detail here.

[0097] Step S08: The first chip 10 calculates output data based on the input data.

[0098] As can be seen from the foregoing, calculating output data based on the input data is a mature technology and will not be elaborated here.

[0099] Step S09 : the first chip 10 sends the output data to the second chip 20 .

[0100] It should be noted that the technology of one chip (ie, the first chip 10 ) sending data to another chip (ie, the second chip 20 ) is a mature technology and will not be described in detail here.

[0101] Step S10 : the second chip 20 receives the output data sent by the first chip 10 .

[0102] It should be noted that the technology of one chip (ie, the second chip 20 ) receiving data sent by another chip (ie, the first chip 10 ) is a mature technology and will not be described in detail here.

[0103] Step S11: the second chip 20 obtains the current transmission duration based on the output data.

[0104] As can be seen from the foregoing, if 1000MB of data transmission takes 50s, and the second chip 20 receives 200MB of output data, the current transmission duration is 10s. That is, in this embodiment, the current transmission duration is directly proportional to the amount of output data received by the second chip 20.

[0105] Step S12: the second chip 20 obtains the remaining transmission duration based on the total transmission duration and the current transmission duration.

[0106] It should be noted that in the embodiment of the present application, the remaining transmission duration = the total transmission duration - the current transmission duration. As mentioned above, if the total transmission duration is 50s and the current transmission duration is 20s, the remaining transmission duration = 50s - 20s = 30s.

[0107] Step S13: If the remaining transmission time is less than the calculation time, the second chip 20 starts the calculation task.

[0108] It should be noted that if the remaining transmission time is less than the calculation time of the second chip 20, it means that before the second chip 20 completes the calculation, all the output data of the first chip 10 (that is, the input data of the second chip 20) can be transmitted to the second chip 20. In other words, it is guaranteed that the calculation of the second chip 20 can be completed smoothly.

[0109] It should be clear that the data processing method proposed in the embodiment of the present application reduces the waiting time during the data transmission process by planning the data transmission path in advance, so that computing resources and data transmission resources can work together more efficiently, thereby significantly improving the processing speed and efficiency of subsequent data. Through a sophisticated synchronization control mechanism, the close coupling and efficient coordination between data transmission and computing tasks are ensured, avoiding the waste of computing resources in the data transmission process of the prior art. At the same time, it also makes more full use of hardware resources and improves the efficiency of hardware resource use. Specifically, Figure 2 and Figure 3 As shown, Figure 2 A flow chart of the stages of processing data for existing DMA technology; Figure 3 The flowchart of the stages of data processing by the improved DMA technology of the embodiment of the present application. It should be clear that each round of calculation based on the DMA technology is divided into three time stages, namely the DMA startup stage, the data transmission stage and the data calculation stage. This is a mature technology and will not be described in detail here. Figure 2 and Figure 3 As shown, A 1i Indicates the time spent in the DMA startup phase in the i-th round of calculation; A 2i A represents the time spent on the data transmission phase in the i-th round of calculation; 3i A represents the time spent on the data calculation phase in the i-th round of calculation; 1i+1 Indicates the time spent in the DMA startup phase in the i+1th round of calculation; A 2i+1 A represents the time spent on the data transmission phase in the i+1th round of calculation; 3i+1 Indicates the time spent on the data calculation phase in the i+1th round of calculation, where i is a positive integer greater than or equal to 1. Figure 2It can be seen that in the process of processing data with the existing DMA technology, the data calculation phase can only be performed after the data transmission phase is completed. Figure 3 It can be seen that in each round of calculation process of the DMA technology improved by the embodiment of the present application, the data transmission stage and the data calculation stage can be carried out in parallel. In other words, the data processing method proposed in this embodiment can make the data transmission and calculation tasks in the DMA technology parallel, that is, the data transmission will not be limited to the rigid serial processing flow, so that the data transmission time is shorter, and the response speed of large-scale machine learning models can be improved.

[0110] As can be seen from the foregoing, the data output after the calculation of the first chip 10 is the input data when the second chip 20 calculates. Since the present embodiment changes the previous serial processing flow into a parallel processing flow, that is, data transmission and calculation tasks may be performed simultaneously at the same time. When the storage space of the second chip 20 is large enough (that is, the storage space of the second chip 20 can accommodate all the output data of the first chip 10 at the same time), each time the first chip 10 calculates an output data, the output data can be transmitted to the second chip 20. In most cases, the storage space of the second chip 20 is limited. In order to avoid overflow of the output data sent by the first chip 10 after reaching the second chip 20 in the serial mode, and to ensure the continuity and timeliness of the output data supply, that is, to ensure the continuity and timeliness of the second chip 20 obtaining the output data of the first chip 10, in one embodiment of the present application, step S09, before the first chip 10 sends the output data to the second chip 20, the method includes steps S091 to S094.

[0111] Step S091: the second chip 20 obtains the remaining storage space.

[0112] It should be noted that obtaining the remaining storage space is a mature technology and will not be described in detail here.

[0113] Step S092 : if the remaining storage space is greater than a preset threshold, the second chip 20 sends a transmission signal to the first chip 10 .

[0114] It should be clear that in the embodiments of the present application, the preset threshold can be set as required. For example, the preset threshold can be 20MB or 30MB, etc. That is to say, if the remaining storage space is greater than 20MB or 30MB, the first chip 10 can send output data to the second chip 20. In order to ensure the continuity and timeliness of the output data of the first chip 10, assuming that the speed at which the second chip 20 processes input data (that is, the output data of the first chip 10) is a, the speed at which the first chip 10 transmits output data to the second chip 20 is b (b is greater than a), and the total capacity of the storage space that can be used by the second chip 20 is A, the preset threshold calculation formula under ideal conditions is as follows:

[0115]

[0116] Wherein, X is the preset threshold; is the total capacity of the storage space that can be used by the second chip 20; a is the speed at which the second chip 20 processes input data; and b is the speed at which the first chip 10 transmits output data to the second chip 20. Setting the preset threshold in this way can not only ensure the continuity and timeliness of the output data supply, but also ensure that the output data will not overflow after reaching the second chip 20, and will not occupy a large amount of storage space of the second chip 20. Of course, in other embodiments of the present application, the preset threshold may also be less than X, for example: the preset threshold may be 0.9X or 0.8X, etc.

[0117] Step S093: the first chip 10 receives the transmission signal.

[0118] In this embodiment, the transmission signal is obtained by sending the second chip 20. It should be noted that it is a mature technology for one chip (ie, the first chip 10) to receive data sent by another chip (ie, the second chip 20), which will not be described in detail here.

[0119] Step S094 : the first chip 10 sends the output data to the second chip 20 based on the transmission signal.

[0120] It should be clear that in the embodiment of the present application, each time the first chip 10 receives a transmission signal, it transmits output data to the second chip 20. In order to avoid the output data overflowing in the second chip 20, in the present embodiment, the first chip 10 transmits a preset amount of output data to the second chip 20 each time. It is easy to understand that the storage space occupied by the preset amount of output data is less than the preset threshold value mentioned above. For example, if the preset threshold value is 20MB, each time the first chip 10 transmits output data to the second chip 20, 18MB (i.e., the preset amount) of data can be transmitted.

[0121] In order to transmit as much output data as possible to the second chip 20 each time to ensure the continuity and timeliness of the output data supply, in one embodiment of the present application, step S094, the first chip 10 sends the output data to the second chip 20 based on the transmission signal also includes steps S095 to S097.

[0122] In this embodiment, the transmission signal at least includes information indicating the remaining storage space of the second chip 20 .

[0123] Step S095: the first chip 10 obtains a plurality of time series data segments based on the output data.

[0124] In an embodiment of the present application, the output data is divided into a plurality of data segments (also referred to as time series data segments) for subsequent sequential transmission, and the storage space occupied by each data segment is smaller than the remaining storage space. That is, in an embodiment of the present application, as much output data as possible is transmitted to the second chip 20 while avoiding data overflow.

[0125] Step S096: the first chip 10 sends each timing data segment to the second chip 20 according to the timing.

[0126] It should be noted that the technology of one chip (ie, the first chip 10 ) sending multiple timing data to another chip (ie, the second chip 20 ) is a mature technology and will not be elaborated here.

[0127] Step S097: After receiving each time series data segment, the second chip 20 splices them together to form the output data.

[0128] It should be noted that concatenating time series data segments and restoring them to their original data is also a mature technology and will not be elaborated here.

[0129] After introducing the data processing method proposed in the embodiment of the present application, the following introduces a chip proposed in the embodiment of the present application, such as Figure 4 As shown, the first chip 10 includes:

[0130] The first processing module 11 is used to obtain the pre-transmission data amount and the pre-transmission path based on the input data; the input data is pre-acquired;

[0131] And, based on the pre-transmission data amount and the pre-transmission path, obtaining a total transmission duration;

[0132] The first transceiver module 12 is used to send the pre-transmission data amount to the second chip 20, so that the second chip 20 can obtain the calculation time based on the pre-transmission data amount and the calculation speed; the calculation speed is pre-acquired by the second chip 20;

[0133] And, sending the total transmission duration to the second chip 20;

[0134] The first processing module 11 is further used to calculate output data based on the input data;

[0135] The first transceiver module 12 is further configured to send the output data to the second chip 20 , so that the second chip 20 can obtain the current transmission duration based on the output data.

[0136] In some embodiments, the first transceiver module 12 is further used to obtain a transmission signal; the transmission signal is obtained by the second chip 20 based on the remaining storage space of the second chip 20;

[0137] And, sending the output data to the second chip 20 based on the transmission signal.

[0138] In some embodiments, the transmission signal at least includes the remaining storage space of the second chip 20; the first processing module 11 is further used to obtain multiple time series data segments based on the output data; the storage space occupied by each time series data segment is smaller than the remaining storage space;

[0139] The first transceiver module 12 is further configured to send each time sequence data segment to the second chip 20 in time sequence.

[0140] It should be clear that the embodiment of the first chip proposed in this application reduces the waiting time during data transmission by planning the data transmission path in advance, so that computing resources and data transmission resources can work together more efficiently, thereby significantly improving the processing speed and efficiency of subsequent data. Through a sophisticated synchronization control mechanism, the close coupling and efficient collaboration between data transmission and computing tasks are ensured, avoiding the waste of computing resources during data transmission. At the same time, it also makes more full use of hardware resources and improves the efficiency of hardware resource use.

[0141] After introducing the first chip proposed in the embodiment of the present application, the following introduces a chip proposed in the embodiment of the present application, such as Figure 5 As shown, the second chip 20 includes:

[0142] The second transceiver module 22 is used to receive the pre-transmission data volume sent by the first chip 10; the pre-transmission data volume is obtained by the first chip 10 based on the input data; the input data is pre-acquired by the first chip 10;

[0143] A second processing module 21 is used to obtain a calculation duration based on the pre-transmitted data volume and a calculation speed; the calculation speed is pre-acquired by the chip;

[0144] The second transceiver module 22 is further used to receive the total transmission time sent by the first chip 10; the total transmission time is obtained by the first chip 10 based on the pre-transmission data volume and the pre-transmission path; the pre-transmission path is obtained by the first chip 10 based on the input data;

[0145] and receiving output data sent by the first chip 10; the output data is calculated by the first chip 10 based on the input data;

[0146] The second processing module 21 is further used to obtain the current transmission duration based on the output data;

[0147] And, based on the total transmission duration and the current transmission duration, obtaining a remaining transmission duration;

[0148] And, if the remaining transmission duration is less than the calculation duration, the calculation task is started.

[0149] In some embodiments, the second processing module 21 is further used to obtain remaining storage space;

[0150] The second transceiver module 22 is further configured to send a transmission signal to the first chip 10 if the remaining storage space is greater than a preset threshold, so that the first chip 10 can send output data to the chip based on the transmission signal.

[0151] It should be clear that the embodiment of the second chip proposed in this application reduces the waiting time during data transmission by planning the data transmission path in advance, so that computing resources and data transmission resources can work together more efficiently, thereby significantly improving the processing speed and efficiency of subsequent data. Through a sophisticated synchronization control mechanism, the close coupling and efficient collaboration between data transmission and computing tasks are ensured, avoiding the waste of computing resources during data transmission. At the same time, it also makes fuller use of hardware resources and improves the efficiency of hardware resource use.

[0152] After introducing the second chip proposed in the embodiment of the present application, the following introduces a data processing system proposed in the embodiment of the present application. Figure 6 As shown, the data processing system 30 includes a first chip 10 and a second chip 20;

[0153] The first chip 10 is used to obtain the pre-transmission data amount and the pre-transmission path based on the input data; the input data is pre-acquired;

[0154] And, based on the pre-transmission data amount and the pre-transmission path, obtaining a total transmission duration;

[0155] and, sending the pre-transmission data amount to the second chip 20;

[0156] The second chip 20 is used to obtain the calculation time based on the pre-transmitted data amount and the calculation speed; the calculation speed is pre-acquired by the second chip 20;

[0157] The first chip 10 is further used to send the total transmission duration to the second chip 20;

[0158] And, based on the input data, calculating output data;

[0159] and, sending the output data to the second chip 20;

[0160] The second chip 20 is further used to obtain the current transmission duration based on the output data;

[0161] And, based on the total transmission duration and the current transmission duration, obtaining a remaining transmission duration;

[0162] And, if the remaining transmission duration is less than the calculation duration, the calculation task is started.

[0163] In some embodiments, the second chip 20 is further used to obtain the remaining storage space;

[0164] and, if the remaining storage space is greater than a preset threshold, sending a transmission signal to the first chip 10;

[0165] The first chip 10 is further configured to send the output data to the second chip 20 based on the transmission signal.

[0166] In some implementations, the first chip 10 is further used to obtain multiple time series data segments based on the output data; the storage space occupied by each time series data segment is smaller than the remaining storage space;

[0167] And, each timing data segment is sent to the second chip 20 according to the timing.

[0168] It should be clear that the embodiment of the data processing system proposed in this application reduces the waiting time during the data transmission process by planning the data transmission path in advance, so that computing resources and data transmission resources can work together more efficiently, thereby significantly improving the processing speed and efficiency of subsequent data. Through a sophisticated synchronization control mechanism, the close coupling and efficient collaboration between data transmission and computing tasks are ensured, avoiding the waste of computing resources during the data transmission process. At the same time, it also makes fuller use of hardware resources and improves the efficiency of hardware resource use.

[0169] After introducing the data processing system proposed in the embodiment of the present application, the following introduces a computer-readable storage medium proposed in the embodiment of the present application, which includes instructions. When the instructions are run on a computer, the computer executes the data processing method described in any of the above embodiments.

[0170] After introducing the computer-readable storage medium proposed in the embodiments of the present application, a computing device proposed in the embodiments of the present application is introduced below, which includes a memory and at least one chip, and a computer program is stored in the memory, wherein the at least one chip implements the data processing method described in any of the above embodiments when executing the computer program.

[0171] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal, a computing device, a server or a similar computing device. Taking running on a server as an example, Figure 7 is a hardware structure block diagram of a server of the data processing method provided in the embodiment of the present application. Figure 7 As shown, the server 700 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 701 (the central processing unit 701 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 709 for storing data, and one or more storage media 708 (such as one or more storage devices) for storing application programs 707 or data 706. Among them, the memory 709 and the storage medium 708 can be temporary storage or permanent storage. The program stored in the storage medium 708 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 701 can be configured to communicate with the storage medium 708 and execute a series of instruction operations in the storage medium 708 on the server 700. The server 700 may also include one or more power supplies 702, one or more wired or wireless network interfaces 703, one or more input and output interfaces 704, and / or one or more operating systems 705, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0172] The input / output interface 704 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 700. In one example, the input / output interface 704 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices through a base station so as to communicate with the Internet. In one example, the input / output interface 704 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0173] It can be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown.

[0174] An embodiment of the present application also relates to a chip system, which includes at least one processor and an interface circuit, wherein the processor includes multiple vector storage units, and the processor is used to execute instructions and / or data interaction through the interface circuit, so that the chip system executes the method of any of the above embodiments.

[0175] In a possible implementation, the chip system may also directly include a memory, in which a computer program or computer instructions are stored.

[0176] For example, the memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0177] An embodiment of the present application further relates to a processor, which includes multiple storage units for calling a computer program or computer instruction stored in the memory so that the processor executes the method described in any of the above embodiments.

[0178] For example, in an embodiment of the present application, the processor is an integrated circuit chip with signal processing capabilities. For example, the processor can be an FPGA, a general-purpose processor, a DSP, an ASIC or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, a SoC, a CPU, a network processor (network processor, NP), a microcontroller (micro controller unit, MCU), a PLD or other integrated chip, which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. In a possible implementation, the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a program code, and when the program code is run on the computer, the computer executes the above method embodiment.

[0179] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0181] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0182] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0184] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0185] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0186] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. A data processing method, applied to a first chip; characterized in that: The method comprises: Based on the input data, obtaining the pre-transmission data amount and the pre-transmission path; the input data is obtained in advance; Based on the pre-transmission data amount and the pre-transmission path, obtaining a total transmission duration; sending the pre-transmitted data amount to the second chip, so that the second chip can obtain the calculation duration based on the pre-transmitted data amount and the calculation speed; the calculation speed is pre-acquired by the second chip; Sending the total transmission duration to the second chip; Based on the input data, output data is calculated; The output data is sent to the second chip, so that the second chip can obtain the current transmission duration based on the output data.

2. The data processing method according to claim 1, characterized in that: The step of sending the output data to the second chip comprises: Acquire a transmission signal; the transmission signal is obtained by the second chip based on the remaining storage space of the second chip; The output data is sent to the second chip based on the transmission signal.

3. The data processing method according to claim 2, characterized in that: The transmission signal at least includes the remaining storage space of the second chip; The step of sending the output data to the second chip based on the transmission signal comprises: Based on the output data, a plurality of time series data segments are acquired; the storage space occupied by each time series data segment is smaller than the remaining storage space; Each time sequence data segment is sent to the second chip in time sequence.

4. A data processing method, applied to a second chip; characterized in that: The method comprises: receiving a pre-transmission data amount sent by a first chip; the pre-transmission data amount is acquired by the first chip based on input data; the input data is pre-acquired by the first chip; Based on the pre-transmitted data volume and the calculation speed, obtaining the calculation duration; the calculation speed is obtained in advance by the second chip; receiving a total transmission duration sent by the first chip; the total transmission duration is acquired by the first chip based on the amount of pre-transmission data and a pre-transmission path; the pre-transmission path is acquired by the first chip based on input data; receiving output data sent by the first chip; the output data is calculated by the first chip based on the input data; Based on the output data, obtain the current transmission duration; Based on the total transmission duration and the current transmission duration, obtaining a remaining transmission duration; If the remaining transmission duration is less than the calculation duration, the calculation task is started.

5. The data processing method according to claim 4, characterized in that: Before receiving the output data sent by the first chip, the method includes: Get the remaining storage space; If the remaining storage space is greater than a preset threshold, a transmission signal is sent to the first chip, so that the first chip can send output data to the second chip based on the transmission signal.

6. A chip, characterized in that: include: A first processing module, configured to obtain a pre-transmission data volume and a pre-transmission path based on input data; The input data is obtained in advance; And, based on the pre-transmission data amount and the pre-transmission path, obtaining a total transmission duration; A first transceiver module is used to send the pre-transmission data amount to the second chip, so that the second chip can obtain the calculation time based on the pre-transmission data amount and the calculation speed; the calculation speed is pre-acquired by the second chip; And, sending the total transmission duration to the second chip; The first processing module is further used to calculate output data based on the input data; The first transceiver module is further used to send the output data to the second chip, so that the second chip can obtain the current transmission duration based on the output data.

7. The chip according to claim 6, characterized in that: The first transceiver module is further used to obtain a transmission signal; the transmission signal is obtained by the second chip based on the remaining storage space of the second chip; And, sending the output data to the second chip based on the transmission signal.

8. The chip according to claim 7, characterized in that: The transmission signal at least includes the remaining storage space of the second chip; the first processing module is further used to obtain a plurality of time series data segments based on the output data; the storage space occupied by each time series data segment is smaller than the remaining storage space; The first transceiver module is also used to send each timing data segment to the second chip in a timing sequence.

9. A chip, characterized in that: include: A second transceiver module, used for receiving the pre-transmission data volume sent by the first chip; The pre-transmission data amount is acquired by the first chip based on input data; The input data is pre-acquired by the first chip; A second processing module is used to obtain a calculation duration based on the pre-transmitted data volume and a calculation speed; the calculation speed is pre-acquired by the chip; The second transceiver module is further used to receive the total transmission duration sent by the first chip; The total transmission duration is acquired by the first chip based on the pre-transmission data volume and the pre-transmission path; the pre-transmission path is acquired by the first chip based on input data; and, receiving output data sent by the first chip; the output data is calculated by the first chip based on the input data; The second processing module is further used to obtain the current transmission duration based on the output data; And, based on the total transmission duration and the current transmission duration, obtaining a remaining transmission duration; And, if the remaining transmission duration is less than the calculation duration, the calculation task is started.

10. The chip according to claim 9, characterized in that: The second processing module is further used to obtain remaining storage space; The second transceiver module is further configured to send a transmission signal to the first chip if the remaining storage space is greater than a preset threshold, so that the first chip can send output data to the chip based on the transmission signal.

11. A data processing system, characterized in that: comprising a first chip and a second chip; The first chip is used to obtain the pre-transmission data amount and the pre-transmission path based on the input data; the input data is pre-acquired; And, based on the pre-transmission data amount and the pre-transmission path, obtaining a total transmission duration; and, sending the pre-transmission data amount to a second chip; The second chip is used to obtain the calculation duration based on the pre-transmitted data amount and the calculation speed; the calculation speed is pre-acquired by the second chip; The first chip is further configured to send the total transmission duration to the second chip; And, based on the input data, calculating output data; and, sending the output data to the second chip; The second chip is further used to obtain the current transmission duration based on the output data; And, based on the total transmission duration and the current transmission duration, obtaining a remaining transmission duration; And, if the remaining transmission duration is less than the calculation duration, the calculation task is started.

12. The data processing system according to claim 11, characterized in that: The second chip is also used to obtain remaining storage space; and, if the remaining storage space is greater than a preset threshold, sending a transmission signal to the first chip; The first chip is further configured to send the output data to the second chip based on the transmission signal.

13. The data processing system according to claim 12, characterized in that: The first chip is further used to obtain multiple time series data segments based on the output data; the storage space occupied by each time series data segment is smaller than the remaining storage space; And, each timing data segment is sent to the second chip according to the timing.

14. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the data processing method according to any one of claims 1 to 5.

15. A computing device, characterized in that: It comprises a memory and at least one chip, wherein the memory stores a computer program, wherein the at least one chip implements the data processing method according to any one of claims 1 to 5 when executing the computer program.

Images