Data processing method applied to dispersion system, and apparatus, equipment, and medium therefor

Abstract

To provide a data processing method, an apparatus, equipment, and a medium applied to a dispersion system.SOLUTION: A method according to the present invention has the steps of dividing each of a plurality of data blocks included in data to be processed into a plurality of first subblocks corresponding to a plurality of respective computation unit groups, dividing the first subblock into a plurality of second subblocks for a plurality of computation units in a target computation unit group, processing the subblock with the use of the computation unit corresponding to each of the second subblocks to determine a plurality of first processing results, and executing data reduction operation for the plurality of first processing results with the use of the target computation unit group to determine the processing results of each of the first subblocks.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer technology, in particular to the field of chip technology and distributed data processing technology, and specifically to a data processing method and apparatus, electronic equipment, computer-readable storage medium, and computer program product applied to a distributed system. [Background technology]

[0002] Artificial intelligence is a field that studies how computers can simulate some human thought processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing, while artificial intelligence software technologies mainly include several major fields such as computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0003] With the development of artificial intelligence technology, more and more applications are based on artificial intelligence technology to achieve results far beyond those of traditional algorithms. Deep learning is a data-intensive and computation-intensive algorithm. In order to improve the learning speed and inference speed of large-scale deep learning models, data processing can be performed using a distributed system including multiple computing units or multiple computing unit groups, thereby meeting the computing power requirements.

[0004] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, it should not be assumed that any approach described in this section constitutes prior art merely by virtue of its inclusion in this section. Likewise, unless otherwise noted, it should not be assumed that any problems addressed in this section have been recognized in any prior art. Summary of the Invention

[0005] The present disclosure provides a data processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product that are applied to a distributed system.

[0006] According to one aspect of the present disclosure, there is provided a method for processing data applied to a distributed system, wherein a plurality of computing units in the distributed system can constitute a plurality of computing unit groups, and data to be processed includes a plurality of data blocks, the method comprising: For each data block of the plurality of data blocks, Dividing the data block into a plurality of first sub-blocks corresponding to the plurality of computing unit groups, respectively; Dividing each first sub-block of the plurality of first sub-blocks into a plurality of second sub-blocks, each second sub-block corresponding to a plurality of computing units in the target computing unit group, based on a target computing unit group corresponding to each first sub-block; determining a plurality of first processing results output by the plurality of calculation units by processing the second sub-blocks using calculation units corresponding to the second sub-blocks; and determining a processing result for each of the first sub-blocks by performing a data reduction operation on the plurality of first processing results using the target computing unit group.

[0007] According to another aspect of the present disclosure, there is provided an apparatus for processing data applied to a distributed system, wherein a plurality of computing units in the distributed system can constitute a plurality of computing unit groups, and data to be processed includes a plurality of data blocks, and the apparatus includes: For each data block of the plurality of data blocks, a first division unit configured to divide the data block into a plurality of first sub-blocks corresponding to the plurality of computing unit groups, respectively; a second division unit configured to divide, based on a target computing unit group corresponding to each of the first sub-blocks, each of the first sub-blocks into a plurality of second sub-blocks, each of the second sub-blocks corresponding to a plurality of computing units in the target computing unit group; a first processing unit configured to process the second sub-blocks using a calculation unit corresponding to each of the second sub-blocks to determine a plurality of first processing results output by the plurality of calculation units, respectively; and a second processing unit configured to determine a processing result for each of the first sub-blocks by performing a data reduction operation on the plurality of first processing results using the target computing unit group.

[0008] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the at least one processor to perform the above-described method of processing data.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause the computer to perform the method for processing data as described above.

[0010] According to another aspect of the present disclosure, there is provided a computer program product including a computer program, which when executed by a processor is capable of implementing the above-mentioned method for processing data.

[0011] According to one or more embodiments of the present disclosure, the stability of data processing results can be guaranteed, and the accuracy of distributed data processing can be further improved.

[0012] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following description. [Brief explanation of the drawings]

[0013] The drawings illustratively illustrate examples, constitute a part of the specification, and together with the written description serve to explain exemplary embodiments of the examples. The illustrated examples are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar, but not necessarily identical, elements.

[0014] [Figure 1] 1 is a schematic diagram of an example system in which various methods described herein may be implemented, according to an example embodiment of the present disclosure. [Figure 2] 1 is a flowchart illustrating a data processing method applied to a distributed system, according to an exemplary embodiment of the present disclosure. [Figure 3] 1 is a structural schematic diagram illustrating a distributed system according to an exemplary embodiment of the present disclosure. [Figure 4]FIG. 2 is a schematic diagram illustrating a data processing process according to an exemplary embodiment of the present disclosure. [Figure 5] FIG. 2 is a schematic diagram illustrating a data processing process according to an exemplary embodiment of the present disclosure. [Figure 6] 1 is a structural block diagram illustrating a data processing apparatus applied in a distributed system according to an exemplary embodiment of the present disclosure; [Figure 7] FIG. 1 is a structural block diagram illustrating an example electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015]

[0023] The following describes exemplary embodiments of the present disclosure in conjunction with the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included therein, but they should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the following description omits descriptions of known functions and structures.

[0016] In this disclosure, unless otherwise specified, the use of terms such as "first," "second," etc. to describe various elements is not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of the element, or in some cases, may refer to different instances based on the context.

[0017] The terms used in the description of various examples of the present disclosure are intended only to describe particular examples and are not intended to be limiting. Unless the context clearly indicates otherwise, and unless the number of elements is specifically limited, the element may be one or more. Furthermore, as used in this disclosure, the term "and / or" covers any and all possible combinations of the listed items.

[0018] In a distributed system, multiple groups of computing units can be used to perform data calculations and collective communication operations to achieve data synchronization. Existing technologies typically dynamically allocate data to be processed based on current load conditions, which may result in the same data being allocated to different groups of computing units in different rounds. As can be seen, when using a distributed system to process data, various types of collective communication operations must be used to achieve data synchronization and data exchange between each node in the system. Differences in the data collection logic of different groups of computing units may affect the stability of the calculation results.

[0019] For example, when multiple computing units in a distributed system can form multiple computing unit groups in a ring topology, each computing unit group in the ring topology can use a ring all reduce algorithm to reduce the data results of each computing unit in the computing unit group. The data exchange and accumulation steps in the ring all reduce algorithm process are based on a unidirectional ring topology and are steps of sequentially accumulating and adding data in each computing unit along a fixed direction. Therefore, if the data accumulation order (i.e., the arrangement order of each computing unit in the ring topology) is different, the accumulation result may differ, and for example, the stability of the data processing result may be affected by a rounding operation in the addition calculation.

[0020] Based on this, the present disclosure provides a method for processing data, in which when multiple computing units in a system can form multiple computing unit groups, data blocks in the data to be processed are used as the division basis, and each data block is divided into multiple sub-blocks and assigned to multiple computing unit groups, thereby ensuring the stability of the division-mapping scheme between the data blocks and the computing unit groups, and further ensuring the stability of the data processing results.

[0021] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0022] 1 illustrates a schematic diagram of an example system 100 capable of implementing various methods and apparatus described herein in accordance with embodiments of the present disclosure. Referring to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to run one or more applications.

[0023] In an embodiment of the present disclosure, the server 120 operates to perform one or more services or software applications of data processing methods.

[0024] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.

[0025] In the configuration shown in FIG. 1 , server 120 may include one or more assemblies that implement the functionality performed by server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that may be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may sequentially utilize one or more client applications to interact with server 120 to utilize services provided by these assemblies. It should be understood that a variety of different system configurations are possible and may differ from system 100. Thus, FIG. 1 is merely one example of a system for implementing various methods described herein and is not intended to be limiting.

[0026] A user may use client devices 101, 102, 103, 104, 105, and / or 106 to submit data processing requests or data to be processed. The client devices may provide an interface through which a user of the client device interacts with the client device. The client devices may also output information to the user through the interface. Although only six client devices are illustrated in FIG. 1, one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0027] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general-purpose computers such as personal computers and laptop computers, workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors and other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems, such as, for example, Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (e.g., Google Chrome OS), and may include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include mobile phones, intelligent phones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, etc. The client device may run a variety of applications, such as Internet-related applications, communication applications (eg, email applications), short message service (SMS) applications, and the like, and may use a variety of communication protocols.

[0028] Network 110 may be any type of network known to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WiFi, etc.), and / or any combination of these and / or other networks.

[0029] Server 120 may include one or more general-purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers, etc.), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization (e.g., one or more flexible pools of virtualized logical storage devices to maintain virtual storage devices for the server). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0030] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above and any commercial server operating system. Server 120 may also run any one of a variety of additional server and / or middle-tier applications, such as an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0031] In some embodiments, server 120 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0032] In some embodiments, server 120 may be a server of a distributed system or a server incorporating blockchain. Server 120 may be a cloud server, or a smart cloud computing server or smart cloud host equipped with artificial intelligence technology. A cloud server is a host product in a cloud computing service system that solves the drawbacks of traditional physical hosts and virtual private server (VPS) services, such as high management difficulty and poor business scalability.

[0033] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data or other information. For example, one or more of databases 130 may be used to store information such as audio files or video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be remote from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of different types. In some embodiments, a database used by server 120 may be a relational database. One or more of these databases may store, update, and retrieve data from the databases in response to instructions.

[0034] In some embodiments, one or more of databases 130 may be used by an application to store data for the application. The databases used by the application may be different types of databases, such as key-value repositories, object repositories, general-purpose repositories backed by a file system, etc.

[0035] The system 100 of FIG. 1 can be configured and operated in a variety of ways to accommodate the various methods and apparatus described in accordance with this disclosure.

[0036] 2 is a flowchart illustrating a method 200 for processing data according to an exemplary embodiment of the present disclosure, where multiple computing units in the distributed system may form multiple computing unit groups, and the data to be processed includes multiple data blocks. As shown in FIG. 2, the method 200 includes: A step S201 of dividing each of the plurality of data blocks into a plurality of first sub-blocks respectively corresponding to the plurality of computing unit groups; A step S202 of dividing each first sub-block among the plurality of first sub-blocks into a plurality of second sub-blocks, each second sub-block corresponding to a plurality of computing units in the target computing unit group, based on a target computing unit group corresponding to each first sub-block; a step S203 of determining a plurality of first processing results output by the plurality of calculation units by processing the second sub-blocks using calculation units corresponding to the second sub-blocks; and step S204 of determining a processing result for each of the first sub-blocks by performing a data reduction operation on the plurality of first processing results using the target computing unit group.

[0037] By applying the above method 200, when there are multiple computing unit groups in a system, each data block in the data to be processed is used as the division basis to divide the data block into multiple first sub-blocks, and then the sub-blocks are assigned to multiple computing unit groups, thereby ensuring the stability of the division-mapping scheme between the data block and the computing unit group, and further ensuring the stability of the data processing results.

[0038] In some examples, the data block information in the data to be processed may be explicit information, for example, the data to be processed may be a data block sequence consisting of multiple data blocks that are naturally separated. In some examples, the data to be processed may be first divided into multiple data blocks before step S201, for example, the data to be processed may be divided into multiple data blocks according to a predetermined block size.

[0039] In some examples, a computing unit group may be a collection of computing units in a distributed system that can complete a set of collective communication operations. As can be understood, the computing units included in different computing unit groups may be the same or different. For example, if the collective communication operation is a ring all-reduce operation performed based on a unidirectional ring topology, and the computing units in the distributed system can form N ring topologies, the number of computing unit groups in the distributed system is 2N.

[0040] According to some embodiments, the plurality of computing units in the target computing unit group form a ring topology, and in step S204, determining each processing result of the first sub-block by performing a data reduction operation on the plurality of first processing results using the target computing unit group includes: sequentially accumulating the first processing results of each computing unit based on the arrangement order of the plurality of computing units in the target computing unit group in the ring topology; and determining each processing result of the first sub-block based on the accumulated results. In this way, in the process of realizing collective communication based on the ring-all-reduce algorithm, by applying the data sharding and computing power allocation method described in method 200, it is possible to ensure that the accumulation order of each intermediate processing result of each computing unit does not change and further ensure the stability of the accumulated results.

[0041] According to some embodiments, the step of sequentially accumulating the first processing results corresponding to each of the computing units in the target computing unit group based on the arrangement order of the computing units in the ring topology includes: using each computing unit of the plurality of computing units to receive data to be accumulated from a preceding computing unit located before the computing unit in the ring topology, determining intermediate data by performing an accumulation operation based on the data to be accumulated and the first processing results of the computing unit, and transmitting the intermediate data to a subsequent computing unit located after the computing unit in the ring topology so that the subsequent computing unit can perform an accumulation operation based on the intermediate data and the first processing results of the subsequent computing unit. In this way, the data reduction operation of all computing units can be more efficiently realized based on the ring iteration operation.

[0042] FIG. 3 illustrates a schematic diagram of a distributed system 300 according to an exemplary embodiment of the present disclosure. As illustrated in FIG. 3 , in this example, system 300 includes two processing cores (processing core 311 and processing core 312), four communication interfaces (composed of communication interface 321, communication interface 322, communication interface 323, and communication interface 324), and eight computing units. In the diagram, dashed-dotted double-headed arrows indicate first communication links between the processing cores, dashed double-headed arrows indicate second communication links between the processing cores and the computing units, and solid double-headed arrows indicate third communication links between the computing units. Referring to the multiple sets of point-to-point third communication links among the eight computing units illustrated in FIG. 3 , the eight nodes can configure the following four sets of unidirectional rings used to perform data accumulate-add operations:

[0043] Unidirectional ring A: calculation unit 331 - calculation unit 332 - calculation unit 336 - calculation unit 335 - calculation unit 337 - calculation unit 338 - calculation unit 334 - calculation unit 333; Unidirectional ring B: calculation unit 333 - calculation unit 334 - calculation unit 338 - calculation unit 337 - calculation unit 335 - calculation unit 336 - calculation unit 332 - calculation unit 331; Unidirectional ring C: calculation unit 331 - calculation unit 335 - calculation unit 338 - calculation unit 336 - calculation unit 337 - calculation unit 333 - calculation unit 332 - calculation unit 334; Unidirectional ring D: calculation unit 334 - calculation unit 332 - calculation unit 333 - calculation unit 337 - calculation unit 336 - calculation unit 338 - calculation unit 335 - calculation unit 331.

[0044] As can be seen, when the interconnection structure between multiple computing units has already been determined, multiple unidirectional ring topologies in the system can be determined based on various existing probing tools and analysis algorithms and further saved as explicit topology information, which can be directly applied when performing the method 200 described in the present disclosure.

[0045] In some examples, the processing core may be a CPU (Central Processing Unit), the compute node may be a GPU (Graphics Processing Unit), the first communication link may be a QPI (Quick Path Interconnect) link, the second communication link may be a PCI (Peripheral Component Interconnect) link, and the third communication link may be an NVLink-based point-to-point communication link.

[0046] As described above, conventional techniques typically dynamically allocate data exchange and accumulation tasks based on the load status of each of multiple ring topologies, and can evenly distribute data to be processed in the same batch across multiple ring topologies based on, for example, the data size (taking into account the batch size parameter batch_size, which is a variable parameter). In this case, when a certain accumulation operation for data processing request A with a data size of 8192 and a certain accumulation operation for data processing request B with a data size of 8192 are merged into the same batch (i.e., batch_size=2), there is a possibility that the first 4096 pieces of data of data processing request A will be allocated to unidirectional ring A, so node data accumulation is performed according to the node arrangement order in unidirectional ring A. However, there is a possibility that the last 4096 pieces of data will be allocated to unidirectional ring B, so node data accumulation is performed according to the node arrangement order in unidirectional ring B. If the accumulation operation in data processing request A is placed in one batch (i.e., batch_size=1) alone, 8192 data items of data processing request A may be allocated to the above four unidirectional rings on average, with each unidirectional ring being responsible for processing 2048 data items. Obviously, in such an embodiment, the accumulation order in the accumulation operation of each data element included in data processing request A may change, which may cause the calculation result to become unstable.

[0047] In one example, when the size of input data to be processed is 8, the data to be processed may be divided and assigned to multiple computing unit groups for processing. When the number of computing unit groups in the distributed system is 2, the size of the first division result divided based on the data to be processed is 4.

[0048] Based on this, within each computing unit group, the first division result of the computing unit group can be processed using a multi-step operation. Figure 4 is a schematic diagram showing a process of processing data according to an exemplary embodiment of the present disclosure. As shown in Figure 4, two computing unit groups consisting of computing unit 0, computing unit 1, computing unit 2, and computing unit 3 can realize data accumulation and addition based on ring iteration steps, and each computing unit performs transmission and accumulation and addition on the second division result having a size of 1 in each step. Figure 4 is a schematic diagram showing the transmission process of the first step. In this case, the cumulative addition result obtained by calculation unit group A is [a0+b0+c0+d0, b1+c1+d1+a1, c2+d2+a2+b2, d3+a3+b3+c3], and the cumulative addition result obtained by calculation unit group B is [d4+a4+b4+c4, c5+d5+a5+b5, b6+c6+d6+a6, a7+b7+c7+d7].

[0049] In this example, assuming that the length of each data processing request is 4, the input data to be processed includes two data processing requests (for example, data processing request query0=[a0,a1,a2,a3] of calculation unit 0 and data processing request query1=[a4,a5,a6,a7]), and the two data processing requests are assigned to calculation unit group A and calculation unit group B, respectively, for processing. Obviously, since the data accumulation addition orders of calculation unit group A and calculation unit group B are different, differences may occur in the processing results of different data processing requests.

[0050] In another example, assuming that the length of each data processing request is 2, the input data to be processed includes four data processing requests, and each computing unit group is responsible for processing two data processing requests (for example, data processing request query0=[a0,a1] of computing unit 0 corresponding to computing unit group A, and data processing request query1=[a2,a3], etc.). Obviously, if the starting positions in the divided results of different data processing requests processed by the same computing unit group are different, the data accumulation order will be different, which may lead to different processing results of different data processing requests.

[0051] 5 is a schematic diagram illustrating a process for processing data according to an exemplary embodiment of the present disclosure. As shown in FIG. 5, in this example, by applying the data division and computing power allocation method described in the above method 200, a data block in the data to be processed can be the smallest division unit. For example, when the size of the data block is 8 and the number of computing unit groups is 2, two first sub-blocks, [a0, a1, a2, a3] and [a4, a5, a6, a7], are obtained, and these two first sub-blocks are assigned to computing unit group A and computing unit group B, respectively, for processing. In this case, regardless of the number of data blocks included in the data processing request, both the computing power allocation scheme and the data processing logic corresponding to each data element can maintain a stable state.

[0052] In one example, the process of processing the data in FIG. 5 may be, for example, a process implemented based on the following steps.

[0053] In step S1, each data block is divided into a first number of first sub-blocks (ie, the number of calculation unit groups) in order, and each of the first sub-blocks is assigned to one calculation unit group for processing.

[0054] In step S2, the interior of each of the first sub-blocks is divided in order into a second number (ie, the number of calculation units in the calculation unit group) of second sub-blocks.

[0055] In step S3, all data blocks in each of the computing units are traversed (scanned), and a second sub-block is selected from the first sub-block corresponding to the computing unit group and sent to the subsequent computing unit in the computing unit group. As can be seen, the array numbers of the second sub-blocks selected by each of the computing units are different from each other.

[0056] By cyclically executing the above step S3, each computing unit in each computing unit group receives data from the preceding computing unit in the computing unit group in each round, performs an accumulative addition calculation with the data corresponding to the computing unit, and then transmits the accumulative addition result to the succeeding computing unit in the computing unit group. After the execution of the ring iteration step is completed, each computing unit in each computing unit group will have the data accumulative addition results of all computing units in the computing unit group.

[0057] According to some examples, the data to be processed is obtained using a feature coding network, and the multiple data blocks included in the data to be processed are data blocks obtained using a division operation to divide the data to be processed into multiple data blocks of the same size based on the output dimensionality of the feature coding network. In this way, the above method 200 can be applied to realize feature data processing, and by determining the data blocks in the data to be processed based on the dimensionality of the feature coding vector space, it is possible to more simply and efficiently obtain data blocks of a fixed size, balance the computation load of each computing unit group in a distributed system, and improve processing efficiency.

[0058] In some examples, the data to be processed may be data obtained by embedding natural language text data. By operating on the coded data of the text data, various types of natural language processing tasks can be realized, for example, by mapping the natural language text data into an embedding space to obtain a semantic feature tensor, and then performing operations such as semantic analysis and entity extraction based on the semantic feature tensor. As can be understood, the data length of the semantic feature tensor in the embedding space must be an integer multiple of the dimensionality of the embedding space. In this case, by dividing the data block of the data to be processed based on the dimensionality of the embedding space, the data to be processed can be simply and efficiently divided into an integer number of data blocks. Then, by performing data division and computing power allocation operations on the data blocks, the stability of the distributed processing logic of each data block and the stability of the computation results of the data to be processed can be ensured.

[0059] According to some examples, the step of dividing the data to be processed into a plurality of data blocks of the same size based on the output dimensionality of the feature coding network includes the step of dividing the data to be processed into a plurality of data blocks of the same size based on the output dimensionality of the feature coding network and the data format of the data to be processed. In this way, in combination with the data format, the amount of data in each data block can be accurately determined, and the computation load of each computation unit group in the distributed system can be balanced to improve processing efficiency.

[0060] According to an embodiment of the present disclosure, there is further provided an apparatus for processing data when applied to a distributed system. Figure 6 is a structural block diagram showing an apparatus 600 for processing data when applied to a distributed system according to an exemplary embodiment of the present disclosure, where multiple computing units in the distributed system can form multiple computing unit groups, and the data to be processed includes multiple data blocks. As shown in Figure 6, the apparatus 600 includes: a first dividing unit 601 configured, for each data block of the plurality of data blocks, to divide the data block into a plurality of first sub-blocks respectively corresponding to the plurality of computing unit groups; a second division unit 602 configured to divide each first sub-block of the plurality of first sub-blocks into a plurality of second sub-blocks, each second sub-block corresponding to a plurality of computing units in the target computing unit group, based on a target computing unit group corresponding to each first sub-block; a first processing unit 603 configured to process the second sub-blocks using a calculation unit corresponding to each of the second sub-blocks to determine a plurality of first processing results output by the plurality of calculation units respectively; and a second processing unit 604 configured to determine a processing result for each of the first sub-blocks by performing a data reduction operation on the plurality of first processing results using the target computing unit group.

[0061] According to some embodiments, the plurality of computing units in the target computing unit group form a ring topology, and the second processing unit 604 includes an accumulating subunit configured to sequentially accumulate and add the first processing results of each of the computing units based on the arrangement order of the plurality of computing units in the target computing unit group in the ring topology, and a determining subunit configured to determine the processing results of each of the first sub-blocks based on the accumulated addition results.

[0062] According to some embodiments, the accumulative addition subunit is configured to use each of the plurality of computation units to perform the following steps: receiving data to be accumulated from a preceding computation unit located before the computation unit in the ring topology; determining intermediate data by performing an accumulative addition operation based on the data to be accumulated and a first processing result of the computation unit; and transmitting the intermediate data to a subsequent computation unit located after the computation unit in the ring topology, so that the subsequent computation unit can perform an accumulative addition operation based on the intermediate data and the first processing result of the subsequent computation unit.

[0063] According to some examples, the data to be processed is data obtained using a feature coding network, and the multiple data blocks included in the data to be processed are data blocks obtained using a division operation that divides the data to be processed into multiple data blocks of the same size based on the output dimensionality of the feature coding network.

[0064] According to some examples, the multiple data blocks included in the data to be processed are data blocks obtained using a division operation that divides the data to be processed into multiple data blocks of the same size based on the output dimensionality of the feature coding network and the data format of the data to be processed.

[0065] According to another aspect of the present disclosure, there is further provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executable by the at least one processor to cause the at least one processor to perform the above-described method of processing data.

[0066] According to another aspect of the present disclosure, there is further provided a non-transitory computer-readable storage medium having stored thereon computer instructions, the computer instructions being used to cause the computer to perform the method for processing data described above.

[0067] According to another aspect of the present disclosure, there is further provided a computer program product including a computer program, the computer program implementing the above-described method for processing data when executed by a processor.

[0068] Next, referring to FIG. 7 , a structural block diagram of an electronic device 700 functioning as a server or client of the present disclosure will be described, which is one example of a hardware device applicable to each aspect of the present disclosure. The electronic device may represent various forms of digital electronic computers, such as laptop computers, desktop computers, stage computers, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing devices, mobile phones, intelligent phones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and do not limit the implementation of the present disclosure as described and / or claimed herein.

[0069] 7, the device 700 includes a computing unit 701, which can perform various appropriate operations and processes according to a computer program stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may further store various programs and data necessary for operating the device 700. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0070] Multiple components of device 700, such as an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709, are connected to an input / output (I / O) interface 705. The input unit 706 may be any type of device capable of inputting information into device 700. The input unit 706 can receive input numeric or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touchscreen, a trackboard, a trackball, a control lever, a microphone, and / or a remote control. The output unit 707 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include, but is not limited to, a magnetic disk, an optical disk, etc. The communication unit 709 enables the device 700 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunications networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0071] The computing unit 701 may be any of a variety of general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the methods and processes described above, e.g., data processing methods. For example, in some embodiments, the data processing methods may be embodied as computer software programs tangibly contained in a machine-readable medium, e.g., the storage unit 708. In some embodiments, some or all of the computer programs may be loaded and / or installed into the device 700 via the ROM 702 and / or the communication unit 709. When the computer programs are loaded into the RAM 703 and executed by the computing unit 701, they may perform one or more steps of the data processing methods described above. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data processing method in any other suitable manner (eg, by firmware).

[0072] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may receive data and instructions from, and send data and instructions to, a storage system, at least one input device, and at least one output device.

[0073] Program code implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, so that when executed by the processor or controller, the program code performs the functions / operations specified in the flowcharts and / or block diagrams. The program code may be entirely executed on a machine, partially executed on a machine, partially executed on a machine and partially executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0074] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include an electrical connection with one or more leads, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0075] To provide for user interaction, the systems and techniques described herein can be implemented on a computer that includes a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user may provide input to the computer. Other types of devices can be used to provide for user interaction, for example, where feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0076] The systems and techniques described herein may be implemented in a computing system including backstage components (e.g., as a data server), middleware components (e.g., as an application server), front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique implementation), or any combination of backstage components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include, for example, a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0077] The computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is created by running computer programs on the corresponding computers that have a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0078] It should be understood that the various forms of flow described above may be used to rearrange, add, or remove steps, and for example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the technical solutions disclosed in this disclosure can achieve the desired results, and the present disclosure is not limited thereto.

[0079] Although embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above-described methods, systems, and devices are merely illustrative embodiments or examples, and that the scope of the present invention is not limited by these embodiments or examples. Various elements of the embodiments or examples may be omitted or replaced by equivalent elements. Furthermore, each step may be performed in an order different from that described in this disclosure. Furthermore, various elements of the embodiments or examples may be combined in various ways. It is important to note that as technology evolves, many elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims

1. A method for processing data applied to a distributed system, wherein a plurality of computing units in the distributed system can constitute a plurality of computing unit groups, and data to be processed includes a plurality of data blocks, the method comprising: For each data block of the plurality of data blocks, Dividing the data block into a plurality of first sub-blocks corresponding to the plurality of computing unit groups, respectively; Dividing each first sub-block of the plurality of first sub-blocks into a plurality of second sub-blocks, each second sub-block corresponding to a plurality of computing units in the target computing unit group, based on a target computing unit group corresponding to each first sub-block; determining a plurality of first processing results output by the plurality of calculation units by processing the second sub-blocks using calculation units corresponding to the second sub-blocks; determining a processing result for each of the first sub-blocks by performing a data reduction operation on the plurality of first processing results using the target computing unit group; method.

2. The plurality of computing units in the target computing unit group form a ring topology, and the step of determining the processing results of each of the first sub-blocks by performing a data reduction operation on the plurality of first processing results using the target computing unit group includes: sequentially accumulating the first processing results of the plurality of computing units in the target computing unit group based on the arrangement order in the ring topology of the plurality of computing units; and determining a processing result for each of the first sub-blocks based on the cumulative sum.

3. The step of sequentially accumulating the first processing results of each of the plurality of computing units in the target computing unit group based on the arrangement order of the plurality of computing units in the ring topology includes: Using each of the plurality of computing units, receiving data to be accumulated from a preceding computing unit that precedes the computing unit in the ring topology; determining intermediate data by performing an accumulation operation based on the data to be accumulated and the first processing result of the calculation unit; and transmitting the intermediate data to a subsequent computing unit that is located after the computing unit in the ring topology, so that the subsequent computing unit can perform an accumulation addition operation based on the intermediate data and the first processing result of the subsequent computing unit.

4. 2. The method of claim 1, wherein the data to be processed is data obtained using a feature coding network, and the multiple data blocks included in the data to be processed are data blocks obtained using a division operation that divides the data to be processed into multiple data blocks of the same size based on the output dimensionality of the feature coding network.

5. 5. The method of claim 4, wherein dividing the data to be processed into a plurality of data blocks of the same size based on the output dimensionality of the feature coding network includes dividing the data to be processed into a plurality of data blocks of the same size based on the output dimensionality of the feature coding network and a data format of the data to be processed.

6. A data processing apparatus applied to a distributed system, wherein a plurality of calculation units in the distributed system can constitute a plurality of calculation unit groups, and data to be processed includes a plurality of data blocks, the apparatus comprising: For each data block of the plurality of data blocks, a first division unit configured to divide the data block into a plurality of first sub-blocks corresponding to the plurality of computing unit groups, respectively; a second division unit configured to divide, based on a target computing unit group corresponding to each of the first sub-blocks, each of the first sub-blocks into a plurality of second sub-blocks, each of the second sub-blocks corresponding to a plurality of computing units in the target computing unit group; a first processing unit configured to determine a plurality of first processing results output by the plurality of calculation units by processing the second sub-blocks using calculation units corresponding to each of the second sub-blocks; a second processing unit configured to determine a processing result for each of the first sub-blocks by performing a data reduction operation on the plurality of first processing results using the target computing unit group; Device.

7. The plurality of computing units in the target computing unit group form a ring topology, and the second processing unit an accumulating subunit configured to sequentially accumulate and add the first processing results of the plurality of computing units in the target computing unit group based on the arrangement order of the plurality of computing units in the ring topology; and a determining subunit configured to determine a processing result of each of the first sub-blocks based on the cumulative addition result.

8. The accumulating subunit uses each of the plurality of calculation units to receiving data to be accumulated from a preceding computing unit that precedes the computing unit in the ring topology; determining intermediate data by performing an accumulation operation based on the data to be accumulated and the first processing result of the calculation unit; transmitting the intermediate data to a subsequent computing unit that is located after the computing unit in the ring topology, so that the subsequent computing unit can perform an accumulation operation based on the intermediate data and the first processing result of the subsequent computing unit.

9. 7. The apparatus of claim 6, wherein the data to be processed is data obtained using a feature coding network, and the multiple data blocks included in the data to be processed are data blocks obtained using a division operation that divides the data to be processed into multiple data blocks of the same size based on the output dimensionality of the feature coding network.

10. 10. The apparatus of claim 9, wherein the plurality of data blocks included in the data to be processed are data blocks obtained using a division operation that divides the data to be processed into a plurality of data blocks of the same size based on the output dimensionality of the feature coding network and the data format of the data to be processed.

11. An electronic device, the electronic device comprising: at least one processor; a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform the method of any one of claims 1 to 5; electronic equipment.

12. A non-transitory computer readable storage medium having stored thereon computer instructions that cause a computer to perform the method of any one of claims 1 to 5.

13. A computer program which, when executed by a processor, causes the processor to implement the method of any one of claims 1 to 5.

Images