Fault processing method and device, equipment and medium

Abstract

The invention provides a fault processing method and device applied to a data processing device, equipment and a medium, and relates to the technical field of computers, in particular to the technical field of data processing, fault processing and chips. According to the implementation scheme, a plurality of unit groups are determined based on the grid topology of the data processing device; determining whether each data processing unit is a fault unit; for each unit group, in response to determining that the unit group does not comprise a fault unit, determining that the unit group is an effective group; in response to determining that the number of the effective groups meets a preset condition, configuration operation is sequentially executed for the unit groups, and the configuration operation comprises the steps that for each unit group, in response to determining that the unit group comprises fault units which are not set to be in the bypass state, the multiple data processing units in the unit group are set to be in the bypass state, and the fault units are not set to be in the bypass state. The data input port and the data output port of the data processing unit which is set to be in the bypass state are in short-circuit connection; and in response to determining that the number of the effective groups does not meet the preset condition, determining that the data processing device is a fault device.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of data processing, fault handling and chip technology, and specifically to a fault handling method, device, electronic device, computer-readable storage medium and computer program product applied to a data processing device. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] With the development of artificial intelligence technology, more and more applications based on artificial intelligence technology have achieved results far exceeding traditional algorithms. Deep learning is a data-intensive and computationally intensive algorithm. In order to improve the training and reasoning speed of large-scale deep learning models, it is necessary to make fuller use of the hardware resources of data processing devices and reduce computing costs.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a fault handling method, device, electronic device, computer-readable storage medium, and computer program product applied to a data processing device.

[0006] According to one aspect of the present disclosure, a fault handling method for a data processing device is provided, wherein the data processing device includes a plurality of data processing units forming a mesh topology, the mesh topology includes a plurality of rows and a plurality of columns, each of the plurality of data processing units includes a data input port, a data processing area, and a data output port connected in sequence, the method comprising: determining a plurality of unit groups based on the mesh topology, wherein the plurality of data processing units included in each unit group can form a row or a column in the mesh topology; determining whether each of the plurality of data processing units is a faulty unit; and determining whether each of the plurality of data processing units is a faulty unit for each of the plurality of data processing units. For each unit group in the unit groups, in response to determining that the unit group does not include the faulty unit, determining that the unit group is a valid group; in response to determining that the number of the valid groups meets the preset conditions, sequentially performing configuration operations on the multiple unit groups, the configuration operations comprising: for each unit group, in response to determining that the unit group includes a faulty unit that has not been set to the bypass state, setting multiple data processing units in the unit group to the bypass state, wherein the data input port and the data output port of the data processing unit set to the bypass state are short-circuited; and in response to determining that the number of the valid groups does not meet the preset conditions, determining that the data processing device is a faulty device.

[0007] According to one aspect of the present disclosure, there is provided a fault handling device applied to a data processing device, wherein the data processing device comprises a plurality of data processing units constituting a mesh topology, wherein the mesh topology comprises a plurality of rows and a plurality of columns, wherein each of the plurality of data processing units comprises a data input port, a data processing area and a data output port connected in sequence, and wherein the device comprises: a first determining unit configured to determine a plurality of unit groups based on the mesh topology, wherein the plurality of data processing units included in each unit group can constitute a row or a column in the mesh topology; a second determining unit configured to determine whether each of the plurality of data processing units is a faulty unit; and a third determining unit configured to determine whether each of the plurality of data processing units is a faulty unit. For each of the multiple unit groups, in response to determining that the unit group does not include the faulty unit, the unit group is determined to be a valid group; a first configuration unit is configured to, in response to determining that the number of the valid groups meets the preset conditions, sequentially perform configuration operations on the multiple unit groups, the configuration operations comprising: for each unit group, in response to determining that the unit group includes a faulty unit that has not been set to the bypass state, setting multiple data processing units in the unit group to the bypass state, wherein the data input port and the data output port of the data processing unit set to the bypass state are short-circuited; and a fourth determination unit is configured to, in response to determining that the number of the valid groups does not meet the preset conditions, determine that the data processing device is a faulty device.

[0008] According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned fault handling method.

[0009] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned fault handling method.

[0010] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program can implement the above fault handling method when executed by a processor.

[0011] According to one or more embodiments of the present disclosure, hardware resources of a data processing device may be more fully utilized.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A flowchart of a fault handling method applied to a data processing device according to an exemplary embodiment of the present disclosure is shown;

[0015] Figure 2 A schematic diagram showing the structure of a data processing system according to an exemplary embodiment of the present disclosure is shown;

[0016] Figure 3 A schematic diagram showing the structure of a data processing device according to an exemplary embodiment of the present disclosure is shown;

[0017] Figure 4 A schematic diagram showing the structure of a data processing unit according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 5 A schematic diagram showing the structure of a data processing area according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 6A structural block diagram of a fault processing device applied to a data processing device according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0022] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0023] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0024] Generally speaking, a large number of memories and computing units are integrated in the chip used to implement the data processing function. In order to improve the computing power of a single chip, a large number of identical storage units and computing units are integrated in it. In the related art, a complete chip (i.e., a data processing device) used to implement the data processing function is usually tested, and when a fault is found therein, the data processing device is determined to be a faulty device. This detection method has a coarse granularity and fails to fully utilize the non-faulty areas in the data processing device.

[0025] Based on this, the present disclosure provides a fault handling method applied to a data processing device, wherein the data processing device includes a plurality of computing units constituting a grid topology, and the validity of each row and each column in the grid topology is determined respectively. When a computing unit with a fault exists in a row or a column, the computing unit in the row or the column is bypassed to obtain a fault-free data processing device that meets preset conditions. By applying this method, the computing power of the data processing device can be guaranteed under the premise of isolating the faulty data processing unit, and the non-faulty area in the data processing device can be fully utilized to avoid the waste of hardware resources.

[0026] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 A flowchart of a fault handling method 100 applied to a data processing device according to an exemplary embodiment of the present disclosure is shown, wherein the data processing device includes a plurality of data processing units forming a mesh topology, wherein the mesh topology includes a plurality of rows and a plurality of columns, and each of the plurality of data processing units includes a data input port, a data processing area, and a data output port connected in sequence. Figure 1 As shown, the method 100 includes:

[0028] Step S101, determining a plurality of unit groups based on the mesh topology, wherein a plurality of data processing units included in each unit group can constitute a row or a column in the mesh topology;

[0029] Step S102: determining whether each of the plurality of data processing units is a faulty unit;

[0030] Step S103: for each unit group in the plurality of unit groups, in response to determining that the unit group does not include the faulty unit, determining the unit group as a valid group;

[0031] Step S104: in response to determining that the number of the valid groups meets the preset condition, sequentially performing configuration operations for the multiple unit groups, the configuration operations comprising: for each unit group, in response to determining that the unit group includes a faulty unit that is not set to the bypass state, setting multiple data processing units in the unit group to the bypass state, wherein the data input port and the data output port of the data processing unit set to the bypass state are short-circuited; and

[0032] Step S105: In response to determining that the number of the valid groups does not meet the preset condition, determining that the data processing device is a faulty device.

[0033] By applying the above-mentioned fault handling method 100, it is possible to divide unit groups based on the grid topology formed by multiple data processing units of the data processing device, and determine the effectiveness of each unit group, so that the faulty data processing units can be isolated in units of unit groups to retain qualified fault-free data processing areas, while fully utilizing the non-faulty areas and ensuring the processing performance of the data processing device after isolating the fault, thereby avoiding waste of hardware resources.

[0034] In some examples, the data processing device includes M×N computing units of a grid topology, based on which (M+N) unit groups corresponding to the M rows and N columns in the grid topology can be determined. When there is a faulty data processing unit in a certain unit group, all the data processing units in the unit group are bypassed to obtain a fault-free device that meets the preset conditions. In one example, a preset condition may be configured to indicate that the difference between the number of valid groups and the number of all unit groups shall not be greater than a preset threshold. For example, when the preset threshold is 1, it can be ensured that the data processing device after isolating the fault can at least form a (M-1)×N grid topology or an M×(N-1) grid topology. As a result, after completing the fault handling, the grid topology can continue to support parallel data processing normally, and at the same time, the non-fault area in the data processing device can be fully utilized to retain the available computing power within the preset range, thereby avoiding the waste of hardware resources.

[0035] In some examples, in step S104, the order of performing configuration operations for multiple unit groups may be set in advance according to demand. It is understandable that each faulty unit in the grid topology must correspond to two unit groups. By performing configuration operations for unit groups in sequence according to a preset order, redundant isolation operations can be avoided to retain as much computing power as possible. In one example, the order may be preset to first perform configuration operations on unit groups with a small number of data processing units. For example, when a certain faulty unit corresponds to a unit group A containing 5 data processing units and a unit group B containing 3 data processing units, a bypass configuration operation may be performed on unit group B first to set the faulty unit to bypass, thereby eliminating the need to perform bypass configuration operations on unit group A to retain more computing power in unit group A.

[0036] In some examples, the data processing device may include a graphics processing unit (GPU), a central processing unit (CPU), or various types of computing chips, computing arrays and other logic devices, such as a field programmable gate array (FPGA), which is not limited in the present disclosure.

[0037] According to some embodiments, the data processing unit further includes a path selector, the data output port includes a first sub-output port and a second sub-output port, wherein the first sub-output port can be connected to the data processing area through the first path of the path selector, and the second sub-output port can be short-circuited to the data input port through the second path of the path selector, and wherein, in step S104, setting the multiple data processing units in the unit group to a bypass state includes: configuring the state of the path selectors of the multiple data processing units in the unit group to select the second path. Thus, the path selector can be used to configure the first path that can support normal data processing functions and the second path for realizing the bypass function of fault isolation for the data processing unit, and the path selector can be used to more conveniently configure the bypass state of the data processing unit.

[0038] In some examples, the bypass state of the data processing unit may be configured in other ways, for example, a short-circuit branch including a controlled switch device may be provided between the data input port and the data output port of the data processing unit. The on-off of the short-circuit branch may be controlled by configuring the control signal of the controlled switch device to achieve a short-circuit connection between the data input port and the data output port.

[0039] According to some embodiments, the method 100 further includes: storing the state information of the path selectors of the plurality of data processing units in the data processing device into a configuration table; and in response to receiving an initialization instruction for the data processing device, configuring the states of the path selectors of the plurality of data processing units based on the configuration table. Thus, after completing the fault handling for the data processing device, the state of each path selector can be stored using the configuration table, so that each path selector can be directly configured based on the configuration table in the process of using the device to implement the data processing function, and a usable data processing device can be obtained conveniently and efficiently.

[0040] In some examples, the data processing area in the data processing unit may be further divided to obtain a plurality of sub-units.

[0041] Based on this, in some embodiments, each of the multiple data processing units includes multiple initial sub-units and at least one backup sub-unit, and the backup sub-unit is not connected to the multiple initial sub-units, and wherein, whether each of the multiple data processing units is a faulty unit is determined by the following process: detecting whether each of the multiple initial sub-units and each of the at least one backup sub-unit is a faulty sub-unit; in response to determining that the number of faulty sub-units in the multiple initial sub-units is not more than the number of non-faulty sub-units in the at least one backup sub-unit, for each of the multiple initial sub-units, in response to determining that the initial sub-unit is a faulty sub-unit, replacing the faulty sub-unit with a non-faulty sub-unit in the at least one backup sub-unit; and in response to determining that the number of faulty sub-units in the multiple initial sub-units is more than the number of non-faulty sub-units in the at least one backup sub-unit, determining that the data processing unit is a faulty unit. Therefore, by configuring a first number of spare sub-units for a data processing unit including multiple initial sub-units, when an initial sub-unit fails and the number of failed initial sub-units is less than the available spare sub-units, the spare sub-units can be directly used to replace the failed sub-units, thereby improving the robustness of the data processing unit through redundant backup and further improving the availability of the data processing device.

[0042] In some examples, each initial subunit in the data processing unit includes a subunit input port, a subprocessing area, and a subunit output port connected in sequence, and multiple initial subunits can be interconnected through the connection between the subunit input port and the subunit output port. It can be understood that in the initial state where the fault processing is not performed, the spare subunit is not interconnected with any initial subunit. By performing the above-mentioned fault processing steps, the faulty subunit can be replaced by the spare subunit by reconfiguring the connection relationship between the input port of the initial subunit, the subunit input port, the subunit output port, and the spare subunit.

[0043] In some examples, each initial subunit may include a first subunit input port, a second subunit input port, a first subunit output port, and a second subunit output port, and a path selector may be used to configure a third path sequentially connecting the first subunit input port, the sub-processing area, and the first subunit output port, and a fourth path sequentially connecting the second subunit input port, the spare subunit, and the second subunit output port. Thus, by configuring the state of the path selector, it is possible to conveniently and efficiently replace the faulty subunit with the spare subunit.

[0044] In some examples, the status information of the path selectors corresponding to the multiple initial sub-units in the data processing unit may be stored in a configuration table, so that the connection relationship between the initial sub-unit and the backup sub-unit can be directly configured based on the configuration table, thereby obtaining an available data processing unit conveniently and efficiently.

[0045] In some examples, a connection branch including a controlled switch device may be provided between the subunit input port and the subunit output port of the initial subunit and the spare subunit. The connection branch may be controlled by configuring a control signal of the controlled switch device to implement the above replacement step.

[0046] According to some embodiments, the detecting whether each of the multiple initial subunits and each of the at least one backup subunit is a faulty subunit comprises: determining input data and reference output data corresponding to the input data; for each of the multiple initial subunits, using the initial subunit to perform data processing based on the input data to obtain test output data; and determining whether the initial subunit is a faulty subunit based on the test output data and the reference output data. Thus, the subunit can be used to perform data processing tasks, and by comparing the reference output data with the test output data, it is simple and efficient to determine whether the subunit is faulty.

[0047] In some examples, the input data may be passed into a data processing model established in a software environment that corresponds to the hardware function of the data processing device. The data processing model may be a mathematical algorithm model or a chip circuit model that can be used for simulation. The model is used to simulate the function of the data processing device to obtain reference output data corresponding to the input data as a reference value for the test output data to be verified.

[0048] According to some embodiments, in response to determining that the number of valid groups does not meet the preset condition in step S105, determining that the data processing device is a faulty device includes: determining a first proportion of valid units that are not placed in a bypass state among the multiple data processing units based on the number of valid groups; and in response to determining that the first proportion is less than a preset proportion, determining that the data processing device is a faulty device. Thus, the proportion of available data processing units to all data processing units can be limited by configuring preset conditions to ensure that the computing power of the data processing device after fault processing is not lower than a preset threshold, and to ensure the normal execution of data processing tasks.

[0049] Figure 2 FIG. 2 shows a schematic diagram of the structure of a data processing system 200 according to an exemplary embodiment of the present disclosure. Figure 2As shown, the data processing system 200 can be regarded as an integrated data processing chip, which includes multiple identical data processing devices 300, a storage unit 210 and a communication unit 220 shared by the multiple data processing devices 300. The multiple data processing devices 300 can be, for example, multiple identical computing cores.

[0050] In some examples, when testing the integrated data processing chip, the above-mentioned fault handling method 100 may be applied to each data processing device 300 respectively to retain as many available data processing devices 300 as possible. When the fault handling process is completed, the data processing chip whose number of available data processing devices 300 meets the preset conditions can be used as an available chip to improve the chip yield.

[0051] Figure 3 FIG. 3 shows a schematic diagram of the structure of a data processing device 300 according to an exemplary embodiment of the present disclosure. Figure 3 As shown, the data processing device includes 16 data processing units 400 that can form a 4×4 grid topology. By applying the above-mentioned step S101, 8 unit groups corresponding to 4 rows and 4 columns can be obtained. When there is a faulty data processing unit in a unit group, all the data processing units in the unit group are bypassed to obtain a fault-free device that meets the preset conditions. In one example, the preset condition may be that the number of valid groups is not less than 7, that is, the computing power of the retained data processing unit is not less than 75% of the initial value. When there is a faulty data processing unit, by executing the fault handling method 100, a fault-free device that can form a 3×4 grid topology or a 4×3 grid topology can be obtained.

[0052] Figure 4 FIG. 4 shows a schematic diagram of the structure of a data processing unit 400 according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the data processing unit 400 includes a data input port 410, a data processing region 420 and a data output port 430 connected in sequence, wherein the data output port 430 includes a first sub-output port 431 and a second sub-output port 432. By configuring a path selector for the data processing unit 400, the path selector can be used to configure the data processing unit with a first path that can support a normal data processing function and a second path for realizing a bypass function for fault isolation.

[0053] Figure 5The schematic diagram of the structure of the data processing area 420 according to the exemplary embodiment of the present disclosure is shown. As described above, by further dividing the data processing area, multiple initial sub-units 421 can be obtained. By applying the method described in the present disclosure, a certain number of spare sub-units 422 can be further configured for the data processing area, and by using the spare sub-units 422 to replace the failed initial sub-units 421, the robustness of the data processing unit can be improved through redundant backup.

[0054] In some examples, after completing the fault handling process for the data processing system 200, the configuration table is used to store information about the available data processing device 300, status information about the path selector of each data processing unit 400, and status information about the path selector of each initial subunit 421, that is, the architecture of the data processing system after isolating the fault is stored using the configuration table, so as to obtain an available data processing system conveniently and efficiently based on this.

[0055] According to one aspect of the present disclosure, a fault handling device applied to a data processing device is also provided. Figure 6 The structure block diagram of a fault processing device 600 applied to a data processing device according to an exemplary embodiment of the present disclosure is shown, wherein the data processing device includes a plurality of data processing units forming a mesh topology, wherein the mesh topology includes a plurality of rows and a plurality of columns, and each of the plurality of data processing units includes a data input port, a data processing area, and a data output port connected in sequence. Figure 6 As shown, the device 600 includes:

[0056] A first determining unit 601 is configured to determine a plurality of unit groups based on the mesh topology, wherein a plurality of data processing units included in each unit group can constitute a row or a column in the mesh topology;

[0057] A second determining unit 602 is configured to determine whether each of the plurality of data processing units is a faulty unit;

[0058] A third determining unit 603 is configured to, for each cell group in the plurality of cell groups, determine that the cell group is a valid group in response to determining that the cell group does not include the faulty cell;

[0059] The first configuration unit 604 is configured to, in response to determining that the number of valid groups meets a preset condition, sequentially perform configuration operations on the multiple unit groups, the configuration operations comprising: for each unit group, in response to determining that the unit group includes a faulty unit that is not set to a bypass state, setting multiple data processing units in the unit group to a bypass state, wherein the data input port and the data output port of the data processing unit set to the bypass state are short-circuited; and

[0060] The fourth determining unit 605 is configured to determine that the data processing device is a faulty device in response to determining that the number of the valid groups does not meet the preset condition.

[0061] According to some embodiments, the data processing unit also includes a path selector, and the data output port includes a first sub-output port and a second sub-output port, wherein the first sub-output port can be connected to the data processing area through a first path of the path selector, and the second sub-output port can be short-circuited to the data input port through a second path of the path selector, and wherein the first configuration unit 604 is configured to: configure the state of the path selectors of multiple data processing units in the unit group to select the second path.

[0062] According to some embodiments, the apparatus 600 further comprises: a storage unit configured to store state information of the path selectors of the plurality of data processing units in the data processing apparatus into a configuration table; and

[0063] The second configuration unit is configured to configure the states of the path selectors of the plurality of data processing units based on the configuration table in response to receiving an initialization instruction for the data processing device.

[0064] According to some embodiments, each of the multiple data processing units includes multiple initial sub-units and at least one backup sub-unit, and the backup sub-unit is not connected to the multiple initial sub-units, and wherein the second determination unit 602 includes: a detection sub-unit, configured to detect whether each of the multiple initial sub-units and each of the at least one backup sub-unit is a faulty sub-unit; a replacement sub-unit, configured to, in response to determining that the faulty sub-units in the multiple initial sub-units are not more than the non-faulty sub-units in the at least one backup sub-unit, replace the faulty sub-unit with a non-faulty sub-unit in the at least one backup sub-unit for each of the multiple initial sub-units in response to determining that the initial sub-unit is a faulty sub-unit; and a determination sub-unit, configured to determine that the data processing unit is a faulty unit in response to determining that the faulty sub-units in the multiple initial sub-units are more than the non-faulty sub-units in the at least one backup sub-unit.

[0065] According to some embodiments, the detection subunit is configured to: determine input data and reference output data corresponding to the input data; for each initial subunit among the multiple initial subunits, use the initial subunit to perform data processing based on the input data to obtain test output data; and determine whether the initial subunit is a faulty subunit based on the test output data and the reference output data.

[0066] According to some embodiments, the fourth determination unit 605 is configured to: determine a first proportion of valid units in the multiple data processing units that are not placed in a bypass state based on the number of valid groups; and in response to determining that the first proportion is less than a preset proportion, determine that the data processing device is a faulty device.

[0067] It should be understood that Figure 6 The operations of the various units of the fault detection device 600 shown in FIG. Figure 1 The steps in the fault detection method 100 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 100 are also applicable to the device 600 and the various units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0068] According to one aspect of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned fault detection method 100.

[0069] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned fault detection method 100.

[0070] According to one aspect of the present disclosure, a computer program product is further provided, including a computer program, wherein the computer program implements the above-mentioned fault detection method 100 when executed by a processor.

[0071] refer to Figure 7, a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0072] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0073] Multiple components in the device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the device 700. The input unit 706 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 707 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include but is not limited to a disk, an optical disk. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0074] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the fault detection method 100. For example, in some embodiments, the fault detection method 100 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the fault detection method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the fault detection method 100 in any other appropriate manner (e.g., by means of firmware).

[0075] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0076] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0077] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0078] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0079] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0080] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0081] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0082] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples. Various elements in the embodiments or examples may be omitted or may be replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A fault handling method applied to a data processing device, the data processing device comprising a plurality of data processing units forming a mesh topology, the mesh topology comprising a plurality of rows and a plurality of columns, each of the plurality of data processing units comprising a data input port, a data processing area and a data output port connected in sequence, the method comprising: Determine a plurality of unit groups based on the mesh topology, wherein a plurality of data processing units included in each unit group can constitute a row or a column in the mesh topology; determining whether each of the plurality of data processing units is a faulty unit; For each of the plurality of cell groups, in response to determining that the cell group does not include the faulty cell, determining the cell group to be a valid group; In response to determining that the number of the valid groups meets the preset condition, sequentially performing configuration operations for the multiple unit groups, the configuration operations comprising: for each unit group, in response to determining that the unit group includes a faulty unit that is not set to a bypass state, setting multiple data processing units in the unit group to a bypass state, wherein the data input port and the data output port of the data processing unit set to the bypass state are short-circuited; and In response to determining that the number of the valid groups does not meet the preset condition, the data processing device is determined to be a faulty device.

2. The method of claim 1, wherein: The data processing unit further includes a path selector, the data output port includes a first sub-output port and a second sub-output port, wherein the first sub-output port can be connected to the data processing area through a first path of the path selector, and the second sub-output port can be short-circuited to the data input port through a second path of the path selector, and wherein placing the plurality of data processing units in the unit group into a bypass state includes: The states of the path selectors of the plurality of data processing units in the unit group are configured to select the second path.

3. The method of claim 2, further comprising: storing the state information of the path selectors of the plurality of data processing units in the data processing device into a configuration table; as well as In response to receiving an initialization instruction for the data processing device, the states of the path selectors of the plurality of data processing units are configured based on the configuration table.

4. The method according to any one of claims 1 to 3, wherein: Each of the plurality of data processing units comprises a plurality of initial sub-units and at least one spare sub-unit, the spare sub-unit being not connected to the plurality of initial sub-units, and wherein whether each of the plurality of data processing units is a faulty unit is determined by the following process: Detecting whether each of the multiple initial subunits and each of the at least one backup subunit is a faulty subunit; In response to determining that the number of failed subunits in the plurality of initial subunits is not more than the number of non-failed subunits in the at least one spare subunit, for each initial subunit in the plurality of initial subunits, in response to determining that the initial subunit is a failed subunit, replacing the failed subunit with a non-failed subunit in the at least one spare subunit; and In response to determining that there are more failed sub-units in the plurality of initial sub-units than non-failed sub-units in the at least one spare sub-unit, the data processing unit is determined to be a failed unit.

5. The method of claim 4, wherein: The detecting whether each of the multiple initial subunits and each of the at least one spare subunit is a faulty subunit comprises: determining input data and reference output data corresponding to the input data; For each initial subunit of the plurality of initial subunits, Using the initial subunit to perform data processing based on the input data to obtain test output data; and Based on the test output data and the reference output data, it is determined whether the initial sub-unit is a faulty sub-unit.

6. The method according to any one of claims 1 to 5, wherein: In response to determining that the number of the valid groups does not meet the preset condition, determining that the data processing device is a faulty device includes: Determining a first proportion of valid units in the plurality of data processing units that are not placed in a bypass state based on the number of valid groups; and In response to determining that the first ratio is less than a preset ratio, the data processing device is determined to be a faulty device.

7. A fault handling device applied to a data processing device, the data processing device comprising a plurality of data processing units forming a mesh topology, the mesh topology comprising a plurality of rows and a plurality of columns, each of the plurality of data processing units comprising a data input port, a data processing area and a data output port connected in sequence, the device comprising: A first determining unit is configured to determine a plurality of unit groups based on the mesh topology, wherein a plurality of data processing units included in each unit group can constitute a row or a column in the mesh topology; a second determining unit, configured to determine whether each of the plurality of data processing units is a faulty unit; a third determining unit configured to, for each of the plurality of cell groups, determine that the cell group is a valid group in response to determining that the cell group does not include the faulty cell; a first configuration unit, configured to, in response to determining that the number of the valid groups meets a preset condition, sequentially perform configuration operations on the plurality of unit groups, the configuration operations comprising: for each unit group, in response to determining that the unit group includes a faulty unit that is not set to a bypass state, setting a plurality of data processing units in the unit group to a bypass state, wherein a data input port and a data output port of the data processing unit set to the bypass state are short-circuited; and The fourth determining unit is configured to determine that the data processing device is a faulty device in response to determining that the number of the valid groups does not meet the preset condition.

8. The device according to claim 7, wherein: The data processing unit further includes a path selector, the data output port includes a first sub-output port and a second sub-output port, wherein the first sub-output port can be connected to the data processing region through a first path of the path selector, and the second sub-output port can be short-circuited to the data input port through a second path of the path selector, and wherein the first configuration unit is configured as: The states of the path selectors of the plurality of data processing units in the unit group are configured to select the second path.

9. The apparatus of claim 8, further comprising: A storage unit configured to store the state information of the path selectors of the plurality of data processing units in the data processing device into a configuration table; as well as The second configuration unit is configured to configure the states of the path selectors of the plurality of data processing units based on the configuration table in response to receiving an initialization instruction for the data processing device.

10. The device according to any one of claims 7 to 9, wherein: Each of the plurality of data processing units includes a plurality of initial subunits and at least one spare subunit, the spare subunit being not connected to the plurality of initial subunits, and wherein the second determining unit includes: a detection subunit, configured to detect whether each of the plurality of initial subunits and each of the at least one backup subunit is a faulty subunit; a replacement subunit, configured to, in response to determining that the number of failed subunits in the plurality of initial subunits is not more than the number of non-failed subunits in the at least one spare subunit, replace each of the plurality of initial subunits with a non-failed subunit in the at least one spare subunit in response to determining that the initial subunit is a failed subunit; and The determining subunit is configured to determine that the data processing unit is a failed unit in response to determining that the failed subunits in the plurality of initial subunits are more than the non-failed subunits in the at least one spare subunit.

11. The device of claim 10, wherein: The detection subunit is configured as follows: determining input data and reference output data corresponding to the input data; For each initial subunit of the plurality of initial subunits, Using the initial subunit to perform data processing based on the input data to obtain test output data; as well as Based on the test output data and the reference output data, it is determined whether the initial sub-unit is a faulty sub-unit.

12. The device according to any one of claims 7 to 11, wherein: The fourth determining unit is configured to: Determining, based on the number of the valid groups, a first proportion of valid units in the plurality of data processing units that are not placed in a bypass state; as well as In response to determining that the first ratio is less than a preset ratio, the data processing device is determined to be a faulty device.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-6.

15. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 6 when executed by a processor.

Images