Debug trace stream for nuclear synchronization
By monitoring and synchronizing the debug tracking data flow of the multi-core system, identifying asynchronous cores and taking corresponding measures, the synchronization problem of the multiprocessor system under SEU conditions was solved, the system's processing flexibility and reliability were improved, and system downtime was reduced.
Patent Information
- Application Number
- CN202111462434.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-12-02
- Filing Date
- 2021-12-02
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-12-02
AI Technical Summary
When faced with a single event flip (SEU), the reset process in existing multiprocessor systems can reduce system availability and increase complexity and size, especially in environments such as spacecraft, where existing technologies struggle to efficiently synchronize and handle desynchronization issues between cores.
By monitoring the debug trace data stream of multiple cores, out-of-sync cores can be identified and actions taken, such as stopping, resetting, or reinitializing the cores. Reduced set operations are used until the system is in a safe environment to perform a core reset, reducing the number of cores that need to be restarted and improving recovery speed.
It improves the system's resilience and reliability under SEU conditions, reduces system downtime, and ensures the continuity and security of critical tasks.
Smart Images

Figure CN114578896B_ABST
Abstract
Description
Technical Field
[0001] This disclosure generally relates to computing systems that use multiple processing cores in parallel. More specifically, this disclosure relates to detecting and handling desynchronization between multiple cores via debugging tools that include a tracing unit, to verify that the cores are in sync with each other, and to taking remedial measures when cores are out of sync. Background Technology
[0002] Various systems use multiple processors to process instructions in parallel with each other, providing additional determinism and resilience to the system. For example, spacecraft, aircraft, and other equipment used in ionized environments are subject to environmental conditions that can lead to single-event flips (SEUs). These devices typically use triple-mode redundancy (TMR) in their computing systems so that three processors can perform the same task in parallel. Systems using TMR use majority decision (i.e., where at least two of the three processors produce the same result) as the output. In cases where complete agreement is not reached (i.e., where at least one of the three processors produces a unique result), systems using TMR resynchronize the inconsistent processors or reset all three processors to ensure that the next computation is performed using the full set of processors. The reset process can reduce system availability at critical moments, and the overhead of the TMR voting logic, which includes multiple processors and the entire processing system, increases the complexity and size of the system. Summary of the Invention
[0003] This disclosure provides a method in one aspect, the method comprising: monitoring a plurality of debug trace data streams of a redundant operating system, the redundant operating system including a plurality of corresponding cores performing tasks in parallel; in response to detecting a state difference of one debug trace data stream relative to the other debug trace data streams in the plurality of debug trace data streams: marking a given core associated with said one debug trace data stream as an affected core; and restarting the affected core.
[0004] In one aspect, in conjunction with any of the above or below example methods, the corresponding plurality of cores consists of two active cores, including an affected core and a second core, and the method further includes, in response to detecting a state difference: marking the second core as a second affected core; and resetting the second core simultaneously with the affected core.
[0005] In one aspect, in conjunction with any of the above or below example methods, the corresponding plurality of cores consists of three active cores, including an affected core, a first unaffected core, and a second unaffected core. The method further includes, in response to detecting a state difference: stopping the affected core; and monitoring the first and second unaffected cores that are executing tasks in parallel in a consistent redundancy mode; and wherein restarting the affected core includes, in response to detecting a safe restart time: loading the affected core with register data from the first unaffected core; and reinitializing the affected core.
[0006] In one aspect, in conjunction with any of the above or below example methods, the corresponding plurality of cores includes more than three active cores, including an affected core, a first unaffected core, a second unaffected core, and a third unaffected core; in response to detecting a state difference: stopping the affected core; and monitoring the first unaffected core, the second unaffected core, and the third unaffected core that are performing tasks in parallel in majority mode; and wherein restarting the affected core includes, in response to detecting a safe restart time: reloading the affected core with register data from the first unaffected core; and reinitializing the affected core.
[0007] In one aspect, in combination with any of the above or below example methods, the corresponding multiple cores are included on a shared on-chip system with a monitor, wherein the monitor is one of the following: a radiation-hardened processor; a hardware-defined state machine; and a maintenance processor.
[0008] In one aspect, in conjunction with any of the above or below example methods, at least one of the corresponding plurality of cores is included on an integrated circuit separate from the other cores of the corresponding plurality of cores.
[0009] This disclosure provides, in one aspect, a system comprising: a plurality of cores configured to perform corresponding instances of a task in parallel; and a monitor configured to: receive debug trace data streams from the plurality of cores; and identify the affected core among the plurality of cores based on state differences between debug trace data streams associated with the affected core and other cores among the plurality of cores.
[0010] In one aspect, combining any of the above or below example systems, multiple cores and monitors are arranged on a single integrated circuit.
[0011] In one aspect, in conjunction with any of the above or below example systems, a monitor is disposed on a first integrated circuit, and at least one of a plurality of cores is disposed on a second integrated circuit.
[0012] In one aspect, in conjunction with any of the example systems above or below, the monitor is also configured to: remove the affected core from among the multiple cores performing the task until a safe restart time is reached; and restart the affected core in response to the safe restart time being reached.
[0013] In one aspect, in conjunction with any of the example systems above or below, when multiple cores, including at least two cores, are removed after the affected core is removed, the monitor is also configured to: reload the affected registers associated with the affected core using data from unaffected registers associated with one of the unaffected cores; reinitialize the affected core; and return the affected core to the multiple cores.
[0014] In one respect, in conjunction with any of the above or below example systems, the safe restart time is determined based on the safety profile of the device controlled by the system.
[0015] In one aspect, in conjunction with any of the above or below example systems, when multiple cores consist of two cores, the monitor is also configured to reset the second core simultaneously with the affected core.
[0016] In one respect, in conjunction with any of the above or below example systems, debug trace data streams are received via at least one of the following: instruction trace; data trace; and system trace.
[0017] In one aspect, this disclosure provides a computer-readable storage device including instructions that, when executed by a processor, perform operations including: monitoring a plurality of debug trace data streams of a redundant operating system, the redundant operating system including a plurality of corresponding cores performing tasks in parallel; in response to detecting a state difference of one debug trace data stream relative to the other debug trace data streams in the plurality of debug trace data streams: marking the core associated with said one debug trace data stream as an affected core; and restarting the affected core.
[0018] In one aspect, in conjunction with any of the above or below examples of computer-readable storage devices, the corresponding plurality of cores consist of two active cores, including an affected core and a second core, the operation further comprising, in response to detecting a state difference: marking the second core as a second affected core; and resetting the second core simultaneously with the affected core.
[0019] In one aspect, in conjunction with any of the above or below examples of computer-readable storage devices, the corresponding plurality of cores comprises three active cores, including an affected core, a first unaffected core, and a second unaffected core, the operation further comprising, in response to detecting a state difference: stopping the affected core; and monitoring the first unaffected core and the second unaffected core performing tasks in parallel in a consistent redundancy mode; and wherein restarting the affected core comprises, in response to detecting a safe restart time: loading the affected core with register data from the first unaffected core; and reinitializing the affected core.
[0020] In one aspect, in conjunction with any of the above or below examples of computer-readable storage devices, the corresponding plurality of cores includes more than three active cores, including an affected core, a first unaffected core, a second unaffected core, and a third unaffected core, the operation further including, in response to detecting a state difference: stopping the affected core; and monitoring the first unaffected core, the second unaffected core, and the third unaffected core performing tasks in parallel in majority mode; and wherein restarting the affected core includes, in response to detecting a safe restart time: loading the affected core with register data from the first unaffected core; and reinitializing the affected core.
[0021] In one aspect, in conjunction with any of the above or below examples of computer-readable storage devices, corresponding multiple cores are included on a shared on-chip system having a monitor configured to monitor multiple debug trace data streams.
[0022] In one aspect, in conjunction with any of the above or below examples of computer-readable storage devices, at least one of the corresponding plurality of cores is included on an integrated circuit separate from the other cores of the corresponding plurality of cores. Attached Figure Description
[0023] To gain a more detailed understanding of the features described above, please refer to the examples provided in the accompanying drawings for a more specific description of the features.
[0024] Figure 1 The satellite is shown in several aspects of this disclosure as an example of a device including a multi-core control system.
[0025] Figure 2A-2C Operational diagrams of a multi-core control system are shown according to various aspects of this disclosure.
[0026] Figure 3 This is a flowchart of a method for kernel synchronization using a debug trace stream, based on various aspects of this disclosure.
[0027] Figure 4Computing devices, which can be used, for example, as multi-core control systems, are shown according to various aspects of this disclosure. Detailed Implementation
[0028] This disclosure provides improved core synchronization in multi-core processor systems. It utilizes real-time (or near-real-time) information from each processing core (typically multiple cores) monitored from debug traces within each core. These debug trace streams are typically transmitted off-chip for external debugger verification, but herein they are used to compare the output of one core with the output of other cores (whether on-chip or off-chip) to detect whether these cores are synchronized with each other. In various embodiments, when core desynchronization is detected, the system continues to utilize a reduced set of synchronized cores (e.g., initially using n cores, then n-1 cores, and possibly adjusting redundancy modes) until the system is in a safe environment or state to perform a core reset. Additionally or alternatively, state information from the locked-step cores is used to re-lockstep the desynchronized cores with other cores. Therefore, this disclosure reduces the number of cores restarted and improves the speed at which non-lockstepped cores return to online operation.
[0029] According to various aspects of this disclosure, Figure 1 Satellite 110 is shown as an example of a redundant operating system 100 including a multi-core control system 130. The multi-core control system 130 uses several processors or processing cores (typically multiple cores) to execute several instances of computation or tasks in parallel with each other, so that consistency between active cores is used to control system 100, rather than the output of a single core. When a core is not locked in time, for example, when it produces inconsistent values for output, the core(s) producing outputs different from the majority are referred to as out of time / out of sync or affected, and can continue to produce different values when the affected core is no longer synchronized with the other cores. The multi-core control system 130 deactivates, disables, or otherwise ignores the affected cores and the resulting values until these cores are brought back into time lock-step with the other cores via a restart event. A restart event may include resetting one or more cores, or it may include loading one or more cores with known good data and reinitializing the affected cores to use the same inputs in the same processing cycles as the unaffected cores.
[0030] This disclosure can be practiced in any system or device that uses a multi-core control system 130 to provide processing resilience related to SEUs, such as spacecraft, aircraft, X-ray equipment, and other devices intended for use in environments such as space, high altitudes, or other environments where ionizing radiation may alter the state of the processor's bits. These effects also occur in terrestrial applications, and addressing these effects is critical for high-reliability applications such as autonomous vehicles. For example, the illustrated satellite 110 is struck by a cosmic ray 120 carrying sufficient energy to induce an SEU in the core of satellite 110 (e.g., changing logic output A to logic output B, or vice versa).
[0031] Because satellite 110 includes a multi-core control system 130, rather than a single-core control system, multiple cores operate in tandem to control satellite 110, ensuring that a SEU (Self-Extended U-turn) that causes one core to produce an anomalous result does not lead to anomalous operation within satellite 110. For example, the multi-core control system 130 may include n cores (where n > 1), each performing computational tasks in parallel so that satellite 110 operates using values consistent with the computation, rather than simply using the output of a single processor. Therefore, when cosmic rays 120 induce an SEU in the multi-core control system 130, the multi-core control system 130 recognizes the SEU and takes appropriate remedial measures. These measures include stopping the affected core, operating with a reduced set of cores, restarting the affected core, and combinations thereof.
[0032] The multi-core control system 130 can control various subsystems of the system 100, such as Figure 1 The satellite 110 is shown with a communication system 140, a control system 150, and a solar panel management system 160. Depending on what task the multi-core control system 130 is processing at a given time, offline (even briefly) resetting one or more processors affected by the SEU can affect various subsystems in different ways, which in turn affects the operational safety of system 100. For example, when the multi-core control system 130 is processing communications in conjunction with the communication system 140, resetting a core may interfere with communications arriving at or from the satellite, potentially requiring data retransmission (or requesting retransmission) while the multi-core control system 130 is offline, loss of data fidelity (e.g., transmission gaps), etc. In another example, when the multi-core control system 130 is controlling the control system 150, resetting a core may disrupt the position and motion of satellite 110, consume limited fuel to undo incorrectly executed maneuvers, miss maneuver windows, increase collision risk, etc. Therefore, the multi-core control system 130 determines how and when it is best to handle the affected core to reduce the impact of having one less available core to process the equipment's tasks and to reduce the downtime required to restart the affected core, thereby improving the uptime of system 100.
[0033] The multi-core control system 130 can execute various restart events based on the number of active / unaffected cores and whether the system 100 is within a safe restart time. Therefore, different subsystems can have different priorities for restarting or continuing to process a reduced set of cores.
[0034] For example, the control system 150 of satellite 110 can be classified as a high-priority subsystem so that when the multi-core control system 130 is processing instructions or tasks of the control system 150 and undergoes a SEU, the restart event is delayed until the task is completed. In a comparative example, the solar panel management system 160 of satellite 110 can be classified as a low-priority subsystem so that when the multi-core control system 130 is processing instructions or tasks of the solar panel management system 160 and undergoes a SEU, the restart event is executed before the task is completed. In another example, the communication system 140 can be classified as high-priority or low-priority based on the data being transmitted, thereby delaying the restart event when processing high-priority transmissions or prioritizing restart when processing low-priority transmissions.
[0035] In some respects, the multi-core control system 130 identifies a safe time to perform a restart of the affected core based on tags indicated in the instructions processed by the core. For example, the tag may indicate the end of a subroutine or function (e.g., an interrupt in a task), meaning that a restart event can be performed subsequently without interfering with the operation of system 100. In another example, the tag may indicate the subsequent action expected within a predetermined time window for a given subroutine or function, or that it may be subject to an interrupt from a higher-priority subsystem, and therefore a restart cannot even be performed during or after this time without potentially interfering with the operation of system 100.
[0036] Figure 2A-2C Operational diagrams of the multi-core control system 130 are shown according to various aspects of this disclosure, such as... Figure 1 As described in [the text]. Figure 2A-2CIn each of these, multiple cores 210a-n (generally or collectively referred to as core 210) are connected to monitor 220 via associated debug trace data streams 230a-n (generally or collectively referred to as debug trace data or debug trace data stream 230). Monitor 220 is in turn connected to each of the cores 210 via associated reboot paths 240a-n (generally or collectively referred to as reboot path 240) to signal when a given core 210 is to be taken offline, reset, reinitialized, reloaded with data from another core 210, or a combination thereof. Core 210 represents various processing devices, and in some respects may comprise a core of an embedded Reduced Instruction Set Computing (RISC) core or an external device. Monitor 220 is selected to be resilient to the SEU and may include various radiation-resistant processors, hardware-defined state machines, maintenance processors, etc., to monitor and control the various cores 210.
[0037] Monitor 220 communicates with each core 210 via various debug trace streams 230, which may include instruction trace, data trace, system trace, address synchronization trace (ASYNC), instruction synchronization trace (ISYNC), and combinations thereof. ASYNC and ISYNC include patterns and data separately encoded into the trace data stream and may optionally include timestamp data.
[0038] Monitor 220 receives data output from core 210 via debug trace data stream 230 and sends commands to various cores 210 via associated restart path 240 (which controls the stopping and restarting of associated core 210). In various aspects, core 210 and monitor 220 reside on a shared integrated circuit or system-on-a-chip (SoC), and debug trace data stream 230 and restart path 240 include various traces defined on that integrated circuit. In other aspects, one or more cores 210 are defined on different integrated circuits than other cores 210 or monitor 220, and debug trace data stream 230 and restart path 240 include communication paths and buses between different integrated circuits.
[0039] In various aspects, monitor 220 optionally includes memory 250 where the state of the connected core 210 is stored and used to manage how comparator 225 operates during operation as the number of active cores 210 changes. In some aspects, memory 250 includes TMR registers, flip-flops, or latches to capture the current state of the state machine, and it may also be used as program and data storage for a small processor. In other aspects, the memory 250 used by monitor 220 employs robust error-correcting coding (ECC) (i.e., ECC memory).
[0040] Depending on the number of active cores 210 being monitored, the monitor 220 configures the comparator 225 for different comparison modes.
[0041] Figure 2A Multiple cores 210 are shown, where active cores 210a-n provide corresponding inputs to monitor 220 for comparison. Monitor 220 configures comparator 225 for majority mode comparison so that when inconsistent values are output by cores 210 via debug trace data stream 230, monitor 220 uses the first value provided by the first group of cores 210 and disables a second group of cores 210 providing the second value, where the second group has fewer members than the first group. For example, if monitor 220 monitors n cores 210, and at time t1, x cores 210 return output A and y cores return output B, where x+y=n and x>y, then monitor 220 uses output A as the value for the control system and disables or ignores the y cores 210 that returned output B from time t2 until these cores 210 restart. In all respects, when the number of active cores 210 is even, the monitor 220 enables a deadlock handling routine so that the comparator 225 marks all cores 210 as affected when an equal number of cores 210 return different values (i.e., n / 2 return output B and n / 2 return output A). This could occur, for example, if only two cores 210 were operating in parallel to provide a technique for detecting inconsistencies and resetting both processors and / or switching to another pair of cores operating in parallel.
[0042] Figure 2B Multiple cores 210 are shown, with three active cores 210a-c providing input to monitor 220 for comparison. In various aspects, monitor 220 monitors the three active cores 210a-c after disabling additional cores 210 (e.g., monitoring n cores 210, where initially n>3 and currently n=3). When monitoring the three active cores 210, monitor 220 configures comparator 225 for majority mode comparison so that when inconsistencies arise in the values output from cores 210 via debug trace data stream 230, monitor 220 uses the first value provided by two cores 210 and disables the core 210 providing the second value. For example, if monitor 220 monitors three cores 210a-c, and at time t1, the first core 210a returns output A, while the second core 210b and the third core 210c both return output B, then monitor 220 uses output B as the value for controlling the system and disables or ignores the first core 210a from time t2 until the first core 210a is restarted.
[0043] Figure 2CMultiple cores 210 are shown, with two active cores 210a-b providing input to monitor 220 for comparison. In various aspects, monitor 220 monitors two active cores 210a-b after disabling a third core 210c (e.g., monitoring n cores 210, where initially n ≥ 3 and currently n = 2). While monitoring two active cores 210, monitor 220 configures comparator 225 for consistent mode comparison so that when inconsistencies arise in the values output from cores 210 via debug trace data stream 230, monitor 220 ignores both values because neither core 210 can be confirmed to provide "correct" output, and using only one core 210 would not provide any redundancy for future calculations. Therefore, when a state difference is detected between cores 210a-b before continuing the task to control the system, monitor 220 signals both cores 210a-b to restart simultaneously.
[0044] Figure 3 This is a flowchart of a method 300 for core synchronization using debug trace streams according to various aspects of this disclosure. Method 300 begins at block 310, where monitor 220 monitors multiple debug trace data streams 230 for a system comprising multiple cores 210 performing tasks in parallel to achieve redundancy. Monitor 220 ensures that debug trace streams from different cores 220 are time-aligned so that (nominal) equivalent outputs are compared, although their arrival times at comparator 225 may differ slightly. In one example, monitor 220 monitors logic values generated by associated cores 210 in the same clock cycle to identify when one core 210 generates a logic value that matches the output logic value from other cores 210. In other examples, when cores 210 operate on different integrated circuits or otherwise generate output asynchronously, monitor 220 may buffer the outputs to align them for comparison. In another example, when the debug trace stream contains timestamps, monitor 220 ensures that data values associated with the same timestamp are compared.
[0045] In various respects, monitor 220 can selectively activate or otherwise ignore values reported from core 210 that has been previously identified as being affected by the SEU or otherwise producing logical values that do not match the output of other cores 210. Therefore, monitor 220 monitors active cores 210 that have been identified as producing mutually matching outputs. Any core 210 marked as affected can be ignored until it is restarted and brought back to lockstep with other cores 210.
[0046] In block 320, comparator 225 in monitor 220 compares the values carried on each debug trace stream 230 to identify whether any of the active cores 210 produces values that differ from each other in the same clock cycle. In response to detecting a state difference relative to the other debug trace streams 230 on one debug trace stream 230, method 300 proceeds to block 330. Otherwise, when the state values carried by each of the debug trace streams 230 are equal (i.e., no state difference exists), method 300 returns to block 310 to continue monitoring the debug trace streams 230.
[0047] In box 330, the monitor marks a core 210 that produces output that does not match the output of other cores 210 as an affected core 210. The system 100 can no longer rely on this affected core to provide redundant computation with the remaining cores 210. In various aspects, one or more cores 210 may be marked as affected cores 210 simultaneously or for a period of time before a restart. For example, when two cores 210 are operating in consistent mode, where the outputs of the two cores 210a-b must match for use by the system, if the first core 210a is recorded as out of sync with the second core 210b, both the first core 210a and the second core 210b are marked as affected. Similarly, when five cores 210 are operating, if the outputs of the first core 210a and the second core 210b do not match the outputs of the other three cores 210 (which match each other), both the first core 210a and the second core 210b are marked as affected.
[0048] In another example, when operating on n cores 210, the first core 210a can be marked as an affected core at time t1, and the second core 210b can be marked as an affected core 210 at time t2. If a restart event occurs between time t1 and time t2, the first core 210a is no longer marked as an affected core 210, but the second core 210b is marked as an affected core 210. Otherwise, if no restart event occurs between time t1 and time t2, both the first core 210a and the second core 210b are marked as affected cores 210. Therefore, the number of affected cores 210 can increase over time until a restart event occurs.
[0049] In block 340, monitor 220 determines how many unaffected cores 210 remain among the plurality of cores 210. When fewer than two cores 210 remain unaffected, method 300 proceeds to block 370. Otherwise, when two or more cores 210 remain unaffected, method 300 proceeds to block 350.
[0050] In box 350, monitor 220 stops any affected cores 210 and continues to monitor the debug trace data stream 230 of unaffected cores 210 (e.g., as per box 310), thereby reducing the number of cores 210 available to provide redundant computation, but ensuring that the remaining cores 210 are locked in a lockstep.
[0051] Based on the number of unaffected cores 210 remaining to provide redundancy for computation among the multiple cores 210, the monitor 220 can change the redundancy mode used by the comparator 225. For example, when the number of multiple cores 210 decreases from three active cores 210 to two active cores 210 (e.g., the first core 210a is affected and stops, while the second core 210b and the third core 210c are unaffected and continue processing), the monitor 220 switches the comparator 225 from majority redundancy mode to uniform redundancy mode. In another example, when the number of multiple cores 210 decreases from four or more active cores 210 to at least three active cores 210 (i.e., from n to n-1 cores 210, where n≥4), the monitor 220 keeps the comparator 225 in majority redundancy mode. Additionally, when the number of active cores 210 changes from odd to even or vice versa in redundancy mode, the monitor 220 can enable or disable the deadlock handling routine to mark all cores 210 as affected when the number of cores 210 returning the first result is equal to the number of cores 210 returning the second result (i.e., n / 2 return outputs B and n / 2 return outputs A).
[0052] In box 360, monitor 220 determines whether the system has reached a safe restart time. In various aspects, system 100, controlled by core 210, can provide a safety profile to indicate when system 100 is in a state where it is safe to temporarily suspend processing to allow time to restart one or more cores 210. For example, satellite 110 (such as...) Figure 1 As shown, the monitor 220 can indicate that a restart event is safe when satellite 110 is not performing any maneuvers (e.g., adjusting its orbit or avoiding debris) or receiving commands from the ground controller. Therefore, when satellite 110 can choose to continue processing using a reduced set of cores 210 (e.g., disabling or removing at least one affected core 210 from multiple cores 210), the monitor 220 waits for the affected core 210 to restart until satellite 110 reaches a safe restart time (also known as a safe reset time). In various aspects, the safe restart time is indicated by a label in the process performed by core 210, which sets the safe reset bit in the monitor 220 to either output A or output B, indicating whether the system is at a safe restart time.
[0053] In response to determining that system 100 is not at a safe restart time, method 300 returns to block 350 to continue operating system 100 using a reduced set of active cores 210. However, in response to determining that system 100 has reached or is at a safe restart time, method 300 proceeds to block 370 to restart one or more cores 210.
[0054] In box 370, monitor 220 restarts the affected core 210. In various aspects, in addition to the affected core 210, monitor 220 also resets the unaffected core 210 to start all cores 210 from scratch in lockstep (e.g., clearing the corresponding registers and starting all cores 210 with the same instructions). In other aspects, monitor 220 reloads the registers of the affected core 210 with data from the registers of the unaffected core 210 (i.e., “good” register data) and reinitializes the affected core 210 to lockstep with the unaffected core 210.
[0055] If core 210 is on a separate integrated circuit or in a separate power domain, another option for restarting the affected core 210 is to power cycle it. Power cycle restart can be particularly useful in cases where the integrated circuit is susceptible to latch-up, and a power cycle that de-energizes and then powers on a given core can correct for latch-up before a hard fault occurs. Latch-up condition detection can be assumed if the device is not hardened for this possibility or if current measurement of the device can be used to perform a tracking current error comparison. Alternatively, system designers can reset the power cycle instead of resetting or stopping the affected core.
[0056] Rebooting the affected core 210 by reloading and reinitializing it allows the unaffected core 210 to remain active and brings the affected core 210 back into agreement with the unaffected core 210. Once rebooted, the monitor 220 marks the affected core 210 as unaffected (e.g., removes the affected state) and reconfigures the comparator 225 to monitor debug trace data streams 230 from all cores 210. In various aspects, the monitor 220 resets the redundancy mode used for core 210 comparisons from a consistency mode (e.g., for two active cores 210) to a majority mode (e.g., for three or more cores 210).
[0057] As used herein, stopping core 210 can mean stopping instruction execution on core 210 by waiting for instructions, by a hardware breakpoint or hardware trigger, by application reset, power reset, software reset, or by keeping core 210 in a reset state. Restarting core 210 as used herein can mean removing the pause condition by restoring power, deactivating the reset, removing the hardware condition that keeps core 210 in a paused state, rewriting the software wait, or rewriting the pause instruction. Reinitializing core 210 as used herein can mean restoring core 210's registers and / or dedicated memory via debug infrastructure, using debug infrastructure to verify that the registers and / or memory are intact (soft error), or using a reset process including a bootloader to restore core 210 to a known state. Debug infrastructure can include any available hardware path providing core 210 to external hardware, such as JTAG (Joint Test Action Group) or a memory-mapped data bus to a debug mechanism.
[0058] After monitor 220 restarts (one or more) cores 210, method 300 returns to box 310 to continue monitoring debug trace data flow 230 using the now restored full set of available cores 210.
[0059] Figure 4 A computing device 400 is shown according to various aspects of the invention, which may be used, for example, as a multi-core control system 130. The computing device 400 includes a processor 410, a memory 420, and a communication interface 430, wherein the processor 410 includes a plurality of cores 210a-n. In various aspects, a monitor 220 may be included with the processor 410 or as a separate element. The processor 410 and the memory 420 provide computing functions to run various programs and / or operations for the respective computing device 400, including storing and retrieving various data described herein.
[0060] Processor 410 may be any computer processor capable of performing the functions described herein, executing commands based on input received from a user or sensor and data received from communication interface 430.
[0061] Memory 420 is a computer-readable storage device that typically includes various processor-executable instructions that, when executed by processor 410, perform various functions related to the control of a system potentially affected by a SEU, as described herein (e.g., satellite 110, spacecraft, aircraft, etc.). In various aspects, memory 420 may be attached to individual processors 410 or cores 210, or may be a memory subsystem shared by several processors 410 or cores 210. Processor-executable instructions may typically be described or organized into various “applications” or “modules” within memory 420, although alternative implementations may have different functionalities and / or combinations of functionalities. Memory 420 typically also includes data structures storing information used or output by the various applications or modules. In this disclosure, memory 420 includes at least instructions for operating system 421 and one or more applications 422. Memory 420 may include redundancy such as parity checks or error-correcting codes in various aspects, such that, in the event that memory data is accessed by the associated core 210, a fault in a separate memory 420 exclusively used by the associated core 210 can be detected via a trace comparison as described herein. Memory 420 may be one or more memory devices, such as random access memory (RAM), read-only memory (ROM), flash memory, magnetic media, optical media, flexible data storage, or any other type of volatile or non-volatile storage medium that includes instructions executable by processor 410. As used herein, memory storage devices should be understood as descriptive means rather than non-transient signals.
[0062] Communication interface 430 connects computing device 400 to external devices, such as external storage devices, external computing devices, power supplies, wireless transmitters, etc., and may include various connection ports (e.g., Universal Serial Bus (USB), Ethernet, Space Cable, coaxial jack) and cables. Communication interface 430 is used for sending and receiving communication between computing devices 400.
[0063] Reference has been made to various aspects in this disclosure. However, it should be understood that this disclosure is not limited to the aspects specifically described. Rather, any combination of the following features and elements, whether or not they relate to different aspects, is contemplated for implementing and practicing the guidance provided herein. Furthermore, when elements of an aspect are described in the form of “at least one of A and B,” it will be understood that including only element A, including only element B, and including both elements A and B are all contemplated. Moreover, while some aspects may achieve advantages over other possible solutions and / or over the prior art, whether a particular advantage is achieved by a given aspect does not limit the invention. Therefore, the aspects, features, aspects, and advantages disclosed herein are merely illustrative and should not be considered elements or limitations of the appended claims unless expressly stated in the claims. Similarly, references to “the invention” should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered elements or limitations of the appended claims unless expressly stated in the claims.
[0064] As those skilled in the art will understand, the aspects described herein can be embodied as systems, methods, or computer program products. Therefore, the aspects can take the form of entirely hardware, entirely software (including firmware, embedded software, resident software, microcode, etc.), or a combination of software and hardware, which herein can be collectively referred to as “circuit,” “module,” or “system.” Furthermore, the aspects described herein can take the form of computer program products contained in one or more computer-readable storage media having computer-readable program code thereon.
[0065] Any suitable medium may be used to transmit program code contained on a computer-readable storage medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.
[0066] Computer program code used to perform operations relating to the aspects of this disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., via the Internet provided by an Internet service provider).
[0067] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to various aspects of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in the blocks of the flowchart illustrations and / or block diagrams.
[0068] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing comprising instructions that implement the functions / operations specified in the boxes of a flowchart and / or block diagram.
[0069] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable data processing apparatus or other device, provide for implementing the functions / operations specified in the boxes of the flowchart and / or block diagram.
[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative embodiments, the functions indicated in the blocks may occur in a non-linear order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order or out of order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified function or operation.
[0071] Furthermore, this disclosure includes embodiments pursuant to the following terms:
[0072] Clause 1. A method comprising:
[0073] Monitoring (310) is used for multiple debug trace data streams (230) of a redundant operating system, which includes multiple corresponding cores (210) that perform tasks in parallel;
[0074] In response to detecting (320) a state difference between one of the plurality of debug trace data streams and the other debug trace data streams in the plurality of debug trace data streams:
[0075] The given kernel tag (330) associated with the aforementioned debug trace data stream is designated as the affected kernel; and
[0076] Reboot (370) the affected kernel.
[0077] Clause 2. The method according to Clause 1, wherein the corresponding plurality of cores comprises two active cores, including the affected core and the second core, the method further comprising, in response to detecting the state difference:
[0078] Mark the second core as the second affected core; and
[0079] The second core is reset simultaneously with the affected core.
[0080] Clause 3. The method according to Clause 1, wherein the corresponding plurality of cores comprises three active cores, including the affected core, a first unaffected core, and a second unaffected core, the method further comprising, in response to detecting the state difference:
[0081] Stop (350) the affected nuclei; and
[0082] Monitor the first unaffected core and the second unaffected core executing the task in parallel in a consistent redundancy mode; and
[0083] The restarting of the affected kernel includes responding to the detection of a safe restart time:
[0084] Load the affected core with register data from the first unaffected core; and
[0085] Reinitialize the affected kernel.
[0086] Clause 4. The method according to Clause 1, wherein the corresponding plurality of cores includes more than three active cores, the more than three active cores including the affected core, a first unaffected core, a second unaffected core, and a third unaffected core, in response to detecting the state difference:
[0087] Stop (350) the affected nuclei; and
[0088] Monitor the first unaffected core, the second unaffected core, and the third unaffected core that are executing the task in parallel in majority mode; and
[0089] The restart of the affected kernel includes, in response to the detection of a (360) safe reboot time:
[0090] The affected core is reloaded with register data from the first unaffected core; and
[0091] Reinitialize the affected kernel.
[0092] Clause 5. The method according to Clause 1, wherein the corresponding plurality of cores are included on a shared system-on-chip (SoC) having a monitor (220), wherein the monitor is one of the following:
[0093] Radiation-resistant processor;
[0094] Hardware-defined state machines; and
[0095] Maintain the processor.
[0096] Clause 6. The method according to Clause 1, wherein at least one of the corresponding plurality of cores is included on an integrated circuit separate from the other cores of the corresponding plurality of cores.
[0097] Clause 7. A system (100) comprising:
[0098] Multiple cores (210) are configured to execute corresponding instances of tasks in parallel; and
[0099] Monitor (220), which is configured as follows:
[0100] Receive debug trace data streams (230) from the multiple cores; and
[0101] The affected core is identified based on the state differences between debug trace data streams associated with the affected core and other cores among the plurality of cores.
[0102] Clause 8. The system according to Clause 7, wherein the plurality of cores and the monitor are arranged on a single integrated circuit.
[0103] Clause 9. The system according to Clause 7, wherein the monitor is disposed on a first integrated circuit, and at least one of the plurality of cores is disposed on a second integrated circuit.
[0104] Clause 10. The system according to Clause 7, wherein the monitor is further configured to:
[0105] Remove the affected core from the plurality of cores performing the task until a safe reboot time; and
[0106] In response to reaching the safe reboot time, the affected kernel is rebooted.
[0107] Clause 11. The system according to Clause 10, wherein, when the plurality of cores comprises at least two cores after the removal of the affected core, the monitor is further configured to:
[0108] The affected registers associated with the affected core are reloaded using data from the unaffected registers associated with one of the at least two unaffected cores;
[0109] Reinitialize the affected kernel; and
[0110] The affected cores are returned to the plurality of cores.
[0111] Clause 12. The system according to Clause 10, wherein the safe restart time is determined based on the security profile of the device controlled by the system.
[0112] Clause 13. The system according to Clause 10, wherein when the plurality of cores consists of two cores, the monitor is further configured to:
[0113] The second core is reset simultaneously with the affected core.
[0114] Clause 14. The system pursuant to Clause 7, wherein the debug trace data stream is received via at least one of the following:
[0115] Instruction tracing;
[0116] Data tracking; and
[0117] System tracing.
[0118] Clause 15. A computer-readable storage device including instructions that, when executed by a processor, perform operations including:
[0119] Monitor multiple debug trace data streams (230) for a redundant operating system, the redundant operating system comprising multiple corresponding cores (210) that execute tasks in parallel;
[0120] In response to detecting a state difference between one of the plurality of debug trace data streams and the other debug trace data streams in the plurality of debug trace data streams:
[0121] Mark the kernel associated with the aforementioned debug trace data stream as the affected kernel; and
[0122] Restart the affected kernel.
[0123] Clause 16. The computer-readable storage device according to Clause 15, wherein the corresponding plurality of cores comprises two active cores, including the affected core and the second core, the operation further comprising, in response to detecting the state difference:
[0124] Mark the second core as the second affected core; and
[0125] The second core is reset simultaneously with the affected core.
[0126] Clause 17. The computer-readable storage device according to Clause 15, wherein the corresponding plurality of cores comprises three active cores, including the affected core, a first unaffected core, and a second unaffected core, the operation further comprising, in response to detecting the state difference:
[0127] Stop the affected nuclei; and
[0128] Monitor the first unaffected core and the second unaffected core to execute the task in parallel in a consistent redundancy mode; and
[0129] The restarting of the affected kernel includes responding to the detection of a safe restart time:
[0130] Load the affected core with register data from the first unaffected core; and
[0131] Reinitialize the affected kernel.
[0132] Clause 18. The computer-readable storage device according to Clause 15, wherein the corresponding plurality of cores includes more than three active cores, the more than three active cores including the affected core, a first unaffected core, a second unaffected core, and a third unaffected core, the operation further comprising responding to the detection of the state difference:
[0133] Stop the affected nuclei; and
[0134] Monitor the first unaffected core, the second unaffected core, and the third unaffected core to execute the task in parallel in majority mode; and
[0135] The restarting of the affected kernel includes responding to the detection of a safe restart time:
[0136] Load the affected core with register data from the first unaffected core; and
[0137] Reinitialize the affected kernel.
[0138] Clause 19. The computer-readable storage device according to Clause 15, wherein the corresponding plurality of cores are included on a shared system-on-chip (SoC) having a monitor (220) configured to monitor the plurality of debug trace data streams.
[0139] Clause 20. The computer-readable storage device as described in Clause 15, wherein at least one of the corresponding plurality of cores is included on an integrated circuit separate from the other cores of the corresponding plurality of cores.
[0140] Clause 21. Any aspect described herein.
[0141] While the foregoing addresses aspects of this disclosure, other and further aspects of this disclosure may be designed without departing from its essential scope, which is defined by the appended claims.
Claims
1. A method comprising: Monitor multiple debug trace data streams for a redundant operating system, the redundant operating system comprising multiple corresponding cores executing tasks in parallel, wherein the consistency of output values among the multiple cores controls the redundant operating system, wherein the multiple cores are in a lockstep when the output values are consistent; In response to detecting a state difference between one of the plurality of debug trace data streams and the other debug trace data streams in the plurality of debug trace data streams: A given core associated with one of the debug trace data streams is marked as an affected core, wherein the affected core is ignored by the redundant operating system until the affected core is rebooted; and The affected cores are restarted, and the restarted cores are brought back to lockstep with the plurality of cores.
2. The method of claim 1, wherein the corresponding plurality of cores comprises two active cores, including the affected core and the second core, the method further comprising, in response to detecting the state difference: Mark the second core as the second affected core; and The second core is reset simultaneously with the affected core.
3. The method of claim 1, wherein the corresponding plurality of cores comprises three active cores, including the affected core, a first unaffected core, and a second unaffected core, the method further comprising, in response to detecting the state difference: stopping the affected core; and Monitor the first unaffected core and the second unaffected core, which execute the task in parallel in a consistent redundancy mode; and The restarting of the affected kernel includes responding to the detection of a safe restart time: The affected core is loaded with register data from the first unaffected core; and Reinitialize the affected kernel.
4. The method according to any one of claims 1-3, wherein the corresponding plurality of cores comprises more than three active cores, the more than three active cores comprising the affected core, a first unaffected core, a second unaffected core, and a third unaffected core, in response to detecting the state difference: Stop the affected kernels; and Monitor the first unaffected core, the second unaffected core, and the third unaffected core, which execute the task in parallel in majority mode; and The restarting of the affected kernel includes responding to the detection of a safe restart time: The affected core is reloaded using register data from the first unaffected core; and Reinitialize the affected kernel.
5. The method of claim 1, wherein the corresponding plurality of cores are included on a shared on-chip system having a monitor, wherein the monitor is one of the following: Radiation-resistant processor; Hardware-defined state machines; and Maintain the processor.
6. The method of claim 1, wherein at least one of the corresponding plurality of cores is included on an integrated circuit separate from the other cores of the corresponding plurality of cores.
7. A system comprising: Multiple cores are configured to execute corresponding instances of a task in parallel, wherein the consistency of the output values of the multiple cores controls the system, and the multiple cores are in a lockstep when the output values are consistent; and The monitor is configured as follows: Receive debug trace data streams from the multiple cores; The affected cores among the plurality of cores are identified based on the state differences between debug trace data streams associated with the affected cores and other cores. Ignore the affected kernels until they are rebooted; and The affected cores are restarted, and the restarted cores are brought back to lockstep with the plurality of cores.
8. The system of claim 7, wherein the plurality of cores and the monitor are arranged on a single integrated circuit.
9. The system of claim 7, wherein the monitor is disposed on the first integrated circuit, and at least one of the plurality of cores is disposed on the second integrated circuit.
10. The system of claim 7, wherein the monitor is further configured to: Remove the affected core from the plurality of cores performing the task until a safe reboot time; and In response to reaching the safe restart time, the affected kernel is restarted.
11. The system of claim 10, wherein when the plurality of cores includes at least two cores after the removal of the affected core, the monitor is further configured to: The affected registers associated with the affected core are reloaded using data from the unaffected registers associated with one of the at least two cores that are not affected. Reinitialize the affected kernel; and The affected cores are returned to the plurality of cores.
12. The system of claim 10, wherein the safe restart time is determined based on the safety profile of the device controlled by the system.
13. The system of claim 10, wherein the plurality of cores comprises two cores, and the monitor is further configured to: The second core is reset simultaneously with the affected core.
14. The system of claim 7, wherein the debug trace data stream is received via at least one of the following: Instruction tracing; Data tracking; and System tracing.
Citation Information
Patent Citations
Detection of runtime failures in a system on chip using debug circuitry
US10754760B1
Reducing the uncorrectable error rate in a lockstepped dual-modular redundancy system
US20070022348A1
Data processor and method for updating data
WO2007074739A1
