Collect runtime information for debugging and profiling

Abstract

A system on chip (SOC) system includes functional modules, including first and second functional modules. The first and second functional modules are configured to send first and second status information associated with the first and second functional modules to an aggregation module according to a first set and a second set of one or more reporting rules and in a standardized message format, respectively. The aggregation module aggregates the first status information in the standardized message format and the second status information in the standardized message format, and inserts a timestamp to obtain a timestamped and aggregated message stream. The timestamped and aggregated message stream is stored, and a visualization system is enabled to analyze the first functional module and the second functional module.

Description

[0001] Cross-references to other applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 222,257, filed on July 15, 2021, entitled COLLECTION OF RUNTIME INFORMATION FOR DEBUG AND ANALYSIS, which is incorporated herein by reference for all purposes. Background Art

[0003] Given the real-time nature and complexity of embedded and / or system-on-chip (SOC) systems, unexpected behavior in such systems is notoriously difficult to debug. Debugging techniques that work with other types of systems (such as adding intrusive debug probes at key locations in non-real-time systems, "brute force" debugging in simple systems, and adding probes at easily accessible points in distributed systems) are not easily portable and / or feasible for some embedded and / or SOC systems. New technologies that provide better tools and / or techniques for analyzing, debugging, and / or monitoring such systems would be desirable. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0005] Figure 1 is a flow chart illustrating an embodiment of a process of generating and storing time-stamped and aggregated messages, which in turn may be used to analyze functional modules in a SOC system.

[0006] Figure 2 is a system diagram illustrating an embodiment of a SOC system, wherein each functional module has a reporting module for collecting and storing status information.

[0007] Figure 3 is a diagram illustrating embodiments of time-based messages, event-based messages, and timestamp messages.

[0008] Figure 4 are diagrams illustrating two embodiments of aggregated and time-stamped message flow in a message capture memory.

[0009] Figure 5 is a diagram illustrating an embodiment of selectable event groups in a flash storage controller application.

[0010] Figure 6 is a flow chart illustrating an embodiment of a process for sending first status information in a standardized message format. DETAILED DESCRIPTION

[0011] The present invention may be implemented in a variety of ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, for example, a processor configured to execute instructions stored in a memory coupled to the processor and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present invention may take may be referred to as techniques. In general, within the scope of the present invention, the order of steps of the disclosed processes may be changed. Unless otherwise stated, components such as processors or memories described as being configured to perform tasks may be implemented as general components that are temporarily configured to perform tasks at a given time, or as specific components that are manufactured to perform the tasks. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data such as computer program instructions.

[0012] A detailed description of one or more embodiments of the present invention is provided below together with the accompanying drawings that illustrate the principles of the present invention. The present invention is described in conjunction with such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses a variety of alternatives, modifications and equivalents. In order to provide a thorough understanding of the present invention, a number of specific details are set forth in the following description. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the sake of clarity, technical materials known in the technical field related to the present invention are not described in detail to avoid unnecessarily obscuring the present invention.

[0013] Various embodiments of techniques and / or systems for capturing, generating, and / or saving information (e.g., state, status, and / or event information) that can be used to analyze embedded and / or system-on-chip (SOC system) are used interchangeably. For applications where there are many SOC system instances in operation, existing debugging and / or analysis techniques are insufficient. For example, assume that a storage controller (such as a solid-state drive (SSD) controller) is implemented on the SOC system. For large storage systems such as cloud and / or enterprise storage systems, there may be hundreds or even thousands of storage controllers in operation. Existing debugging techniques may only Exposing a limited number of I / O test pins (e.g., approximately four or eight I / O test pins) is often inadequate given the number and complexity of the various functional modules in a single SOC system, coupled with the sheer number of embedded and / or SOC systems in operation. Using the techniques described below, information can be generated and stored that enables a visualization system to review, debug, or otherwise analyze the operation of functional modules in one or more SOC systems in greater detail and / or with better (e.g., timestamps and / or cross-functional module) annotations than was previously available. The following figure depicts one embodiment of this technique for generating such information.

[0014] Figure 1 is a flow chart illustrating an embodiment of a process of generating and storing time-stamped and aggregated messages, which in turn can be used to analyze functional modules in a SOC system. In some embodiments, the process is performed by a SOC system configured to perform the steps described below.

[0015] At 100 , a first functional module among a plurality of functional modules is configured to send first status information associated with the first functional module to an aggregation module according to a first set of one or more reporting rules and in a standardized message format.

[0016] At 102, a second functional module in the plurality of functional modules is operable to send second status information associated with the second functional module to the aggregation module in accordance with a second set of one or more reporting rules and in a standardized message format. Note that the status information, content, and / or payload in the standardized message format may be different for the first functional module (e.g., at step 100) and the second functional module (e.g., at step 102), but the two functional modules use the same standardized message format to package or otherwise encapsulate their information.

[0017] As used herein, the term "functional module" refers to any logical or defined module (e.g., defined in RTL or code) whose status information (e.g., including state information, event information, etc.) is captured. A functional module may (for example) perform or support one or more operations and have inputs and outputs, and the functional module communicates with and / or is coupled to other functional modules in the SOC system via these inputs and outputs. For example, in an RTL describing hardware or a code including firmware, a functional module may correspond to a hardware or firmware module in the RTL or firmware code, respectively. In various embodiments, a functional module may be a firmware module or a hardware module. In various embodiments, the first and second functional modules may be of the same type (e.g., both the first and second functional modules are hardware) or of different types (e.g., the first functional module is firmware, and the second functional module is hardware). In some embodiments, the functional module includes a CPU (or other embedded processor) and firmware configured to run on the CPU in the (same) functional module.

[0018] The reporting rules (e.g., those mentioned in steps 100 and 102) define what status information is included in the standardized message format for the corresponding functional module and when such information is sent. In one example, the first and second functional modules have different functions and / or support different sets of functions, and therefore the reporting rules for those functional modules are different (e.g., appropriate for each functional module and what will be of interest during debugging and / or analysis of the functional module). Alternatively, if there are duplicate functional modules, the reporting rules may be the same.

[0019] In some embodiments, the reporting rules include event-based reporting rules, wherein if a predefined condition or event (e.g., described in the reporting rules) is met or otherwise satisfied in a given functional module, status information is sent (e.g., once each time a given event is detected). The reporting rules in such embodiments further include what status information to include in a standardized message format when a given event occurs. For example, assume that the functional module is associated with communicating over a channel, where the operation is bookended by a start event and an end event with one or more intermediate events (e.g., event(s) associated with handshaking, obtaining access to a shared communication channel, confirmation of a transmission, etc.). If one of these events occurs, the reporting rules may (as an example) instruct the functional module to include (e.g., in a standardized message format): (1) information identifying the event, and (2) selected information related to the particular event (e.g., an associated state machine, the status of an associated buffer, the status of a communication partner, a pointer or location at which a transmitted message is located, a sequence number or other metadata associated with the transmitted message, etc.), allowing space in the standardized message format.

[0020] In some embodiments, the reporting rules include time-based reporting rules, where if a functional module meets a condition specified in the time-based reporting rule, then the status information (specified by the time-based reporting rule) is periodically sent (e.g., at predefined or set time intervals) until the condition is no longer met. Conceptually, time-based reporting is similar to I / O test pins, which expose select signals or registers within the SOC, except that reporting occurs at discrete times (non-continuous times) and reporting only occurs when the condition is met. In addition, the range and / or number of signals reported exceeds the typical four or eight signals exposed by a limited number of I / O test pins. In addition, the timestamps applied by the aggregation module allow different signals or pieces of information to be cross-referenced during analysis and / or display (e.g., within a given functional module or across functional modules). In fact, in order to keep the amount of status information reported to a more manageable or more processable amount of information, time-based reporting (at least in this example) only occurs when the condition is met in this example. (Alternatively, in some embodiments, there may be no condition, and the specified status information is always reported. In one example, if the status information being reported is the value or state of a finite state machine in a functional module, then the value or state is reported (only) when the finite state machine is not in an idle state.

[0021] At 104, an aggregation module is used to aggregate the first status information in the standardized message format and the second status information in the standardized message format. In some embodiments, each functional module has a dedicated link to the aggregation module via which messages (e.g., in the standardized format and containing the status information) are sent to the aggregation module. When messages are received on these dedicated links, they can be merged from multiple streams into a single stream.

[0022] At 106, the aggregation module is used to insert the timestamp to obtain a timestamped and aggregated message stream, wherein the timestamped and aggregated message stream enables the visualization system to analyze the first functional module and the second functional module. In various embodiments, as desired, aggregation can be performed first (e.g., at step 104), and then timestamping (e.g., at step 106), or vice versa.

[0023] The insertion of timestamps enables a visualization system (which ingests the timestamped and aggregated streams) to align different status information from various functional modules along a common time axis. Similarly, within a single functional module, timestamps throughout the timestamped and aggregated message streams permit comparison and / or temporal correlation of status information associated with different reporting rules (e.g., one event-based reporting rule is searching for event X, while another event-based reporting rule is searching for event Y within the single functional module).

[0024] In some embodiments, time stamping (e.g., at step 106) is performed in a manner that permits subsequent comparison of status information of different SOC systems using a common timeline or time reference. For example, this can be accomplished by synchronizing the timestamp value in the SOC with the timestamp value of the host system with which the SOC system is interacting. Alternatively, in some embodiments, no attempt is made to establish a common timeline or time reference because when comparing different SOCs for a given sequence of events, the time between events and the time for 1 I / O to flow from one end to the other are comparison factors.

[0025] At 108, a memory is used to store the time-stamped and aggregated message stream. As will be described in more detail below, the memory can act as a buffer with the time-stamped and aggregated message stream exported from the SOC system as quickly as possible so that the memory does not overflow. In some embodiments, the time-stamped and aggregated message stream is sent from the memory (e.g., on the SOC system) to a visualization system (e.g., for just-in-time and / or real-time debugging or analysis). Alternatively, the time-stamped and aggregated message stream can be sent from the memory (e.g., on the SOC system) to some non-SOC storage medium, from which the visualization system can subsequently retrieve the message stream if analysis is desired later.

[0026] As described above, in some embodiments, a first set of one or more reporting rules includes an event-based reporting rule; and sending first status information in a standardized message format includes: determining whether the first functional module satisfies the event-based reporting rule; and in response to determining that the first functional module satisfies the event-based reporting rule, sending status information specified by the event-based reporting rule to the aggregation module in a standardized message format.

[0027] As described above, in some embodiments, a first set of one or more reporting rules includes a time-based reporting rule; and sending first status information in a standardized message format includes: determining whether the first functional module satisfies the time-based reporting rule; and when the first functional module satisfies the time-based reporting rule, periodically sending status information specified by the time-based reporting rule to the aggregation module in a standardized message format.

[0028] In contrast to debugging techniques that are limited to a small number of test I / O pins, using Figure 1 The process of testing I / O pins can obtain a more complete and richer picture of the SOC system. For example, a limited number of test I / O pins may usually only expose one state machine associated with one functional module (as an example); given the limited number of test I / O pins, this does not provide insight into what is happening in other functional modules at the same time.

[0029] Furthermore, since the functional modules can be hardware modules and / or firmware modules, Figure 1 The process permits comparison and / or cross-correlation of what both the firmware and the hardware are doing. This is supported by a common insertion point (e.g., aggregation module) of timestamps so that hardware status information (e.g., from a given hardware module) and firmware status information (e.g., from a given firmware module) can be appropriately placed along a common timeline for presentation or display to a user.

[0030] Note that design and construction execution Figure 1 The process of debugging infrastructure and / or (one or more) modules is relatively time consuming. Large-scale storage systems (e.g., cloud and / or enterprise storage systems) that use hundreds or even thousands of storage controllers (e.g., each storage controller may be implemented on a SOC system) are much more difficult to debug; in such applications, investing in a robust debugging infrastructure is a worthwhile investment. Even for smaller tools with only one storage controller, investing in the infrastructure and / or (one or more) modules described in this article may be worthwhile because it facilitates performance tuning because one storage device represents all storage devices.

[0031] Graphical execution Figure 1 An example system of the process may help. The following figure shows one such example.

[0032] Figure 2 The system diagram of the embodiment of the SOC system is shown, and each functional module has a reporting module for collecting and storing status information. In this example, the exemplary SOC system (200) includes multiple functional modules (202). In various embodiments, the exemplary SOC system (200) is an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. In various embodiments, the functional module (202) is a hardware module composed of only electronic circuits, a firmware module composed of instruction codes operating on a processing unit, etc.

[0033] In this example, the ability to analyze (e.g., in real time or after some latency) the behavior and / or operation of each of a plurality of functional modules (202) is desired. For example, each functional module may contribute to critical processing or movement of data, making subsequent analysis and / or review highly desirable. To this end, each functional module (202) has a subcomponent, referred to in the figure as a reporting module (204), which collects status information from within its functional module, encapsulates the status information into a message, and sends the encapsulated information over a dedicated link (206) to a central message collection module, referred to in the figure as an aggregation module (208). In some embodiments, less critical and / or less interesting (from a debugging perspective) functional modules do not have a reporting module.

[0034] In this example, each reporting module (204) includes one or more reporting rules (not shown) that describe the conditions under which time-based messages and / or event-based messages are generated and sent. These reporting rules can also describe what specific registers, values, nodes, states, etc. should be included or otherwise used as status information sent to the aggregation module (208). For example, each reporting module (204) can include a controller that compares the condition(s) specified in the reporting rules with the relevant variables, states, events, etc. in the particular functional module.

[0035] In this example, each reporting module (204) has a dedicated link (206) to the aggregation module (208). Although shared links can be used, they have the potential for lost messages at the reporting module if the shared link is being used by another reporting module and the underserved reporting module runs out of buffers. For example, suppose one of the functional modules has bursty messages; with a shared link, the bursty functional module can starve the non-bursty functional modules. Using dedicated links reduces the likelihood of such lost messages.

[0036] Since timestamping is performed at the aggregation module (208) rather than at the reporting module (204), it is important that messages arrive at the aggregation module in a timely manner (i.e., messages from the various links are time sensitive). This is another reason why it may be desirable to use dedicated links (206) rather than shared links. Aggregation is also performed at the aggregation module (208), so it is important to maintain an accurate representation of the input sequence (i.e., messages from the various links are sequence sensitive). This is another benefit of using dedicated links (206): they help maintain correct and / or accurate message sequences. In some embodiments, the aggregation module (208) is implemented in hardware (e.g., rather than firmware) to better keep up with the messages generated by the various reporting modules (204).

[0037] In this example, to save power, the link (206) between the reporting module (204) and the aggregation module (208) operates at a fraction of the nominal operating frequency of the rest of the SOC system. For example, if the nominal operating frequency in the SOC system is 1 GHz, the link can operate at 1 / 4 of that frequency (e.g., 250 MHz) or 1 / 5 of that frequency (e.g., 200 MHz). Similarly, because the messages are relatively small (e.g., in the range of 16-64 bits), the width of each link (206) can be on the order of 16 bits, while some buses in the rest of the SOC system (200) are on the order of 64 bits. The low frequency and small amount of data generated by this debug structure allow the overhead of a typical system to be very low, allowing this feature to be enabled at all times during real-world operation. This in turn enables real fault (data) capture, rapid fault analysis, and / or classification of real-world field and system faults.

[0038] In this example, the aggregation module (208) receives messages from the various links (206) and aggregates them into a single aggregate stream. The timestamp is then inserted into the aggregate stream to obtain a timestamped and aggregated stream. (Note that while this example describes first aggregating and then timestamping, Figure 1 The process of merging is in no specified order, and timestamping can be performed before aggregation if desired).

[0039] The time stamped and aggregated stream is passed from the aggregation module (208) to the memory (210). The time stamped and aggregated stream is transported from the memory (210) to the visualization system (214) from the SOC system (200) via the communication interface (212). In this example, the visualization system (214) is configured to store, filter, display, present and / or analyze the received stream as instructed by the user. For example, if the user wants to immediately observe and / or analyze the operation of the SOC system (200) in real time, this path from the communication interface (212) to the visualization system (214) may be desirable.

[0040] In this example, the timestamped and aggregated streams in the memory (210) are also passed to the storage interface (216) so that the information can be exported off-chip to an (external) storage medium (218), such as flash and / or solid-state drive (SSD) memory. For example, the test infrastructure is first prototyped and / or implemented on a storage controller, and thus the storage interface (216) is already implemented. In other words, in some embodiments, the storage interface (216) is a "production" interface that is already implemented and / or used by routine, non-debugging related operations supported by the SOC (200).

[0041] For illustrative purposes, multiple interfaces or paths (e.g., 212 and 216) are shown in this figure, via which timestamped and aggregated flows are exported from the SOC system (200). In some embodiments, only a single interface or path is implemented (e.g., depending on the application and / or desire (or lack thereof) for real-time analysis of the SOC system and / or its functional modules).

[0042] In some cases, the memory (210) becomes full. In some embodiments, if this happens, older messages are overwritten to continue capturing new information, or new messages can be discarded to allow old information to be saved. In some cases, older messages are overwritten with newer messages to make future debuggability better, assuming that any error event will trigger an interrupt to the processor, and the processor can interrupt such an overwrite or clear the log containing the error. The aggregation module (208) in this example is responsible for detecting when the memory (210) becomes full and deciding how to handle it.

[0043] In one particular example, the SOC (200) is a flash storage controller and the storage medium (218) is a flash storage medium being managed and / or controlled by the flash storage controller. In this application, because there is a readily available storage medium (218) and the SOC (200) controls writing to the storage medium (e.g., so there is no concern that any captured information is accidentally overwritten by another device), the information in the memory (210) is sent off-chip via the storage interface (216) and the communication interface (212) is not implemented. If analysis by a visualization system is desired, the visualization system can access the storage medium (216) to retrieve (e.g., debug) information stored therein.

[0044] It may be helpful to give some examples of standardized message formats that may be used by the reporting module (204). The following figure depicts some such examples.

[0045] Figure 3 is a diagram illustrating embodiments of time-based messages, event-based messages, and timestamp messages. These are some examples of standardized message formats, where Figure 1 In steps 100 and 102, the status information may be included. Figure 2 In the example standardized message formats shown herein, the reporting module (204) can send status information to the aggregation module (208). In various embodiments, the system can support any number of standardized message formats (ie, message types).

[0046] In this example, three message types are shown, identified by a type field (322) in a fixed position in the message, which in this example is the first three bits of each message. In this example, the first message type is a time-based message (300), where the first field (comprising 3 bits) is the type field (302) having a value identifying the message as a time-based message.

[0047] The Type field (302) is followed by a 3-bit Sequence field (304). This field is a time-based sequence identifier that is used when a given functional module is operating more frequently than a given reporting module (e.g., Figure 2 204 in ) and aggregation modules (e.g. Figure 2 This field is included in the case where the link frequency between the ports and the aggregation module is different (e.g., 208). For example, this is a common situation when the interconnect between the ports and the aggregation module operates at (as an example) 25% of the nominal system operating frequency, which (as described above) saves power. With a system so configured, messages may occasionally be dropped because the rate of generation (e.g., by the reporting module) is greater than the rate of absorption or transportation (e.g., by the aggregation module). This condition is detected and post-processed as a non-uniform increment of the sequence identifier of two adjacent messages. In one example, the sequence field (304) is incremented by one each time a new time-based message is sent so that if there are two or more hops, the post-processor knows that some messages have been lost. In some embodiments, the sequence number (304) is based on and / or includes some bits from a timestamp used to timestamp the message (340).

[0048] The next field in the time-based message (300) is a payload field (306) having 10 bits. For example, the time-based reporting rules may specify what status information is included in the payload field (306) of the time-based message (300). In one example, the time-based messages (300) including the value or state of the finite state machine are sent when (or when) the finite state machine is not in the idle state. These time-based messages (300) continue to be sent periodically until the finite state machine returns to the idle state, at which point the time-based message generation ceases (at least in this example).

[0049] The second type of message in this example is an event-based message (320). As with the other message types, the first field is a 3-bit type field (322) where the value identifies the message as an event-based message.

[0050] Next is a 13-bit tag field (324) and a 3-bit subtype field (326). The subtype field (326) is sometimes referred to as an event identifier field because the various events captured and reported by the event-based message are each assigned an event number or identifier. In this example, because the subtype (i.e., event identifier) ​​field (326) has 3 bits, the event identifier ranges from 0 to 7, inclusive.

[0051] For some functional modules, having eight event identifiers is sufficient to uniquely identify all events that are desired to be recorded. However, some functional modules support many different types of operations (e.g., transmission as well as reception), have many intermediate events of interest between start and end events, and / or have control channels and data channels (each with its own associated events). Therefore, in some cases, eight event identifiers are insufficient to uniquely identify all events that are desired to be reported. In this example, to accommodate such situations, some events share event identifier values, and a portion of the tag field (324) is used to distinguish between events that share the event identifier value. In the case where the event identifier value is unique (i.e., it is associated with only one event), the tag is used to transmit other information, such as additional status information (e.g., according to instructions in the associated event-based reporting rule).

[0052] Next in the event-based message (320) are two reserved fields. The first reserved field (328) is a 5-bit space reserved for replacing the identifier of the function module that generated the message. For example, in Figure 2 In the example embodiment, the functional module (202) generates the event-based message (320), but leaves the first reserved field (328) empty. At the aggregation module (208), the first reserved field (328) is populated by the aggregation module (208) with an identifier associated with the functional module (202) from which the event-based message (320) is received (e.g., known because the link (206) is dedicated, not shared).

[0053] The second reserved field (330) is an 8-bit field that is reserved for use as a replacement for the timestamp. The event message timestamp (330) is used for the same purpose as the time-based sequence identifier (304) in the time-based message (300). However, because event-based messages occur more sporadically than time-based messages, they require a larger field to capture larger time differences between event-based messages (320) or between an event-based message (320) and a timestamp message (340). Like the first reserved field (328), the second reserved field is used by a reporting module (e.g., Figure 2 204) and / or functional modules (e.g., Figure 2 202 in the ) and leave it blank in the aggregation module (for example, Figure 2Fill in 208).

[0054] The last field in the event-based message (320) is a 32-bit payload field (332) that carries the payload associated with the relevant event. The specific (status) information included in the payload (332) is specified by the relevant event-based reporting rule.

[0055] Conceptually, event-based reporting (e.g., using event-based reporting rules) can be viewed as a (more) sparse reporting technique because only one event-based message is generated when a condition is detected. In contrast, time-based reporting (e.g., using time-based reporting rules) is a (more) intensive reporting technique because time-based messages are continuously generated when a condition is met. As such, time-based reporting will tend to generate significantly more messages than event-based reporting. To account for this difference, in this example, the size of the exemplary time-based message (300) is smaller than the size of the exemplary event-based message (320) (e.g., the exemplary time-based message (300) is 16 bits, while the exemplary event-based message (320) is 64 bits).

[0056] The third message in this example is a timestamp message (340) that includes a 3-bit type field (342) for identifying the message as a timestamp message and a 29-bit payload field (344) for storing a timestamp value. In some embodiments, the timestamp written into the second reserved field (330) of the event-based message (320) is the least significant eight bits of a longer timestamp included in the payload (344) of the timestamp message (342).

[0057] In some embodiments, the generation or insertion of a timestamp message is time driven (e.g., it has been longer than some time threshold since the last timestamp message was generated, so it is time to generate another timestamp message). In some embodiments, it is event driven. For example, since time-based messages (300) have a sequence but no timestamps, in some embodiments, a timestamp is inserted if (e.g., in an aggregated stream) a certain threshold number of consecutive time-based messages has been reached without an event-based message (which has a timestamp) or a timestamp message occurring.

[0058] In this example, the timestamp message (340) does not cross message interconnects (e.g., Figure 2 Instead, if the message flow is interrupted for any reason (e.g., temporary storage is unavailable), the timestamp message in this example is sent by the aggregation module (e.g., Figure 2 208) is generated and inserted into the write message capture memory (e.g., Figure 2210) for accurate representation.

[0059] As shown in this example, in some embodiments, timestamping (e.g., Figure 1 ) includes inserting a timestamp message (eg, 340) into the aggregate message stream.

[0060] This example also shows that in some embodiments, time stamping (e.g., Figure 1 ) includes writing the timestamp to a reserved field (eg, 330) in the standardized message format (eg, 320) reserved for the timestamp.

[0061] The message sizes and formats illustrated herein are exemplary only and are not intended to be limiting. In various embodiments, different message sizes, field sizes, and / or field locations may be used.

[0062] In this example, the sizes of the three exemplary message types (300, 320, and 340) are carefully selected so that they can be stored in a capture memory (e.g., Figure 2 In 210), messages may be evenly staggered on message boundaries to prevent partial messages from overlapping. The following figure shows an example of this.

[0063] Figure 4 is a diagram illustrating two embodiments of aggregated and time-stamped message flows in a message capture memory. In this figure, a first memory segment (400) and a second memory segment (420) illustrate two examples of aggregated and time-stamped message flows stored in a message capture memory. For example, these two memory segments (400 and 420) illustrate Figure 2 An example of a memory (210) in FIG.

[0064] In the first memory segment (400), a 32-bit timestamp message (402) occurs first, occupying half of a 64-bit memory slot; the remainder of the slot is unused. In this example, each slot of the memory is 64 bits, and the timestamp message (402) has Figure 3 The example sizes and formats shown in .

[0065] Next, three 64-bit event messages (404) each occupy one memory slot, such that the second, third, and fourth slots are occupied by event messages.

[0066] Then, a sequence of 12 time-based messages (406) is stored. Each time-based message (406) is 16 bits long, so the fifth, sixth, and seventh slots each have four time-based messages. In this example, the time-based messages are used to capture the state or value of the finite state machine when the finite state machine is not in the idle state (e.g., from the time it leaves the idle state until the time it returns to the idle state). As such, in this example, the term "state capture" is used to describe the time-based messages (406), but in other embodiments, time-based messages are used to capture or record other types of information in the functional module.

[0067] The last slot is occupied by an event-based message (408).

[0068] In this example, each event-based message (e.g., 404) occupies a single slot, while time-based messages (e.g., 406) are written to four slots. The advantage of maintaining the sizes and (slot) offsets shown here is that less overhead information needs to be saved, which would be necessary if (as an example) an event message could start at any offset within a slot instead of zero offset. Similarly, if the message sizes are not multiples of each other, and the messages are not aligned with the slots shown here, the random mixing of the two messages will make it very difficult to distinguish message boundaries when the old message is overwritten with the new message, thereby partially creating messages. Partial messages may lack type identifiers (to distinguish message types) and / or lack message content (making it difficult to identify the end of the message).

[0069] Note that the various event messages (404 and 408) and state capture (ie, time-based) messages (406) may come from various functional modules, and not necessarily from a single functional module. Rather, they are aggregated at the aggregation module in the order in which they are received.

[0070] A disadvantage of the embodiment that generates the first memory segment (400) shown is that supporting multiple message types (each with a different message size) adds complexity to the aggregation module. For example, to ensure that the last event message (408) begins at the beginning of the last slot, the aggregation module must keep track of the number of previous state capture (i.e., time-based) messages (406) so that if that number is not a multiple of four, one or more unused fourth slots are inserted before the last event message (408). In some applications, a less complex implementation is desired, particularly if a debugging system is being implemented for the first time. The following figure shows a less complex embodiment in which only event messages are stored.

[0071] The second memory segment (420) shows an example in which only event-based messages (422) are generated and stored. That is, in this example, neither time-stamp messages nor state-capture (i.e., time-based) messages are generated (e.g., by a reporting module or an aggregation module).

[0072] One benefit of an embodiment in which only event-based messages are generated and stored (as shown in the second memory segment (420)) is that it is much simpler for the aggregation module to store messages because only one type (and therefore size) of messages is supported. The aggregation module does not need to keep track of the number of timestamped messages or state-capturing (i.e., time-based) messages and insert an unused fourth or half slot where needed. It is also more memory efficient because there is no unused portion, and the total amount of memory used is less than the first memory configuration because there tend to be many state-capturing (i.e., time-based) messages, while event-based messages tend to be more sparse. For these reasons, in some embodiments, only event-based messages are generated and stored.

[0073] Another way to save memory is to have configurable or selectable (e.g. event) reporting. The following figure shows an example of this.

[0074] Figure 5 is a diagram illustrating an embodiment of an optional event group in a flash storage controller application. In this example, the SOC system is a flash storage controller (not shown) that writes to and reads flash storage media. For example, in Figure 2 In some embodiments, the SOC system (200) is a flash storage controller and the (external) storage medium (218) is a flash storage medium controlled by the flash storage controller. In some embodiments, the exemplary tables (500 and 540) are included in a reporting rule set (see, e.g. Figure 1 100 and 102 in ) and / or in a reporting module (see e.g. Figure 2 The reporting module (204) in FIG.

[0075] In this example, the flash storage controller with the table shown has hardware functional modules (e.g., Figure 2 202 in ), which includes an NVM Express (NVMe) processor, a NAND flash controller, and an error correction decoder (e.g., to correct errors introduced when data is stored for a long time and / or stored in a "leaky" flash storage medium). The hardware function module typically has one or more data processing units of 4K bytes, 8K bytes, etc. associated with each host command.

[0076] Firmware functional modules in an exemplary flash storage controller (e.g., Figure 2202) includes host management firmware, a flash translation layer (FTL), and a back-end module (eg, associated with multiple NAND flash interface controllers and / or a back-end command parser), all running on one or more CPUs.

[0077] For both hardware and firmware functional modules, important events that indicate successful or failed completion of data operations include the start of data processing, the end of data processing, and any unexpected error events, such as CRC errors or certain index mismatches. Capturing these events is highly valuable for debugging system problems or understanding the internal behavior of the system.

[0078] However, depending on the application, in some cases a more "memory-efficient" or sparse approach is desired, where less information is generated and stored, while in other applications an approach that consumes much more memory is acceptable. For example, in the absence of known defects, the former approach may be desirable, but if a defect is known to exist and rapid identification of a fix is ​​critical, the latter approach may be preferred. To this end, in this example, a given reporting module has one or more groups, where each group includes one or more events, and based on the selected group, corresponding events are reported (if selected).

[0079] The first table (500) shows the groups and their events defined for the NVMe processor. Group 0 (502) includes three events: a write transfer end event (504) with event ID 0, a read transfer end event (506) with event ID 1, and an NVMe command start event (508) with event ID 2.

[0080] Group 1 (510) in the NVMe processor table (500) includes a single event: an NVMe error event (512) with event ID 7.

[0081] Group 2 (514) in the NVMe processor table (500) includes three events: a write transfer command start (516) with event ID 3, a read transfer command start (518) with event ID 4, and an NVMe command end event (520) with event ID 5.

[0082] The NVMe processor has many events of interest, and therefore in this table (500), some groups have multiple events (e.g., group 0 (502) and group 2 (514)). In contrast, the error correction decoder has fewer events of interest, and therefore each group contains only a single event. In the error correction decoder table (530), group 0 (532) includes a decoder end event (534), which has event ID 0 and is associated with the end of writing decoded data to some destination. Group 1 (536) in the error correction decoder table (530) includes a decoder error event (538) with event ID 7. Group 2 (540) in the error correction decoder table (530) includes a decoder start event (542), which has event ID 1 and is associated with the start of reading encoded data from some source.

[0083] In this example, all error events (eg, NVMe error (512) and decoder error (538)) have an event ID of 7 (ie, all ones), making it possible to quickly and / or more easily locate the error.

[0084] The example also organizes the groups according to (e.g., typical or expected) importance or interest. For example, events in group 0 (e.g., 502 and 532) are considered to be the most important or interesting events (e.g., for debugging and / or analysis purposes), those events in group 1 (e.g., 510 and 536) are (slightly) less important and / or less interesting, and those events in group 2 (e.g., 514 and 540) are the least important and / or less interesting.

[0085] In this example, each group has a corresponding select bit in the select column (522 and 544) so ​​that each group can be selected independently of all other groups - even groups in other functional groups. In one example, if a very sparse reporting method is desired, only group 0 (502 and 532) is selected. If more information is desired, then groups 0 and 1 (502, 510, 532, and 536) are selected. If it is important to have all the information available, then all groups (502, 510, 514, 532, 536, and 540) are selected. This is just an example, and with each group (at least in this example) having a corresponding select bit, any combination of groups can be selected.

[0086] Below is an example of an event-based message generated by the NVMe processor functional module when selecting all groups (502, 510, and 514) in the table (500). T1, T2, T3, etc. are incremented in chronological order. Tn can be any time after T1.

[0087] First, a write transfer command start event message corresponding to a write transfer command start event is generated (516). The message includes a functional module identifier identifying the NVMe processor, a subtype and / or event ID 3 (corresponding to the event that caused the message to be generated), a payload having a logical block address number from the host (e.g., associated with a write), and a timestamp = T1.

[0088] Then, a write transfer end event message is generated, corresponding to the write transfer end event (504). The message includes a functional module identifier identifying the NVMe processor, a subtype and / or event ID of 0 (corresponding to the event that caused the message to be generated), a payload with a logical block address number from the host (e.g., associated with the write transfer), and a timestamp of T2.

[0089] The last message in this example is an error event message, corresponding to an NVMe error event (512). The message includes a functional module identifier identifying the NVMe processor, a subtype and / or event ID of 7 (corresponding to the event that caused the message to be generated), a payload with a link or address to the command where the error occurred, and a timestamp of T3.

[0090] In one example of a problem that can be debugged using the techniques described herein, assume that a hardware functional module is defective and, in some abnormal circumstances (e.g., caused by very rare signal interactions), the module does not send a completion status for a data operation. When this happens, system resources (such as buffer slots) may be unnecessarily occupied and not released (e.g., because it is waiting for a completed status). At the system level, this manifests as a long system stop or hang after the hardware functional module discards the event. However, using the reporting techniques described herein permits reporting and / or generating two event-based messages associated with the hardware functional module in question: a start data transfer event message and an end data transfer event message for each data transfer operation. The exception is an erroneous transaction in which an end data transfer event message is not generated. By parsing the generated and saved aggregated and time-stamped streams, SOC developers are able to identify this discrepancy and the root cause (i.e., a specific hardware functional module started the data transfer, but did not complete the data transfer (or at least did not correctly signal the completion of the data transfer). Without the techniques described herein, the ability to identify the root cause and fix the defect would take much longer and could be imagined to never be found.

[0091] The following diagram describes the above example more generally and / or formally in a flow chart (e.g., with optional groups of one or more events).

[0092] Figure 6is a flow chart of an embodiment of a process for sending first status information in a standardized message format. In some embodiments, Figure 6 The process used to Figure 1 Step 100 in . Figure 1 Steps 100 and 102 in are similar (and in some cases identical), and thus Figure 6 The techniques described in can be applied to Figure 1 At step 102 in the method, the second status information is sent in a standardized message format. Figure 6 The process can be composed of functional modules (for example, Figure 2 202) in the reporting module (for example, Figure 2 204) to execute.

[0093] At 600, a first set of one or more reporting rules is retrieved, wherein the first set of one or more reporting rules includes: a first group of one or more events and a corresponding first selection bit associated with selecting the first group of one or more events; and a second group of one or more events and a corresponding second selection bit associated with selecting the second group of one or more events. For example, Figure 5 Some examples of groups ( 502 , 510 , 514 , 532 , 536 , and 540 ) with one or more events ( 504 , 506 , 508 , 512 , 516 , 518 , 520 , 534 , 538 , and 542 ) are shown.

[0094] like Figure 5 As shown in the example of , in some embodiments, the first group of one or more events includes an end operation event, and the second group of one or more events includes an error event.

[0095] Figure 5 It is also shown that in some embodiments, the first group of one or more events includes an end operation event; the second group of one or more events includes an error event; and the first set of one or more reporting rules further includes a third group of one or more events including a start operation event.

[0096] At 602, it is determined whether a first select bit is asserted. In the event that it is determined that the first select bit is asserted, first status information based at least in part on a first group of one or more events is sent in a standardized message format at 604. In the event that it is determined that the first select bit is not asserted, the first status information is sent in a standardized message format independent of the first group of one or more events at 606. In other words, the reporting module does not examine events in the unselected first group, and those events are not reported.

[0097] At 608, it is determined whether the second select bit is asserted. In the event that it is determined that the second select bit is asserted, first status information is sent in a standardized message format based at least in part on the second group of one or more events, at 610. In the event that it is determined that the second select bit is not asserted, sending the first status information in a standardized message format occurs independently of the second group of one or more events, at 612. That is, events in the second group are not reported because the group is not selected.

[0098] Although the foregoing embodiments have been described in detail for the purpose of clear understanding, the invention is not limited to the details provided. There are many alternative ways to implement the invention. The disclosed embodiments are illustrative rather than restrictive.

Claims

1. A system on chip SOC, comprising: A plurality of functional modules, including a first functional module and a second functional module, wherein: The first functional module is configured to send first status information associated with the first functional module to the aggregation module according to a first set of one or more reporting rules and in a standardized message format; and The second functional module is configured to send second status information associated with the second functional module to the aggregation module according to a second set of one or more reporting rules and in a standardized message format; The aggregation module is configured to: aggregating first status information in a standardized message format and second status information in a standardized message format; and inserting the timestamps to obtain a timestamped and aggregated message stream, wherein the timestamped and aggregated message stream enables a visualization system to analyze a first status message associated with the first functional module and a second status message associated with the second functional module using a common timeline; and A storage device configured to store the timestamped and aggregated message streams.

2. The SOC according to claim 1, wherein: The first set of one or more reporting rules includes event-based reporting rules; as well as Sending the first status information in a standardized message format includes: determining whether the first functional module satisfies event-based reporting rules; and In response to determining that the first functional module satisfies the event-based reporting rule, status information specified by the event-based reporting rule is sent to the aggregation module in a standardized message format.

3. The SOC according to claim 1, wherein: The first set of one or more reporting rules includes a time-based reporting rule; as well as Sending the first status information in a standardized message format includes: determining whether the first functional module satisfies a time-based reporting rule; and When the first functional module satisfies the time-based reporting rule, the first functional module periodically sends status information specified by the time-based reporting rule to the aggregation module in a standardized message format. The SOC of claim 1 , wherein time stamping comprises inserting a time stamp message into an aggregate message stream.

5. The SOC of claim 1, wherein timestamping comprises writing the timestamp into a reserved field in a standardized message format reserved for timestamps.

6. The SOC of claim 1, wherein: The SOC includes a flash storage controller; as well as The plurality of functional modules include one or more of the following: an NVM Express (NVMe) processor, a NAND flash controller, an error correction decoder, a host management firmware, a flash translation layer (FTL) module, or a back-end module.

7. The SOC of claim 1 , wherein sending the first status information in a standardized message format comprises: Retrieving a first set of one or more reporting rules, wherein the first set of one or more reporting rules comprises: a first group of one or more events and a corresponding first selection bit associated with selecting the first group of one or more events; and a second group of one or more events and a corresponding second selection bit associated with selecting the second group of one or more events; determining whether the first selection bit is asserted; In the event of determining that the first select bit is asserted, sending first status information in a standardized message format based at least in part on a first group of one or more events; In the event that it is determined that the first select bit is not asserted, sending the first status information in the standardized message format occurs independently of the first group of one or more events; determining whether the second select bit is asserted; In the event that a second select bit is determined to be asserted, sending first status information in a standardized message format based at least in part on a second set of one or more events; and In the event that it is determined that the second select bit is not asserted, sending the first status information in the standardized message format occurs independently of the second set of one or more events.

8. The SOC according to claim 7, wherein: The first group of one or more events includes an end operation event; and The second group of one or more events includes error events.

9. The SOC according to claim 7, wherein: The first group of one or more events includes an end operation event; a second set of one or more events comprising error events; and The first set of one or more reporting rules further includes a third group of one or more events including a start operation event.

10. A method comprising: A plurality of functional modules are provided, including a first functional module and a second functional module, wherein: The first functional module is configured to send first status information associated with the first functional module to the aggregation module according to a first set of one or more reporting rules and in a standardized message format; and The second functional module is configured to send second status information associated with the second functional module to the aggregation module according to a second set of one or more reporting rules and in a standardized message format; An aggregation module is provided, the aggregation module being configured to: aggregating first status information in a standardized message format and second status information in a standardized message format; and inserting the timestamps to obtain a timestamped and aggregated message stream, wherein the timestamped and aggregated message stream enables a visualization system to analyze a first status message associated with the first functional module and a second status message associated with the second functional module using a common timeline; and A memory is provided, the memory being configured to store the time-stamped and aggregated message stream.

11. The method according to claim 10, wherein: The first set of one or more reporting rules includes event-based reporting rules; and Sending the first status information in a standardized message format includes: determining whether the first functional module satisfies event-based reporting rules; and In response to determining that the first functional module satisfies the event-based reporting rule, status information specified by the event-based reporting rule is sent to the aggregation module in a standardized message format.

12. The method according to claim 10, wherein: The first set of one or more reporting rules includes a time-based reporting rule; as well as Sending the first status information in a standardized message format includes: determining whether the first functional module satisfies a time-based reporting rule; and When the first functional module satisfies the time-based reporting rule, the first functional module periodically sends status information specified by the time-based reporting rule to the aggregation module in a standardized message format.

13. The method of claim 10, wherein time stamping comprises inserting a time stamp message into an aggregate message stream.

14. The method of claim 10, wherein timestamping comprises writing the timestamp into a reserved field in a standardized message format reserved for timestamps.

15. The method of claim 10, wherein: The SOC includes a flash storage controller; as well as The plurality of functional modules include one or more of the following: an NVM Express (NVMe) processor, a NAND flash controller, an error correction decoder, a host management firmware, a flash translation layer (FTL) module, or a back-end module.

16. The method of claim 10, wherein sending the first status information in a standardized message format comprises: Retrieving a first set of one or more reporting rules, wherein the first set of one or more reporting rules comprises: a first group of one or more events and a corresponding first selection bit associated with selecting the first group of one or more events; and a second group of one or more events and a corresponding second selection bit associated with selecting the second group of one or more events; determining whether the first selection bit is asserted; In the event of determining that the first select bit is asserted, sending first status information in a standardized message format based at least in part on a first group of one or more events; In the event that it is determined that the first select bit is not asserted, sending the first status information in the standardized message format occurs independently of the first group of one or more events; determining whether the second select bit is asserted; In the event that a second select bit is determined to be asserted, sending first status information in a standardized message format based at least in part on a second set of one or more events; and In the event that it is determined that the second select bit is not asserted, sending the first status information in the standardized message format occurs independently of the second set of one or more events.

17. The method of claim 16, wherein: The first group of one or more events includes an end operation event; and The second group of one or more events includes error events.

18. The method of claim 16, wherein: The first group of one or more events includes an end operation event; a second set of one or more events comprising error events; and The first set of one or more reporting rules further includes a third group of one or more events including a start operation event.

Images