Multi-dimensional usage space testing of memory components

Abstract

A target vector is generated representing a usage parameter corresponding to a test of a memory component. A test sample is assigned to the target vector and a set of path variables is generated for the test sample. A test procedure of the test is performed according to the set of path variables using the test sample to generate a test result. A failure associated with the test result is identified.

Description

Technical Field

[0001] The present disclosure relates generally to memory subsystems and, more particularly, to testing of memory components for memory subsystems. Background Art

[0002] The memory subsystem may be a storage system, such as a solid state drive (SSD) or a hard disk drive (HDD). The memory subsystem may be a memory module, such as a dual in-line memory module (DIMM), a small outline DIMM (SO-DIMM), or a non-volatile dual in-line memory module (NVDIMM). The memory subsystem may include one or more memory components that store data. The memory components may be, for example, non-volatile memory components and volatile memory components. In general, the host system may utilize the memory subsystem to store data at the memory components and retrieve data from the memory components. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The present disclosure will be more fully understood from the detailed description given below and the accompanying drawings of various embodiments of the present disclosure.

[0004] Figure 1 An example environment is described for allocating test resources to perform testing of memory components according to some embodiments of the present disclosure.

[0005] Figure 2 is a flow chart of an example method of allocating test resources for performing testing of a memory component according to some embodiments of the present disclosure.

[0006] Figure 3 An example interface associated with testing of a configuration memory component according to some embodiments of the present disclosure is described.

[0007] Figure 4 An example test platform that performs testing of memory components according to some embodiments of the present disclosure is described.

[0008] Figure 5 is a flow chart of an example method of repeating a failure test of a memory component according to some embodiments.

[0009] Figure 6 is a flow chart of an example method for deploying test samples in a usage space associated with memory component testing.

[0010] Figures 7A-7C is an example representation produced in performing a multi-dimensional usage space test of a memory component according to some embodiments.

[0011] Figure 8 is a block diagram of an example computer system in which embodiments of the present disclosure may operate. DETAILED DESCRIPTION

[0012] Aspects of the present disclosure are directed to multi-dimensional usage space testing of memory components. A memory subsystem is also referred to below as a "memory device". An example of a memory subsystem is a storage device coupled to a central processing unit (CPU) via a peripheral interconnect (e.g., an input / output bus, a storage area network). Examples of storage devices include solid-state drives (SSDs), flash drives, universal serial bus (USB) flash drives, and hard disk drives (HDDs). Another example of a memory subsystem is a memory module connected to a CPU via a memory bus. Examples of memory modules include dual in-line memory modules (DIMMs), small form factor DIMMs (SO-DIMMs), non-volatile dual in-line memory modules (NVDIMMs), and the like. In some embodiments, a memory subsystem may be a hybrid memory / storage device subsystem. In general, a host system may utilize a memory subsystem that includes one or more memory components. The host system may provide data to be stored at the memory subsystem, and may request to retrieve data from the memory subsystem.

[0013] Memory components used in a memory subsystem may be tested before being utilized in the memory subsystem. In a conventional test process, the memory components may be placed into a chamber (i.e., an oven) to test the memory components under various conditions (e.g., temperature conditions). For example, a single chamber may be used to test multiple memory components at a specific temperature at one time. The test process may indicate that various operations are to be performed at the memory component at a specific temperature. Such operations may include, but are not limited to, read operations, write operations, erase operations, fast raw bit error rate measurements, conditional threshold voltage (Vt) distribution collection, thermal cycle control, asynchronous power-off control, open NAND flash interface (ONFI) testing, and the like. The performance and behavior of the memory components may be observed while the test process is being performed. For example, performance characteristics (e.g., read or write latency) and reliability of data stored at the memory component may be measured and recorded during the test process. However, since a chamber can only apply a limited set of test conditions (e.g., a single temperature) to a memory component at any particular time, testing of a memory component under multiple different test conditions (e.g., different temperatures) requires a significant amount of time due to the need to perform a test process for each of the one or more desired test conditions. Additionally, a chamber can only perform a single test process at a time. Thus, if a test process for a memory component requires multiple different conditions, performing multiple different tests on the memory component under different operating conditions (e.g., different temperatures) can take a significant amount of time.

[0014] Aspects of the present disclosure solve the above and other defects by allocating test components to perform test processes. A distributed test platform may include multiple test resources. Each test component (e.g., test resource) may be a test socket including a memory component. The test socket may additionally include a temperature control component for applying a specific temperature condition to the memory component included in the test socket. The test platform may additionally include multiple test boards, each of which includes one or more of a test socket to enable the use of multiple geographically dispersed test resource locations to perform tests under multiple conditions. The test board may be organized into groups or racks, and multiple racks may be at a specific location or site. Thus, multiple sites may include racks of test boards, including multiple test components (e.g., test sockets and memory components). The racks of the test platform may be coupled to a centralized resource allocator (e.g., one or more servers or other such computing devices) via a network, and the resource allocator is used to control, manage and facilitate the execution of tests at each test component of the test platform across different test boards, test racks and sites or locations.

[0015] The test components or test resources of the test platform can be used to perform multiple different test processes on the test platform at the same time. For example, a first subset of test sockets can be used to perform a first test, while a second subset of test sockets of the test platform can be used to perform a second test in parallel. In this way, the resource allocator can allocate or assign test sockets to be used for new tests of memory components in view of the screening information. In some embodiments, the screening information can include one or more elements of characteristic information and usage information. Example characteristic information can include one or more of product type, probe revision, category revision, version information (i.e., design or manufacturing revision) of the memory component, etc. Example usage information associated with the component to be tested can include a minimum block cycle count, a maximum block cycle count, a desired number of blocks or range, etc.

[0016] The new test may specify a specific characteristic or type of memory component to be used for the test and a specific temperature that the temperature control component will apply to the memory component at the test socket. Such characteristics may include, but are not limited to, the type of memory component (e.g., design version) and the usage characteristics of the memory component. The resource allocator may then assign the available test sockets of the test platform to the new test based on the characteristics specified by the new test and additional information of the test platform. For example, the available test sockets may be further assigned based on the locations of the test sockets across various test racks or sites of the test platform and the temperature control component may be used to apply the temperature specified by the test. Thus, the test platform may be a distributed set of test sockets across different locations or sites. Different groups of test sockets of the test platform may be assigned to different tests based on the specific characteristics of the test sockets specified by different tests. In some embodiments, the test components or test resources of the test platform may be used to perform the same test process under different test conditions at the same time.

[0017] In one embodiment, the one or more test processes may identify one or more failures, including a failure in the operation of a memory component (e.g., a problem with the operation of the memory component), a failure in the operation of a memory component causing the corresponding software layer to respond to the failure (e.g., raise an exception), a failure due to the operation of a memory component generated by the test program (e.g., an exception), or a software, hardware, or firmware bug or error in the test process itself.

[0018] Advantages of the present disclosure include, but are not limited to, reducing the amount of time a test platform takes to perform testing of memory components. In addition, multiple different tests can be performed at multiple distributed test resources to test multiple different conditions (e.g., different temperatures). In addition, multiple different sequences of operations can be performed using a distributed system to achieve more robust testing of memory components and improved reliability and performance. In this way, since many different tests of memory components can be performed more quickly, the reliability of memory components can also be improved because any potential defects or flaws can be identified and later addressed in the design or manufacture of memory components to be included in the memory subsystem.

[0019] Figure 1 An example environment for allocating test resources to perform testing of memory components according to some embodiments of the present disclosure is described. Test resources 100 may include one or more racks 110A, 110B, and 110N. Each of racks 110A, 110B, and 110N may include multiple test boards 120, wherein each test board 120 includes one or more test sockets (i.e., test resources). Test resources 100 may include any number of racks or test sockets.

[0020] As shown, the test board 120 may include one or more test sockets. For example, the test board 120 may include a first test socket 121, a second test socket 122, and a third test socket 123. Although three test sockets are shown, the test board 120 may include any number of test sockets. Each test socket may include a memory component that has been embedded in a respective test socket. In addition, each test socket may include a temperature control component for applying a temperature condition to the embedded memory component. In some embodiments, the temperature control component may be a dual Peltier device (e.g., two Peltier devices) that utilizes the Peltier effect to apply a heating or cooling effect at the surface of the dual Peltier device coupled to the embedded memory component. In the same or alternative embodiments, the temperature control component may be placed on top of the memory component in the respective test socket.

[0021] As shown, each test rack 110A, 110B, and 110N may include a plurality of test boards 120. Each of the test boards 120 of a particular test rack may be coupled with a local test component. For example, each test rack 110A, 110B, and 110N may include local test components 111A, 111B, and 111N, respectively. Each of the local test components 111A, 111B, and 111N may receive instructions to execute a test or a portion of a test to be executed at a test socket of the corresponding test rack. For example, a resource allocator component 130 may receive (e.g., from a user) conditions for the test to be executed, and the resource allocator component 130 may determine a specific test socket across different test boards 120 at one or more of the test racks 110A, 110B, and 110N that may be used by the test. In some embodiments, the resource allocator component 130 may be provided by a server 131. In some embodiments, server 131 is a computing device or system coupled to local test components 111A, 111B, and 111N over a network.

[0022] The temperature control assembly of each test socket 121, 122, and 123 of each test board 120 can be used to apply different temperature conditions to respective embedded memory components. In addition, each test socket 121, 122, and 123 can be used to perform different operations at the embedded memory components.

[0023] Resource allocator component 130 may receive test input from a user. In some embodiments, the test input may be received via an interface (e.g. Figure 3 ) receives a test input (or test request) from a user. The test input may specify conditions (also referred to as "test conditions") for a test to be performed on one or more memory components. For example, a test may specify specific temperature conditions to be applied to the memory components and a sequence of operations to be performed at the memory components under the specific temperature conditions. The resource allocator 130 may retrieve a data structure that identifies available test sockets across the test platform 100 and characteristics of the available test sockets. Subsequently, the resource allocator component 130 may assign a test socket at the test platform 100 that includes an embedded memory component that matches or satisfies the conditions of the test. In some embodiments, the resource allocator component 130 may identify a test socket that matches the test conditions by analyzing characteristic information and usage information associated with the available test sockets. In one embodiment, with respect to a rack (e.g., Figure 1The characteristic information for each test socket of the racks 110A, 110B, 110N of the test racks may be stored in one or more databases (referred to as "characteristic databases"). In one embodiment, usage information for each test socket of the racks may be stored in one or more databases (referred to as "usage databases"). The resource allocator component 130 may then transmit instructions to the local test components of the test racks including the test sockets that are identified for use in testing based on the associated characteristic and usage information. Additional details regarding the resource allocator component 130 are described below.

[0024] Figure 2 An example method 200 for allocating test resources for performing tests of memory components according to some embodiments of the present disclosure is described. The method 200 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, dedicated logic, programmable logic, microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, the method 200 is performed by Figure 1 The resource allocator component 130 of the embodiment of the present invention is executed. Although shown in a specific order or sequence, unless otherwise specified, the order of the process can be modified. Therefore, it should be understood that the illustrated embodiment is only an example, and the illustrated process can be performed in a different order, and some processes can be performed in parallel. In addition, one or more processes can be omitted in various embodiments. Therefore, not all processes are required in each embodiment. Other process flows are also possible.

[0025] As shown, at operation 210, processing logic receives a request to perform a test on one or more memory components. For example, a user may provide a request to run a test of a memory component through a test platform. The test may specify an operation to be performed by the memory component. Such operations may include, but are not limited to, a read operation, a write operation, and an erase operation. In some embodiments, the test specifies a sequence of multiple operations. Thus, a user may provide information specifying the conditions under which the test of the memory component will be performed at the test platform.

[0026] The test may specify operating or test conditions under which the operation is to be performed at a particular memory component. When the operation is performed at the memory component, the test condition may be a temperature condition of the memory component. In some embodiments, as described below, the test request may specify the test condition or may be used to identify the test condition corresponding to the desired characteristics (e.g., component type) and usage (e.g., historical usage data) of the memory component to be assigned for testing.

[0027] At operation 220, processing logic determines filtering information (e.g., one or more elements of characteristic information and usage information) associated with memory components to be assigned for testing. In some embodiments, the test conditions may be used as filtering information to filter and identify memory components having corresponding characteristic information and / or usage information. In some embodiments, each test platform (e.g., a rack of boards) may be associated with one or more databases (e.g., Figure 4 The characteristic databases 416-1 and 416-N) are associated.

[0028] In some embodiments, the usage information of the memory component may include, but is not limited to, the number of operations that have been performed by the memory component. For example, the test conditions may be used to identify the desired usage information corresponding to a memory component having a greater or lesser number of program-erase operations or cycles and read operations than a specific number of program-erase operations or cycles and a specific number of read operations. In some embodiments, the desired usage information may include previous temperature conditions that have been applied to the memory component. For example, the usage information may include the previous temperatures at which operations have been performed at the memory component for previous tests (i.e., a temperature profile of the memory component), which may be specified.

[0029] The test request may specify test conditions for the test or may be used to identify test conditions that correspond to desired screening information for the memory component to be used in the test (e.g., test components that match criteria associated with the identified characteristic information and usage information). In some embodiments, the available test sockets may be test sockets that are not currently used by another test and that match the desired characteristics and / or usage information of the memory component specified by the test. Thus, a first subset of the test sockets (i.e., test resources) of the test platform may be available test sockets that match the conditions specified by the test and a second subset of the test sockets may be test sockets that are unavailable and / or do not match the conditions specified by the test. In combination Figure 3 Describes additional details relative to matching desired characteristics.

[0030] At operation 230, the processing logic reserves one or more test sockets in view of the identified characteristic information and usage information associated with the test. In some embodiments, the processing logic determines the available test sockets of one or more test platforms (e.g., racks). For example, a test socket at a test board of the test platform that is not currently used to perform another test or operation and is not scheduled to perform another test can be identified as available. For example, the test platform may include several test sockets and a subset of the test sockets may be currently used by another test of the memory component, while a second subset of the test sockets is not currently being used by another test of the memory component. Therefore, the second subset of the test sockets may be considered as available test sockets. In some embodiments, the test may specify a time period in which the test is to be completed. The available test sockets in this case may be test sockets that can be used for testing within the time period. In this way, if a specific test socket is currently being used by a test and will then be scheduled as idle or available and the test can perform an operation at a specific test socket within a specified time period, then the specific test socket may be considered as an available test socket.

[0031] The processing logic may identify a set of one or more test sockets that are available and match the characteristic information and usage information associated with the test. In some embodiments, the processing logic filters the available test resources (e.g., available boards or available test sockets) in view of the characteristic information and the usage information. For example, each rack at each location may maintain a local usage database (e.g., Figure 4 of the usage databases 414-1 and 414-N) and local property databases (e.g., Figure 4 The characteristic databases 416-1 and 416-N) are used to store usage and characteristic information about the corresponding test resources for use in filtering the available test resources according to the test conditions associated with the test.

[0032] At operation 240, processing logic configures and executes tests for one or more memory components at available test sockets reserved based on characteristic information and usage information of the one or more memory components. For example, the test may be performed at a test socket that is not currently used by another test and has characteristics that match the desired characteristics specified by the test. The test may be performed at different test sockets included in different test racks across the test platform. The execution of the test may apply different temperature conditions at different test sockets while performing different or identical sequences of operations at the test sockets. For example, a first socket may run a sequence of operations at a first temperature and a second test socket may run the same sequence of operations at a different second temperature. In some embodiments, a first test socket may run a first sequence of operations at a temperature and a second test socket may run a different second sequence of operations at the same temperature. In the same or alternative embodiments, the temperature conditions applied to a particular test socket may change during the execution of the sequence of operations.

[0033] In some embodiments, to configure the test for execution, processing logic generates one or more source code dependency lists associated with the test. In some embodiments, processing logic identifies one or more file locations associated with the source code dependencies (e.g., corresponding folders or uniform resource locators (URLs) associated with a code repository). In some embodiments, processing logic identifies one or more command lines (i.e., run properties) to be run in conjunction with the code associated with the test. In some embodiments, processing logic may identify a relative folder location where running the one or more command lines begins. In some embodiments, processing logic may identify a relative folder location where output associated with the test (herein, "test result output") is generated.

[0034] According to some embodiments, a device communicatively coupled to Figure 1 In some embodiments, the user can request, initiate and manage the resource allocator 130 through the user interface of the resource allocator 130. Figure 2 The operation provides information that will be used to generate the test request (for example, according to Figure 2 Request for operation 210).

[0035] Figure 3 An example interface 300 is illustrated that is configured to collect various information from a user in conjunction with a test request. For example, after identifying and reserving a plurality of memory component boards in conjunction with a test, the interface 300 (eg, a network-based interface, such as Figure 4 The network interface 422 of the example program allows the test to be started by identifying the location of the source code and the command line to run the source code (e.g., locally in a folder or in a code repository). In some embodiments, one or more additional parameters may be identified in conjunction with the test execution, such as the relative location of the folder in which the main function is located, the command line to run the main function, and the relative output folder location.

[0036] In some embodiments, interface 300 may provide filters for identifying characteristic information associated with the component to be tested, such as product type, probe revision, and class revision. Interface 300 may also provide filters for identifying usage information associated with the component to be tested, such as minimum block cycle count, maximum block cycle count, desired number of blocks or range, etc.

[0037] In some embodiments, interface 300 may also be used to enable a user to configure and initiate testing by identifying source code and the location of command lines that run the source code. Figure 3 As shown, additional parameters associated with the test may be identified, such as the location of the folder in which the main function resides (eg, a run folder), the command line to run the main function, and the relative output folder location.

[0038] In some embodiments, a client device (e.g., Figure 4 The client 420) interfaces 300 with the resource allocator (e.g., Figure 4 ) to replicate code to one or more racks (e.g., Figure 4 In some embodiments, the execution of the identified code for testing causes one or more test processes (e.g., Figure 4 When a test is completed (e.g., successfully completed or failed), the results are repeated to a network drive from which the test results can be collected. In some embodiments, the test resources reserved and used in conjunction with the completed test can be released after the test is completed. In some embodiments, when collecting test result data, the processing logic can store the data in a folder (e.g., Figure 3 The " / client_output_name" folder identified in the "Find data at:" field of interface 300) or other storage location accessible to the user.

[0039] In some embodiments, during the testing process, the client device may not be aware of the location of the physical test components (e.g., physical boards, sockets, or memory components). In some embodiments, code associated with the test is run on one or more geographically dispersed workstations (e.g., Linux workstations) associated with the board containing the reserved test components. In some embodiments, the resource allocator may optimize the location selection of the test components to manage (e.g., reduce) data transfer time.

[0040] Figure 4 An example environment is described that includes a management module 480 communicatively connected to a client 420 requesting to perform a test according to an embodiment of the present disclosure. In some embodiments, the management module 480 coordinates with the resource allocator 130 to perform a test using test components (e.g., test components 411-A and 411-N) that are physically located in one or more geographically dispersed component racks (e.g., rack A 110A, rack B 110B, rack C 110C). As described above with respect to method 200, the client 420 may submit a test request to the resource allocator 130 via a network interface 422 of the client 420. The resource allocator 130 identifies and reserves test components for performing a test process. In some embodiments, the reserved test components 411 may be located in geographically dispersed racks 110. For example, the requested test may include executing a set of test processes, including process 1, process 2...process N and process X, process Y...and process Z. In Figure 4 In the example shown, the resource allocator 130 may identify the test components 411-A of the rack A 110A as being used to perform process 1, process 2, ... process N. In addition, the resource allocator 130 may identify the test components 411-N of the rack N 110N as being used to perform process X, process Y, ... process Z. In this example, the rack A 110A and the rack N 110N are in different geographic locations. In some embodiments, the resource allocator 130 may analyze the usage information stored in the usage database 440 and the characteristic information stored in the communicatively connected characteristic database 450 to identify one or more test components that meet the test conditions associated with the requested test.

[0041] In some embodiments, resource allocator 130 communicates with a resource program (e.g., resource program 412-A, 412-N), such as a daemon, to transmit information associated with the requested test (e.g., identifying source code, command line, etc.). Figure 4 As shown in , each resource program of each rack workstation may be associated with a usage database (e.g., usage database 414-A, 414-N) and a characteristic database (e.g., characteristic database 416-A, 416-N) for storing usage and characteristic information associated with the test components 411-A, 411-N of the local rack. For example, the usage database 414-A stores usage information associated with the test component 411-A of the rack A 110A, and the characteristic database 416-A stores characteristic information associated with the test component 411-A of the rack A 110A.

[0042] The test information (e.g., test instructions) received by the resource program 412-A, 412-N is provided to the local test manager 410-A, 410-N to facilitate the execution of one or more test processes (e.g., process 1, process 2, ... process N for rack A 110A) by the identified and reserved test components 411A, 411N. In one embodiment, the test component 411 may include one or more sockets including one or more memory components, such as Figure 1 For example, the resource allocator 130 may be configured according to Figure 2 The method 200 performs operations to coordinate the execution of tests using various multiple test components 411A, 411N of multiple geographically dispersed racks 110A, 110B, 110C, ... 110N.

[0043] In some embodiments, management module 480 is communicatively connected to client 420 and is configured to execute container process 482 to manage execution of one or more test processes across multiple racks (also referred to as a rack cluster). Management module 480 can be used to enable feedback communication from each of the individual test processes (e.g., process 1, process 2 ... process N, process X, process Y ... process Z). In some embodiments, execution of container process 482 enables communication between management module 480 and client 420 via a client API (e.g., a Python API) to coordinate reservation of test components by resource allocator 130 based on filters (e.g., usage and feature filters), as described below with reference to Figure 5 Detailed description. In some embodiments, the management module 480 establishes a feedback communication link with the individual test processes by providing unique connection information (e.g., IP address 1, IP address 2, IP address 3, IP address X, IP address Y, and IP address Z) to each of the test processes. In some embodiments, unique connection information (e.g., unique IP address or other such network identification information) is provided to each of the individual processes to allow each individual process to communicate directly with the management module 480. For example, the first test process (e.g., process 1 executed on rack A 110A) can receive the connection information assigned by the master control module 480 and establish a connection to provide feedback information to the master control module 480. For example, the feedback information may include the results of the corresponding test process, conditions associated with the test (e.g., temperature conditions, characteristic information, usage information, etc.), and an enumeration of one or more events or operations performed by the test process (e.g., events or operations performed before and / or including a failure associated with the operation of the memory component (e.g., an actual operational error or an anomaly caused by noise or some other factor), or a failure associated with the test process (e.g., a portion or part of a failed test process indicates an execution problem)). In some embodiments, the management module 480 may collect feedback information from multiple different processes via corresponding unique connections and store the feedback information in the test management database 484.

[0044] Figure 5Flowchart of an example method 500 (also referred to as a "container process") for collecting feedback information from multiple test processes that perform tests associated with multiple distributed test components (e.g., test sockets containing one or more memory components) and repeating failed test processes to identify failures as errors (e.g., operational errors associated with one or more memory components) or anomalies (e.g., failures due to noise). In some embodiments, having identified a failure with respect to a test process associated with one or more memory components (i.e., a first memory component), processing logic may repeat the failed test process on one or more different memory components (also referred to as a "duplicate component"). In some embodiments, processing logic uses feedback information received from the failed test process to reproduce the conditions of the failed test process. In some embodiments, in the event that repetition of the test process would result in a failure with respect to one or more duplicate components (i.e., memory components having the same characteristics and usage information as the first memory component), processing logic identifies the failure associated with the first memory component as an actual error (e.g., a verified or confirmed error related to the operation of the memory component). In the event that repetitions of the failure test process produce a successful result (i.e., no failure is detected), the processing logic identifies the failure associated with the first memory component as an anomaly (e.g., due to noise). The method 500 may be performed by processing logic that may include hardware (e.g., a processing device, a circuit system, dedicated logic, programmable logic, microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, the method 500 is performed at least in part by Figure 4 The management module 480 of the embodiment of the present invention is executed. Although shown in a specific order or sequence, unless otherwise specified, the order of the process can be modified. Therefore, it should be understood that the illustrated embodiments are only examples, and the illustrated processes can be performed in different orders, and some processes can be performed in parallel. In addition, one or more processes can be omitted in various embodiments. Therefore, not all processes are required in each embodiment. Other process flows are also possible.

[0045] As shown, at operation 510, the processing logic of management module 480 (e.g., the processing logic of container process 482) reserves one or more test components in view of the screening information associated with the test. In some embodiments, management module 480 communicates with client 420 via an API (e.g., client API 424 of client 420). Figure 4 The resource allocator 130 communicates with the memory component to identify the test and corresponding screening information (eg, characteristic information and usage information). For example, the resource allocator receives an instruction to perform a test on a memory component.

[0046] At operation 520, processing logic generates test execution information to define one or more test processes for the test. In some embodiments, the test execution information includes, but is not limited to, a list of code dependencies associated with the test, a run property for each board corresponding to each of the reserved test components, and unique connection information (e.g., a unique IP address) for each of the individual test processes associated with the test. In an example, the management module may provide the connection information as a run type input in a corresponding command line. In some embodiments, the management module communicates text execution information including a list of each reserved host name (e.g., rack workstation) and command line for each host to a client (e.g., Figure 4 The client 420). The client may then use the test execution information to trigger or initiate one or more test processes on the reserved board and the corresponding test components.

[0047] At operation 530, processing logic receives feedback information from each of the plurality of test processes via a corresponding connection established using the connection information provided for each of the test processes. For example, the feedback information may include the number of programming and erasing cycles that the failed memory component has received, the temperature when the failure of the memory component occurred, one or more temperatures when data was written on the memory component, the state of the memory component (e.g., suspended), the time since the last memory operation, the state of the memory block (e.g., partially or fully programmed), etc.

[0048] In some embodiments, the feedback information may include information associated with the corresponding test process, such as information identifying a failure (e.g., an exception or error) associated with a process executed on one or more memory components, an enumeration of one or more events that occurred with the failure of the test process, associated characteristic information and usage information of one or more memory components, and a set of conditions associated with the execution of the test process. Example conditions associated with the test may include temperature conditions of the one or more memory components being tested, temperature over time (e.g., temperature under test per second), Vcc values, identification of suspended and restarted execution entities, etc. The management module may collect and aggregate the feedback information received from multiple test processes and store the collected feedback information in one or more databases (e.g., Figure 4 In some embodiments, the test process may identify the occurrence of an error and generate a list of events that caused or contributed to the error. The test process may set a flag indicating the error and store the flag and the list of events in a file that is provided to the management module in the form of feedback information via a unique IP address.

[0049] At operation 540, the processing logic determines the failure of the first test process and the corresponding first test information. In some embodiments, the processing logic determines the failure of the test process and the information associated with the test process (e.g., the test information) in view of the feedback information. For example, the management module may review the feedback information from the Figure 4 The management module may determine that process 1 failed during execution. The management module may further identify feedback information including first test information associated with process 1 received via IP address 1. In some embodiments, the first test information may include one or more of the following: identification of command line information associated with process 1, associated characteristic information, associated usage information, an enumeration of one or more events or operations performed in conjunction with process 1, a set of one or more conditions associated with execution of process 1, and any other information associated with execution of process 1.

[0050] At operation 550, the management module may repeat the execution of the first process on one or more repeat test components. The repeat test component may be a selected test component that matches the test component used during the failed test process. For example, the repeat test component may have the characteristics and usage information of the test component that matches the failed test process. The management module may repeat the first test process using the same or substantially similar test conditions used during the identified failure execution of the test process. In this regard, the management module may repeat all variables (e.g., test conditions, test parameters, component parameters, etc.) to perform the failed test process in a controlled and monitored environment to identify the cause or root cause of the failure. In some embodiments, the repeat test component may be located on different boards of the same rack or on different racks in different orientations. In some embodiments, the management module may use one or more boards dedicated to acting as a repeat test component. The event set identified in the feedback information can be used to repeat the error using a dedicated repeat test board.

[0051] Advantageously, feedback information returned by the corresponding test process via a direct connection established with the management module can be used to identify failures associated with one or more of the distributed test processes. The management module can repeat or regenerate specific test conditions associated with the failed test process to determine whether the failed test process is an anomaly (e.g., due to noise) or an error that requires remediation. In some embodiments, in the case where the repetition of the failed test process produces a failure with respect to one or more repeated components, the processing logic can identify the failure associated with the first memory component (e.g., one or more memory components involved in the first failed test process) as an actual error associated with the memory component. In the case where the repetition of the failed test process produces a successful result (i.e., no failure is detected), the processing logic identifies the failure associated with the first memory component as an anomaly (e.g., an initial failure due to noise), rather than an error or failure of the first memory component. In some embodiments, the management module can identify that the failed process is due to an error and can identify the root cause or cause of the error. In some embodiments, the management module can coordinate the release of the test component immediately after the test is completed.

[0052] As described above, conventional quality and reliability testing of memory components is limited to testing a limited number of usage parameters (e.g., intrinsic charge loss (ICL), read disturb loss, program disturb loss, etc.) in a single chamber under a single set of conditions (e.g., temperature conditions). As detailed above, aspects of the present disclosure address these deficiencies by employing a distributed test platform that is configured to distribute multiple test components to perform testing and collect feedback information covering a wider set of conditions in the usage space associated with the memory components, thereby reducing the gaps in conventional qualification testing.

[0053] In some embodiments, the test platform of the present disclosure can perform spatially utilized testing associated with a memory component, including test operations (eg, erase, program, and read operations) to generate multiple different test samples using multiple different test sizes using usage parameters.

[0054] A test method is provided for multi-dimensional usage space testing so as to force normally low probability events to occur at high frequencies. The test method applies analytical techniques to process a large amount of test data. Advantageously, the test platform generates a set of different usage conditions for testing memory components.

[0055] In some embodiments, the test platform identifies and extracts failures that may occur in the memory component that may be observed during write or read operations. For example, a write failure may be identified as an abnormal deviation in a failure state or programming time. In another example, a read failure may be identified as a loss of margin, an abnormally high error rate, or as unrecoverable data. The test platform may further identify the boundaries or limits of the usage space in which the memory component remains operational. For example, the test platform may determine whether the memory component is operable at a multi-dimensional usage space, such as at the following combination of usage conditions: at half of the total cycle, at the full crossover temperature, and at half of the maximum allowed read. It should be noted that any appropriate combination of conditions of the usage space may be combined to produce a multi-dimensional space (e.g., a set of usage conditions) for testing the memory component. In some embodiments, the test platform may develop a model that predicts the optimal media management recovery process for a given set of usage conditions.

[0056] In some embodiments, the test system uses multiple different sockets for multiple different tests of the usage space of the associated memory components. Figure 1-5 Described in detail, the test system collects test results using connections to multiple test processes for multiple tests, identifies failures, and determines the occurrence of errors. In one embodiment, dimensions of a usage space associated with a memory component may be identified and classified as vectors, path variables, or media management properties. In one embodiment, a vector may represent a property describing an end state or constant usage of a sample, such as total writes during the life of the memory component, retention at the end of the life of the memory component, etc. In some embodiments, a usage parameter that may be defined as a path variable includes a property or condition that continuously changes during the operation of the memory component. In one embodiment, a path variable may be a property that changes during the life of the memory component. In one embodiment, a media management property may be a property that represents or describes how to read or recover data from a test sample (e.g., default read, read entry, auto read cal, soft read, cRLC, etc.)

[0057] In some embodiments, a plurality of different tests are associated with a common purpose or target vector that is associated with a usage parameter (e.g., number of writes) of a memory component. For example, 100 memory components may be tested to generate test samples in view of a defined purpose (or target vector) (i.e., usage parameters) of performing 10,000 write operations. The steps and conditions (i.e., path variables) associated with each of the test samples used to achieve the 10,000 write operations may be different. In some embodiments, different test conditions (i.e., set of path variables) are applied to each of the respective tests to generate a distributed sampling of test samples corresponding to the target vector.

[0058] Figure 66 is a flow chart of an example method 600 (also referred to as a "container process") for spreading test samples in a usage space associated with memory component testing and reducing test gaps. The method 600 may be performed by processing logic that may include hardware (e.g., a processing device, a circuit system, dedicated logic, programmable logic, microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, the method 600 is performed at least in part by Figure 4 The management module 480 of the embodiment of the present invention is executed. Although shown in a specific order or sequence, unless otherwise specified, the order of the process can be modified. Therefore, it should be understood that the illustrated embodiments are only examples, and the illustrated processes can be performed in different orders, and some processes can be performed in parallel. In addition, one or more processes can be omitted in various embodiments. Therefore, not all processes are required in each embodiment. Other process flows are also possible.

[0059] In operation 610, processing logic generates a target vector (e.g., a vector representation) for usage parameters corresponding to the test of the memory component. In some embodiments, one or more first usage parameters of the usage space of the memory component can be established as a target vector. In some embodiments, the target vector represents the ultimate purpose or result that the test is configured to test or to achieve. For example, the test can be configured to cause 100 memory components to perform 10,000 write operations. In this example, the target vector represents the end point for each memory component under test.

[0060] Example usage parameters that may be represented as target vectors include write count, read count, block type, partial programming rate, interrupt type, time between writes, time between reads, block erase type, read page fraction, interrupt rate, time between writes and reads, block type rate, and interrupt time. In some embodiments, a target vector is defined to describe an end state or constant usage property for a test sample.

[0061] In operation 620, processing logic assigns the test samples to the target vector. In some embodiments, the test samples refer to one or more memory blocks of the memory component that are subject to the same set of test conditions during their lifetime. In some embodiments, processing logic assigns each test sample in a set of multiple test samples to the target vector. In some embodiments, Figure 1 and Figure 4 The resource allocator 130 may coordinate the assignment or allocation of various memory components for the test samples through the corresponding target vector information.

[0062] In operation 630, processing logic generates a set of path variables to associate with the test samples. In some embodiments, processing logic randomly selects the set of path variables so that each test sample follows a randomized path to the target vector. In some embodiments, one or more usage parameters of the usage space are selected as path variables. In one embodiment, the path variable may be a property that changes continuously during the life of the memory component. In one embodiment, the test samples assigned to the target vector follow the path variable random path to the target vector for reads using media management techniques. Example usage parameters that may be selected as path variables include write temperature, write Vcc, Vcc noise, read temperature, write temperature, read Vcc, and parallelism of write and read operations.

[0063] In some embodiments, each test sample (e.g., a group of memory component blocks) is assigned an individual target vector and a random path (e.g., a randomly selected set of path variables) leading to the target vector is followed. In some embodiments, processing logic establishes a plurality of test samples corresponding to a usage space of the memory component to identify execution results at a multi-dimensional hierarchy of the usage space associated with the memory component. Fig. 7A An example of two three-dimensional target vectors (e.g., representing holding usage parameters, program-erase (P / E) cycles, and reads) and three associated random paths leading to the respective target vectors is illustrated. For example, the target vector may specify usage parameters for 10,000 write operations to be performed at each memory component assigned to the test. Each memory component may perform 10,000 write operations corresponding to the usage parameters, but the path or execution of additional operations based on path variables may be different for each of the memory components used in the test. For example, a first memory component may perform 10,000 write operations for an intermediate read operation performed after every 100 write operations. A second memory component may perform 10,000 write operations, but perform an intermediate erase operation after every 120 write operations. Thus, each memory component used in the test may perform the same 10,000 write operations, but the intermediate operations or other such parameters during the execution of the 10,000 write operations may be different. For example, each memory element may perform 10,000 write operations, but under different temperature conditions varying at different rates and over time.

[0064] In some embodiments, processing logic generates a plurality of test sample points corresponding to the usage space by generating a plurality of sets of path variables to be associated with a plurality of different test samples. Figure 7BAn example generation of 10,000 test sample points illustrating a two-dimensional sequence. In some embodiments, the test sample points are generated using a sequencing method (e.g., a low-bias method, such as the Sobol sequencing method, etc.). In some embodiments, a target vector sample set is generated based on the test sample points. Figure 7C A set of 1,000 target vector samples including vector type labels generated by the processing logic of the present disclosure is illustrated. In some embodiments, the target vector samples are generated using a low-bias sequence, such as the Sobol sequencing method.

[0065] In block 640, processing logic performs a test according to the path variable set using the test sample to generate a test result. In some embodiments, the path variable set includes one or more randomized path variables. In some embodiments, processing logic can identify and reserve memory components to be tested by having established a target vector and a path variable set corresponding to the test sample. In some embodiments, processing logic (e.g., Figure 1 and 4 The processing logic of the resource allocator 130 of the embodiment of the present invention identifies and reserves test resources corresponding to the path variable set (eg, test sockets containing memory components that match the usage parameters of the path variable set) for use as test samples.

[0066] In operation 650, processing logic analyzes the test results (e.g., the first test result associated with the first test sample and other test results corresponding to the respective test samples) to identify failures. In some embodiments, one or more failures associated with one or more test samples may be identified. In some embodiments, the dimension data from the randomized tests is stored in a file and parsed into a smaller representation (e.g., a table). In one embodiment, failure detection and the associated determination of whether the failure is a true error or an anomaly (e.g., due to noise) may be performed as described above in conjunction with Figure 4 and 5 In some embodiments, the test results associated with each test sample are provided as feedback information to the management module (e.g., Figure 4 management module 480).

[0067] In an example, one or more functional failures associated with erase, program, and read operations are made visible and constitute a 'hit' to a test block involved in the test process. In an example, a read raw bit error rate (RBER) threshold may be applied to trigger an error handling process flow that is configured to recover the read operation (e.g., by selecting various trims (e.g., commands that enable an operating system to notify a memory component (e.g., a NAND flash solid state drive (SSD)) which data blocks it may erase because the data blocks are no longer in use)), an adaptive replacement cache (ARC), a Vt scan for failure breakdown due to errors (e.g., 'hit') related to the memory component. In some embodiments, after reaching a threshold number of hits (e.g., after three hits), the block may be retired or removed from the test. In some embodiments, anomalies may also be classified in an analysis (e.g., RBER, typical page programming time (tProg), tErase response, etc.) as statistical outliers, unstable manifestations, or other pathological behaviors. In some embodiments, anomaly detection analysis may be performed in conjunction with the parsing described above and / or during post-processing of the output table.

[0068] Figure 8 An example machine illustrating a computer system 800 may execute a set of instructions within the computer system 800 for causing the machine to perform any one or more of the methodologies discussed herein. In some embodiments, the computer system 800 may correspond to a host or server system that includes, is coupled to, or uses a computer that includes Figure 4 The resource allocator 130 and Figure 4 One or more test platforms in the management module 480 (e.g., to execute corresponding Figure 2 , 5 In some embodiments, the machine may be connected (e.g., using a network) to other machines in a LAN, an intranet, an extranet, and / or the Internet. The machine may operate in the capacity of a server or a client machine in a client-server network environment, or in the capacity of a peer machine in a peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0069] The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a network appliance, a server, a network router, a switch or a bridge, or any machine capable of executing (sequentially or otherwise) a set of instructions that specify actions to be taken by the machine. In addition, while a single machine is described, the term "machine" should also be construed to include any collection of machines that individually or collectively execute one (or more) sets of instructions to perform any one or more of the methodologies discussed herein.

[0070] The example computer system 800 includes a processing device 802, a main memory 804 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), etc.), a static memory 806 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage system 818, which communicate with each other via a bus 830.

[0071] The processing device 802 represents one or more general processing devices, such as a microprocessor, a central processing unit, etc. More specifically, the processing device can be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processing device 802 can also be one or more special processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processing device 802 is configured to execute instructions 826 for performing the operations and steps discussed herein. The computer system 800 may additionally include a network interface device 808 to communicate on a network 820.

[0072] The data storage system 818 may include a machine-readable storage medium 824 (also referred to as a computer-readable medium) on which is stored one or more sets of instructions 826 or software embodying any one or more of the methods or functions described herein. The instructions 826 may also reside, completely or at least partially, within the main memory 804 and / or within the processing device 802 during execution by the computer system 800, the main memory 804 and the processing device 802 also constituting machine-readable storage media. The machine-readable storage medium 824, the data storage system 818, and / or the main memory 804 may correspond to a memory subsystem.

[0073] In one embodiment, instructions 826 include implementing a test process corresponding to that described herein (e.g., Figure 2 , 5 and 6 processes 200, 500 and 600), Figure 4 The resource allocator 130 and Figure 4 The functional instructions of the management module 480 of the present invention are stored in the form of a memory. Although the machine-readable storage medium 824 is shown as a single medium in the example embodiment, the term "machine-readable storage medium" should be considered to include a single medium or multiple media storing one or more sets of instructions. The term "machine-readable storage medium" should also be considered to include any medium that is capable of storing or encoding a set of instructions for execution by a machine and causing the machine to perform any one or more of the methods of the present disclosure. The term "machine-readable storage medium" should therefore be considered to include, but not be limited to, solid-state memory, optical media, and magnetic media.

[0074] Some portions of the previous detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing can most effectively convey the substance of their work to others skilled in the art. In this article, and generally, an algorithm is conceived to be a self-consistent sequence of operations that produce a desired result. An operation is one that requires physical manipulation of physical quantities. Typically (but not necessarily), these quantities take the form of electrical or magnetic signals that can be stored, combined, compared, and otherwise manipulated. It has proven to be convenient at times, primarily for common reasons, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, and the like.

[0075] It should be borne in mind, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage systems.

[0076] The present disclosure also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the desired purpose, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk, including a floppy disk, an optical disk, a CD-ROM, and a magnetic optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card or an optical card, or any type of medium suitable for storing electronic instructions, each of which is connected to a computer system bus.

[0077] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems may be used in conjunction with the programs according to the teachings herein, or it may prove convenient to construct a more specialized device to perform the method. The structures of various such systems will be presented as set forth in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It should be appreciated that the teachings of the present disclosure as described herein may be implemented using various programming languages.

[0078] The present disclosure may be provided as a computer program product or software, which may include a machine-readable medium having stored thereon instructions that can be used to program a computer system (or other electronic device) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium, such as a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory component, etc.

[0079] In the foregoing description, embodiments of the present disclosure have been described with reference to specific example embodiments thereof. It will be apparent that various modifications may be made to the present disclosure without departing from the broader spirit and scope of the embodiments of the present disclosure as set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative sense rather than a restrictive sense.

Claims

1. A method comprising: generating a target vector representing usage parameters corresponding to the test of the memory component; Assigning a test sample to the target vector; randomly selecting a set of path variables to associate with the test sample; executing, by a processing device, a test process of the test using the test sample according to the set of path variables to generate a test result; and A failure associated with the test result is identified.

2. The method according to claim 1, further comprising: One or more memory components are reserved as the test samples.

3. The method of claim 1, further comprising: A plurality of different target vectors are generated, each of the different target vectors representing a different usage parameter.

4. The method according to claim 3, further comprising A set of one or more test samples is assigned to each of the plurality of different target vectors.

5. The method according to claim 4, further comprising: A randomly selected set of path variables is generated to be associated with each of the set of one or more test samples.

6. The method of claim 1, further comprising: Feedback information including the test result is received via a connection corresponding to a test process of the test sample.

7. A system comprising: Memory components; and A processing device operatively associated with the memory component to perform the following operations: generating a plurality of target vectors corresponding to a plurality of first type usage parameters of the memory component; Assigning a first test sample to a target vector among the plurality of target vectors; assigning a second test sample to the target vector of the plurality of target vectors; generating a first set of randomly selected path variables for the first test sample; generating a second set of randomly selected path variables for the second test sample; performing a first test process according to the first set of randomly selected path variables and the target vector using the first test sample to produce a first test result; and A second testing process is performed using the second testing sample according to the second set of randomly selected path variables and the target vector to generate a second testing result.

8. The system of claim 7, wherein the first type of usage parameters comprises at least one of: write count, read count, block type, partial programming rate, interrupt type, time between writes, time between reads, block erase type, read page fraction, interrupt rate, time between writes and reads, block type rate, or interrupt time.

9. The system of claim 7, wherein the first set of path variables includes one or more usage parameters that exhibit changes during operation of a corresponding memory component.

10. The system according to claim 7, wherein the processing device further performs the following operations: identifying a first failure associated with the first test result; and It is determined that the first failure comprises an anomaly.

11. The system according to claim 7, wherein the processing device further performs the following operations: reserving a memory component of the test resource as the first test sample; and Connection information is assigned to the first test procedure performed using the first test sample.

12. The system according to claim 11, wherein the processing device further performs the following operations: Feedback information including the first test result is received from the first test process via the connection established using the connection information.

13. A non-transitory computer-readable medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising: generating a target vector representing usage parameters corresponding to the test of the memory component; Assigning a test sample to the target vector; randomly selecting a set of path variables to associate with the test sample; Using the test sample to perform a test according to the path variable set to generate a test result; and A failure associated with the test result is identified.

14. The non-transitory computer-readable medium of claim 13, wherein the operations further comprise: One or more memory components are reserved as the test samples.

15. The non-transitory computer-readable medium of claim 13, wherein the operations further comprise: A plurality of different target vectors are generated, each of the different target vectors representing a different usage parameter.

16. The non-transitory computer-readable medium of claim 15, wherein the operations further comprise: A set of one or more test samples is assigned to each of the plurality of different target vectors.

17. The non-transitory computer-readable medium of claim 16, wherein the operations further comprise: A randomly selected set of path variables is generated to be associated with each of the set of one or more test samples.

18. The non-transitory computer-readable medium of claim 13, wherein the operations further comprise: Feedback information including the test results is received via a connection with a test process corresponding to the test sample.

Images