Method and device for identifying performance bottleneck in large model software and hardware system, and electronic equipment

Abstract

The invention relates to the technical field of large-model software and hardware system testing, and discloses a method and device for identifying performance bottlenecks in a large-model software and hardware system and electronic equipment. The identification method comprises the steps of determining a second index acquisition script suitable for a to-be-tested large model software and hardware system according to a first index acquisition script of a representative system; controlling the to-be-tested large model software and hardware system to execute the standard test case, and calling a second index collection script to collect multiple test data when the to-be-tested large model software and hardware system executes the standard test case; determining various performance indexes of the to-be-tested large model software and hardware system in a preset performance index set according to the multiple test data; and comparing the performance indexes of the to-be-tested large model software and hardware system and the representative system in the preset performance index set, and determining the performance bottleneck of the to-be-tested large model software and hardware system. According to the invention, identification of the performance bottleneck of a large-model software and hardware system can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of large-model software and hardware system testing, for example, to a method and device for identifying performance bottlenecks in large-model software and hardware systems, and electronic equipment. Background Art

[0002] With the continued growth of large model parameter and data volumes, the demand for large-scale AI model hardware and software systems has expanded from a single machine and a single graphics card to multiple machines and multiple graphics cards, even reaching tens of thousands of cards. Therefore, identifying performance bottlenecks in large-scale model hardware and software systems, optimizing them, and improving their operational smoothness are essential.

[0003] In related technologies, when testing large-model software and hardware systems, the test benchmarks used mainly focus on the overall end-to-end performance of the large-model software and hardware systems. This approach cannot determine the specific performance bottlenecks that exist in large-model software and hardware systems. For example, when using MLPerf as a test benchmark, the main focus is on single indicators such as the throughput of large-model software and hardware systems, the average mixed precision processed per second, and the number of AI floating-point operations. The focus is on evaluating the ultimate performance that the large-model software and hardware systems can ultimately achieve, and it cannot determine which performance bottlenecks exist in the large-model software and hardware systems. Therefore, how to provide a solution to identify the performance bottlenecks of large-model software and hardware systems and provide a reference basis for the optimization of large-model software and hardware systems has become a technical problem that needs to be solved urgently.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] The embodiments of the present disclosure provide a method and device, and an electronic device for identifying performance bottlenecks in a large-model software and hardware system, which can realize the identification of performance bottlenecks in a large-model software and hardware system.

[0007] In some embodiments, a method for identifying performance bottlenecks in a large model software and hardware system includes: determining a second indicator collection script applicable to the large model software and hardware system to be tested based on a first indicator collection script of a representative system; controlling the large model software and hardware system to be tested to execute standard test cases, and calling the second indicator collection script to collect multiple test data when the large model software and hardware system to be tested executes the standard test cases; determining the various performance indicators of the large model software and hardware system to be tested in a preset performance indicator set based on the multiple test data; comparing the various performance indicators of the large model software and hardware system to be tested and the representative system in the preset performance indicator set to determine the performance bottleneck of the large model software and hardware system to be tested.

[0008] Optionally, based on the first indicator collection script of the representative system, a second indicator collection script suitable for the large model software and hardware system to be tested is determined, including: determining the configuration information of the large model software and hardware system to be tested; converting part of the content in the first indicator collection script according to the configuration information to determine the second indicator collection script.

[0009] Optionally, the preset performance indicator set includes a system performance indicator set and a GPU hardware chip performance indicator set; the second indicator collection script includes a first process script and a second process script; the second indicator collection script is called to collect multiple test data when the large model software and hardware system to be tested executes standard test cases, including: when the large model software and hardware system to be tested executes standard test cases, calling the first process script to collect test data for calculating each performance indicator in the system performance indicator set, and synchronously calling the second process script to collect test data for calculating each performance indicator in the GPU hardware chip performance indicator set.

[0010] Optionally, the system performance indicator set includes: CPU utilization and memory utilization of the software and hardware system of the large model to be tested.

[0011] Optionally, the GPU hardware chip performance indicator set includes: GPU computing power utilization, GPU memory utilization, GPU node communication bandwidth and power consumption of the large model hardware and software system to be tested.

[0012] Optionally, the performance indicators of the large model software and hardware system to be tested and the representative system in a preset performance indicator set are compared to determine the performance bottleneck of the large model software and hardware system to be tested, including: taking the performance indicator of the large model software and hardware system to be tested that performs worse than the representative system as the target performance indicator; and determining the target performance indicator as the performance bottleneck of the large model software and hardware system to be tested.

[0013] Optionally, after determining the performance bottleneck of the large model software and hardware system to be tested, the identification method further includes: determining an optimization solution for the large model software and hardware system to be tested based on the performance bottleneck of the large model software and hardware system to be tested.

[0014] In some embodiments, a device for identifying performance bottlenecks in a large model software and hardware system includes: a script determination module, configured to determine a second indicator collection script applicable to the large model software and hardware system to be tested based on a first indicator collection script of a representative system; a data collection module, configured to control the large model software and hardware system to be tested to execute standard test cases, and call the second indicator collection script to collect multiple test data when the large model software and hardware system to be tested executes the standard test cases; an indicator determination module, configured to determine the various performance indicators of the large model software and hardware system to be tested in a preset performance indicator set based on multiple test data; a bottleneck determination module, configured to compare the various performance indicators of the large model software and hardware system to be tested and the representative system in the preset performance indicator set to determine the performance bottleneck of the large model software and hardware system to be tested.

[0015] In some embodiments, a device for identifying performance bottlenecks in a large model software and hardware system includes a processor and a memory storing program instructions. The processor is configured to execute the above-mentioned method for identifying performance bottlenecks in a large model software and hardware system when running the program instructions.

[0016] In some embodiments, an electronic device includes: a device body; and the above-mentioned device for identifying performance bottlenecks in a large-model software and hardware system is installed in the device body.

[0017] The method, device, and electronic device for identifying performance bottlenecks in large-scale model software and hardware systems provided by the embodiments of the present disclosure can achieve the following technical effects:

[0018] In the embodiment of the present disclosure, a set of fine-grained, multi-dimensional performance indicators covering aspects such as computing, communication, storage and power consumption is preset for key components and connections in the large-model software and hardware system that may become performance bottlenecks, and a large-model software and hardware system that is widely used and has excellent performance in all aspects is determined as a representative system through big data statistics. When identifying the performance bottleneck of the large-model software and hardware to be tested, first, the first indicator collection script of the representative system will be referenced to determine the second indicator collection script applicable to the large-model software and hardware system to be tested. Then, the second indicator collection script will be called to collect a number of test data when the large-model software and hardware system to be tested executes the standard test case. Finally, based on a number of test data, each performance indicator of the large-model software and hardware system to be tested in the preset performance indicator set will be determined, and each performance indicator will be compared with the corresponding performance indicator in the representative system to determine the performance bottleneck of the large-model software and hardware system to be tested. In this way, the performance bottleneck of the large-model software and hardware system is accurately identified, which is conducive to improving the subsequent performance optimization efficiency of the large-model software and hardware system to be tested.

[0019] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] One or more embodiments are exemplarily described by corresponding drawings. These exemplary descriptions and drawings do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation. In addition,

[0021] Figure 1 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of a method for identifying performance bottlenecks in a large-model software and hardware system provided by an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of a large model software and hardware system provided by an embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of another method for identifying performance bottlenecks in a large-model software and hardware system provided by an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of a device for identifying performance bottlenecks in a large-model software and hardware system provided by an embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of another device for identifying performance bottlenecks in a large-model software and hardware system provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0028] In the description and claims of the embodiments of the present disclosure, as well as in the accompanying drawings, the terms "first," "second," and the like are used to distinguish similar items and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate to describe the embodiments of the present disclosure herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

[0029] Unless otherwise stated, the term "plurality" means two or more.

[0030] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0031] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0032] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0034] like Figure 1 As shown, the electronic device 100 provided by the embodiment of the present disclosure includes a device body 110 and a device 500 (600) for identifying performance bottlenecks in a large-scale software and hardware system. The device 500 (600) for identifying performance bottlenecks in a large-scale software and hardware system is provided in the device body.

[0035] Optionally, the device 600 for identifying performance bottlenecks in a large model software and hardware system includes a processor, which can determine a second indicator collection script applicable to the large model software and hardware system to be tested based on a first indicator collection script of a representative system. The second indicator collection script can be called to collect multiple test data when the large model software and hardware system to be tested executes a standard test case. After the test data collection is completed, the various performance indicators of the large model software and hardware system to be tested in a preset performance indicator set can be determined based on the multiple test data; and the various performance indicators can be compared with the corresponding performance indicators in the representative system to determine the performance bottlenecks in the large model software and hardware system to be tested.

[0036] In combination with the above electronic device, the embodiment of the present disclosure provides a method for identifying performance bottlenecks in a large model software and hardware system, such as Figure 2 As shown, the identification method includes:

[0037] S201, the processor determines a second indicator collection script applicable to the large model software and hardware system to be tested based on the first indicator collection script of the representative system.

[0038] Specifically, representative systems are large-scale hardware and software systems that are widely used and exhibit excellent performance across all aspects, as determined through big data statistics. For example, if big data statistics determine that the PyTorch software framework is used in 60% of papers, that NVIDIA GPUs hold a 98% market share in global data center GPU hardware chips, and that the large-scale hardware and software system consisting of the PyTorch software framework and NVIDIA GPU hardware chips exhibits excellent performance across all aspects, then the large-scale hardware and software system consisting of the PyTorch software framework and NVIDIA GPU hardware chips can be considered a representative system.

[0039] Specifically, the first indicator collection script is designed for a representative system and is capable of collecting multiple test data when the representative system executes a test case. The collected multiple test data need to be able to be used to calculate each performance indicator in the preset performance indicator set.

[0040] Specifically, since the large model software and hardware system to be tested and the representative system both belong to the artificial intelligence large model software and hardware system. Therefore, the large model software and hardware system to be tested and the representative system will only have some differences in the software framework or hardware chip used. Therefore, it is only necessary to modify some parameters in the first indicator collection script or convert some scripts according to these differences, so that it can be used to collect test data of the large model software and hardware to be tested. Therefore, based on the first indicator collection script of the representative system, the second indicator collection script suitable for the large model software and hardware system to be tested can be determined.

[0041] S202, the processor controls the large model software and hardware system to be tested to execute standard test cases, and calls the second indicator collection script to collect multiple test data when the large model software and hardware system to be tested executes the standard test cases.

[0042] Specifically, the standard test cases include the language large model (llama), multimodal large model (Stable Diffusion), supporting data sets (MMLU or COCO) and execution scripts that need to be run in the large model software and hardware system to be tested.

[0043] Specifically, when the large model software and hardware system under test executes standard test cases, its performance can more accurately reflect its performance in actual applications. Therefore, it is necessary to call the second indicator collection script to collect multiple test data of the large model software and hardware under test when the large model software and hardware system under test executes standard test cases.

[0044] Specifically, when the large model software and hardware system to be tested executes standard test cases, the various data generated will be stored in the form of log files. Therefore, when collecting multiple test data, it can be done by querying the log files of the large model software and hardware system to be tested.

[0045] S203: The processor determines various performance indicators of the large model software and hardware system to be tested in a preset performance indicator set based on the multiple test data.

[0046] Specifically, based on multiple test data, it is possible to clearly determine the specific performance of the large-scale software and hardware system under test on each performance indicator in the preset performance indicator set when executing the standard test case. Therefore, based on multiple test data, it is possible to determine the various performance indicators of the large-scale software and hardware system under test.

[0047] Specifically, the preset performance indicator set is a fine-grained, multi-dimensional performance indicator set covering computing, communication, storage, and power consumption, set for key components and connections in large-model hardware and software systems that may become performance bottlenecks.

[0048] Specifically, the large model hardware and software system consists of Figure 3 The system consists of one or more nodes as shown. Each node includes one or more storage components, computing components, and communication components. The performance indicator set is a set of performance indicators set for these components and the connections between them. To accurately identify performance defects in large-scale model software and hardware systems, corresponding indicators are set from multiple dimensions. Specific dimensions include:

[0049] Storage dimension: The performance indicators of storage dimension include Figure 3 The GPU storage capacity indicator, GPU storage utilization indicator, and memory utilization indicator of the storage component A, and the theoretical memory access bandwidth indicator and actual memory access bandwidth indicator of D at the connection between the storage component and the computing component.

[0050] Calculation dimension: The performance indicators of the calculation dimension include Figure 3 The GPU theoretical computing power peak index, GPU computing power utilization index, and CPU computing utilization index of the medium B computing component;

[0051] Communication dimension: The performance indicators of the communication dimension include Figure 3 The theoretical communication bandwidth index within the node and the actual communication bandwidth index within the node at the connection point E between the C communication component and the B computing component in the same node; the theoretical communication bandwidth index between nodes and the actual communication bandwidth index between nodes at the connection point F between the C communication components between different nodes.

[0052] Power consumption: The performance indicator of the power consumption dimension is the heat released after the computing components in the node are adjusted to a state close to full load, measured in watts.

[0053] S204, the processor compares the performance indicators of the large model software and hardware system to be tested and the representative system in the preset performance indicator set to determine the performance bottleneck of the large model software and hardware system to be tested.

[0054] Specifically, by comparing the various performance indicators of the large model software and hardware system to be tested and the representative system, it can be clearly determined on which performance indicators the large model software and hardware system to be tested does not meet the standards of the representative system. These performance indicators that do not meet the standards of the representative system are performance bottlenecks during the optimization of the large model software and hardware system to be tested.

[0055] In the disclosed embodiment, when identifying the performance bottleneck of the large model software and hardware system to be tested, first, the first indicator collection script of the representative system will be referenced to determine the second indicator collection script applicable to the large model software and hardware system to be tested. Then, the second indicator collection script will be called to collect multiple test data when the large model software and hardware system to be tested executes the standard test case. Finally, based on the multiple test data, the various performance indicators of the large model software and hardware system to be tested in the preset performance indicator set will be determined, and the various performance indicators will be compared with the corresponding performance indicators in the representative system to determine the performance bottleneck of the large model software and hardware system to be tested. In this way, the performance bottleneck of the large model software and hardware system is accurately identified, which is conducive to improving the subsequent performance optimization efficiency of the large model software and hardware system to be tested.

[0056] In some embodiments, based on the first indicator collection script of the representative system, a second indicator collection script applicable to the large model software and hardware system to be tested is determined, including: determining the configuration information of the large model software and hardware system to be tested; converting part of the content in the first indicator collection script according to the configuration information to determine the second indicator collection script.

[0057] Specifically, since the large model software and hardware system to be tested and the representative system both belong to the artificial intelligence large model software and hardware system. The large model software and hardware system to be tested and the representative system will only have some differences in the software framework or hardware chip used. According to the configuration information of the large model software and hardware system to be tested, the specific differences between the large model software and hardware system to be tested and the representative system can be determined, and some parameters that need to be modified or some scripts that need to be converted in the first indicator collection script can be determined based on the specific differences. Therefore, part of the content in the first indicator collection script can be converted according to the configuration information of the large model software and hardware system to be tested to determine the second indicator collection script.

[0058] Specifically, a conversion tool is pre-installed. By inputting the configuration information of the large model's hardware and software systems and the first indicator collection script into the conversion tool, the second indicator collection script can be determined. If the conversion cannot be completed, the user will be prompted to refer to the first indicator collection script and manually rewrite the second indicator collection script.

[0059] In some embodiments, the preset performance indicator set includes a system performance indicator set and a GPU hardware chip performance indicator set; the second indicator collection script includes a first process script and a second process script; the second indicator collection script is called to collect multiple test data when the large model software and hardware system to be tested executes the standard test case, including: when the large model software and hardware system to be tested executes the standard test case, calling the first process script to collect test data for calculating each performance indicator in the system performance indicator set, and synchronously calling the second process script to collect test data for calculating each performance indicator in the GPU hardware chip performance indicator set.

[0060] Specifically, by dividing the preset performance indicator set into a system performance indicator set and a GPU hardware chip performance indicator set, and dividing the second indicator collection script into a first process script and a second process script. When the large model software and hardware system to be tested executes the standard test case, the first process script is called to collect test data for calculating each performance indicator in the system performance indicator set, and the second process script is simultaneously called to collect test data for calculating each performance indicator in the GPU hardware chip performance indicator set. This can achieve dual-process (1+1 process) test data collection. In this way, the efficiency of test data collection can be improved.

[0061] Optionally, the system performance indicator set includes: CPU utilization and memory utilization of the software and hardware system of the large model to be tested.

[0062] Optionally, the GPU hardware chip performance indicator set includes: GPU computing power utilization, GPU memory utilization, GPU node communication bandwidth and power consumption of the large model hardware and software system to be tested.

[0063] It's important to note that since the first process script is used to collect each performance indicator in the system performance indicator set, and different large-scale model software and hardware systems share the same collection scripts for these performance indicators, when determining the second indicator collection script based on the first indicator collection script, there's no need to modify the first process script. This allows the large-scale model software and hardware system under test to directly reuse portions of the first process script in the first indicator collection script.

[0064] In some embodiments, the performance indicators of the large model software and hardware system to be tested and the representative system in a preset performance indicator set are compared to determine the performance bottleneck of the large model software and hardware system to be tested, including: taking the performance indicator of the large model software and hardware system to be tested that performs worse than the representative system as the target performance indicator; and determining the target performance indicator as the performance bottleneck of the large model software and hardware system to be tested.

[0065] Specifically, if the performance of the large-scale model software and hardware system under test is worse than that of the representative system on certain performance indicators, it indicates that the performance of the large-scale model software and hardware system under test on these performance indicators does not meet the standards of the representative system. Therefore, these performance indicators can be used as target performance, and the target performance indicators can be determined as the performance bottlenecks of the large-scale model software and hardware system under test.

[0066] The present disclosure provides another method for identifying performance bottlenecks in large-scale software and hardware systems. Figure 4 As shown, the identification method includes:

[0067] S401, the processor determines a second indicator collection script applicable to the large model software and hardware system to be tested based on the first indicator collection script of the representative system.

[0068] S402, the processor controls the large model software and hardware system to be tested to execute standard test cases, and calls the second indicator collection script to collect multiple test data when the large model software and hardware system to be tested executes the standard test cases.

[0069] S403: The processor determines various performance indicators of the large model software and hardware system to be tested in a preset performance indicator set based on the multiple test data.

[0070] S404, the processor compares various performance indicators of the large model software and hardware system to be tested and the representative system in a preset performance indicator set to determine the performance bottleneck of the large model software and hardware system to be tested.

[0071] S405 , the processor determines an optimization solution for the software and hardware system of the large model to be tested based on the performance bottleneck of the software and hardware system of the large model to be tested.

[0072] Specifically, different performance bottlenecks require different optimization strategies. For example, if the bottleneck lies in insufficient computing power, then adding high-performance computing equipment and optimization algorithms can be helpful. If the bottleneck lies in slow data transmission, then improving data transmission protocols and increasing network bandwidth can be helpful. Therefore, optimization strategies for the large-scale model software and hardware systems under test can be determined based on their performance bottlenecks.

[0073] In the disclosed embodiment, after identifying the performance bottlenecks of the large model software and hardware to be tested, an optimization solution for the large model software and hardware system to be tested can be provided to the user based on the specific performance bottlenecks. This can improve the performance optimization efficiency of the large model software and hardware system to be tested.

[0074] Combine Figure 5 As shown, an embodiment of the present disclosure provides a device for identifying performance bottlenecks in large-model software and hardware systems, including: a script determination module 501, a data acquisition module 502, an indicator determination module 503, and a bottleneck determination module 504. The script determination module 501 is configured to determine a second indicator collection script applicable to the large-model software and hardware system to be tested based on a first indicator collection script of a representative system. The data acquisition module 502 is configured to control the large-model software and hardware system to be tested to execute standard test cases, and call the second indicator collection script to collect multiple test data when the large-model software and hardware system to be tested executes standard test cases. The indicator determination module 503 is configured to determine the various performance indicators of the large-model software and hardware system to be tested in a preset performance indicator set based on multiple test data. The bottleneck determination module 504 is configured to compare the various performance indicators of the large-model software and hardware system to be tested and the representative system in the preset performance indicator set to determine the performance bottleneck of the large-model software and hardware system to be tested.

[0075] Combine Figure 6 As shown, the embodiment of the present disclosure provides a device 600 for identifying performance bottlenecks in a large model software and hardware system, including: a processor (processor) 601 and a memory (memory) 602. Optionally, the device may also include a communication interface (Communication Interface)

[0076] 603 and bus 604. The processor 601, communication interface 603, and memory 602 can communicate with each other via bus 604. Communication interface 603 can be used for information transmission. Processor 601 can call logic instructions in memory 602 to execute the method for identifying performance bottlenecks in a large-scale hardware and software system according to the above embodiment.

[0077] In addition, the logic instructions in the memory 602 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0078] Memory 602, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. Processor 601 executes the program instructions / modules stored in memory 602 to execute functional applications and data processing, thereby implementing the method for identifying performance bottlenecks in large-scale hardware and software systems in the above-mentioned embodiments.

[0079] The memory 602 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal device. Furthermore, the memory 602 may include high-speed random access memory and non-volatile memory.

[0080] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned method for identifying performance bottlenecks in a large-model software and hardware system.

[0081] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0082] The technical solution of the embodiments of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code, or a transient storage medium.

[0083] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to also include plural forms. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variations "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, an element defined by the sentence "comprising a..." does not exclude the presence of other identical elements in the process, method or device that includes the element. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments can be referenced to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can be found in the description of the method part.

[0084] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0085] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units can be merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, and can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of the present disclosure may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0086] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for identifying performance bottlenecks in large-scale software and hardware systems, characterized in that: include: Determine a second indicator collection script suitable for the large model software and hardware system to be tested based on the first indicator collection script of the representative system; Controlling the large model software and hardware system to be tested to execute standard test cases, and calling the second indicator collection script to collect multiple test data when the large model software and hardware system to be tested executes the standard test cases; Determine the performance indicators of the large model software and hardware system to be tested in the preset performance indicator set based on multiple test data; Compare the performance indicators of the large model hardware and software system to be tested and the representative system in the preset performance indicator set to determine the performance bottleneck of the large model hardware and software system to be tested.

2. The identification method according to claim 1, characterized in that Based on the first indicator collection script of the representative system, determine the second indicator collection script applicable to the large model software and hardware system to be tested, including: Determine the configuration information of the software and hardware system of the large model to be tested; Part of the content in the first indicator collection script is converted according to the configuration information to determine the second indicator collection script.

3. The identification method according to claim 1, characterized in that The preset performance indicator set includes a system performance indicator set and a GPU hardware chip performance indicator set; the second indicator collection script includes a first process script and a second process script; Call the second indicator collection script to collect multiple test data when the large model software and hardware system to be tested executes standard test cases, including: When the large model software and hardware system to be tested executes standard test cases, the first process script is called to collect test data for calculating each performance indicator in the system performance indicator set, and the second process script is synchronously called to collect test data for calculating each performance indicator in the GPU hardware chip performance indicator set.

4. The identification method according to claim 3, characterized in that The system performance indicator set includes: CPU utilization and memory utilization of the hardware and software system of the large model to be tested.

5. The identification method according to claim 3, characterized in that: The set of GPU hardware chip performance indicators includes: GPU computing power utilization, GPU memory utilization, GPU node communication bandwidth and power consumption of the large model hardware and software system to be tested.

6. The identification method according to claim 1, characterized in that Compare the performance indicators of the large model software and hardware system to be tested with those of representative systems in the preset performance indicator set to identify the performance bottlenecks of the large model software and hardware system to be tested, including: The performance indicators of the large model software and hardware system that are worse than those of the representative system are used as target performance indicators; The target performance indicator is determined as the performance bottleneck of the hardware and software system of the large model to be tested.

7. The identification method according to any one of claims 1 to 6, characterized in that: After determining the performance bottlenecks of the software and hardware systems of the large model to be tested, the identification method also includes: Based on the performance bottlenecks of the large model software and hardware system to be tested, determine the optimization plan for the large model software and hardware system to be tested.

8. A device for identifying performance bottlenecks in large-scale software and hardware systems, characterized in that: include: a script determination module configured to determine a second indicator collection script applicable to the large model software and hardware system to be tested based on the first indicator collection script of the representative system; The data acquisition module is configured to control the large model software and hardware system to be tested to execute the standard test case, and call the second indicator acquisition script to collect multiple test data when the large model software and hardware system to be tested executes the standard test case; An indicator determination module is configured to determine various performance indicators of the large model software and hardware system to be tested in a preset performance indicator set based on a plurality of test data; The bottleneck determination module is configured to compare the performance indicators of the large model software and hardware system to be tested and the representative system in a preset performance indicator set to determine the performance bottleneck of the large model software and hardware system to be tested.

9. A device for identifying performance bottlenecks in a large-scale model software and hardware system, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the method for identifying performance bottlenecks in a large model software and hardware system according to any one of claims 1 to 7 when running the program instructions.

10. An electronic device, characterized in that: include: Equipment body; The device for identifying performance bottlenecks in a large-model software and hardware system as described in claim 8 or 9 is installed on the device body.

Images