Task observation method and electronic device

Abstract

The application discloses a task observation method and electronic equipment, and relates to the technical field of computers, and the method comprises the following steps: collecting the start time and the end time of each reasoning task, and determining the time range of each reasoning task according to the start time and the end time of each reasoning task; collecting the start time and the end time of various events in an initialization stage and a reasoning calculation stage; determining various events corresponding to each reasoning task according to the time range of each reasoning task and the start time and the end time of various events, and forming an event queue of each reasoning task according to the occurrence order of various events.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, to a task observation method and an electronic device. BACKGROUND

[0002] In an artificial intelligence (AI) inference task, performance optimization and fault positioning are crucial links. In the related art, task observation mainly relies on tools provided by manufacturers. Although these tools can provide certain performance indicators, they have many limitations. First, the observation granularity is coarse and cannot be accurately traced to each key link of the inference task. Second, the observation indicators are limited and cannot provide fine-grained performance data. More importantly, most task observation technologies focus on a single GPU (Graphics Processing Unit), which makes it impossible to comprehensively analyze the cooperation between the CPU (Central Processing Unit) and the GPU in the inference task, and thus cannot accurately identify performance bottlenecks and fault points.

[0003] Therefore, how to achieve fine-grained and comprehensive event observation is a technical problem to be solved by those skilled in the art. SUMMARY

[0004] The present application relates to the technical field of computer, more particularly, to a task observation method and an electronic device.

[0005] To achieve the above-mentioned purpose, the present application provides a task observation method, comprising: collecting the start time and the end time of each inference task, and determining the time range of each inference task according to the start time and the end time of each inference task; collecting the start time and the end time of each type of event in the initialization stage and the inference calculation stage; determining each type of event corresponding to each inference task according to the time range of each inference task and the start time and the end time of each type of event, and forming an event queue of each inference task according to the occurrence order of each type of event.

[0006] To achieve the above-mentioned purpose, the present application provides a task observation device, comprising: a determination module configured to collect the start time and the end time of each inference task, and determine the time range of each inference task according to the start time and the end time of each inference task; a collection module configured to collect the start time and the end time of each type of event in the initialization stage and the inference calculation stage; and an association module configured to determine each type of event corresponding to each inference task according to the time range of each inference task and the start time and the end time of each type of event, and form an event queue of each inference task according to the occurrence order of each type of event.

[0007] To achieve the above object, the present application provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program to realize the steps of the above task observation method.

[0008] To achieve the above object, the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to realize the steps of the above task observation method.

[0009] To achieve the above object, the present application provides a computer program product, comprising a computer program, the computer program being executed by a processor to realize the steps of the above task observation method.

[0010] The task observation method provided by the present application firstly comprehensively covers the cooperation process of the inference task between the host and the accelerator by jointly observing various events executed in the host and the accelerator. This joint observation method not only includes the start and end time of the inference task, but also covers key events in the task initialization phase and the inference calculation phase, thereby providing more comprehensive performance analysis data. Secondly, by forming an event queue according to the occurrence sequence of various events, the execution flow of the inference task can be clearly displayed, which facilitates the identification of performance bottlenecks and fault points. Finally, the method significantly improves the comprehensiveness and accuracy of task observation through joint observation of events of the host and the accelerator, provides stronger support for performance optimization and fault location of the AI inference system, and improves the overall performance and reliability of the system.

[0011] The present application also discloses a task observation device, an electronic device, a computer readable storage medium and a computer program product, which can also achieve the above technical effects.

[0012] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1 A flowchart of a task observation method according to an exemplary embodiment.

[0015] Figure 2 A flowchart of a task observation method in an application embodiment provided by the present application.

[0016] Figure 3 FIG. 1 is a structural diagram of a task observation device according to an example embodiment.

[0017] Figure 4 FIG. 1 is a structural diagram of a task observation device according to an example embodiment. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0019] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0020] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0021] The embodiments of the present application provide a task observation method, and the method is described in detail in combination with the execution flow of the task observation method. Referring to Figure 1 FIG. 2 is a flowchart of a task observation method according to an example embodiment.

[0022] S101: Collect the start time and end time of each reasoning task, and determine the time range of each reasoning task according to the start time and end time of each reasoning task.

[0023] The reasoning task refers to a complete “request-computation-response” life cycle in the AI reasoning service, which is usually triggered by a user or an external system, for example, an image recognition request or a natural language processing call. In this step, the start time and end time of each reasoning task are collected, which can be accurate to nanoseconds, and then the start time and end time of each reasoning task are used as boundaries to determine the time range of each reasoning task.

[0024] As a feasible implementation, the start time and the end time of each inference task are collected, including: setting a mounting point at the entrance and exit of the main function of the user-mode inference service to collect the start time and the end time of each inference task.

[0025] The main function of the user-mode inference service is the entrance and exit of the inference task, and by setting the mounting point at the entrance and exit of the main function, the start and end time of the inference task can be accurately captured. The mounting point is a technical means that allows additional logic to be inserted at runtime, such as recording timestamps, which can be obtained by bpf_ktime_get_ns(). In specific implementation, the mounting point is set at the entrance of the main function of the inference service, and when the inference task starts, the mounting point records the current timestamp as the start time; the mounting point is set at the exit of the main function, and when the inference task ends, the mounting point records the current timestamp as the end time. In this way, the execution time of each inference task can be accurately measured. Taking python as an example, using eBPF user-mode tracking technology uprobe, mounting point uprobe is added at the following position: / usr / bin / python:request_function, the entrance records the start time of the event starttime, and uretprobe: / usr / bin / python: request_function, the return records the end time of the event endtime. At the same time, the process identifier (PID) of the current process can also be collected at the entrance and stored in the process set P (eBPF map).

[0026] S102: Collect the start time and the end time of various events in the initialization phase and the inference calculation phase.

[0027] The inference task is usually divided into an initialization phase and an inference calculation phase, and in this step, the start and end time of various events in the initialization phase and the inference calculation phase in the life cycle of the inference task are collected. The events in the initialization phase include: host memory allocation event, model loading event, accelerator video memory allocation event, etc., and the events in the inference calculation phase include: host memory to accelerator video memory copy event, accelerator kernel start event, accelerator calculation result synchronization event, accelerator video memory to host memory copy event, etc.

[0028] As a feasible implementation, the start time and the end time of various events in the collection initialization stage and the inference calculation stage are collected, including: mounting points are respectively set at the entrance and the exit of the memory mapping system call to collect the start time and the end time of the host memory allocation event; mounting points are respectively set at the entrance and the exit of the file system opening call to collect the start time and the end time of the model loading event; mounting points are respectively set at the entrance and the exit of the display memory allocation function to collect the start time and the end time of the accelerator display memory allocation event; mounting points are respectively set at the entrance and the exit of the memory copy function to collect the start time and the end time of the host memory to accelerator display memory copy event and the accelerator display memory to host memory copy event; mounting points are respectively set at the entrance and the exit of the kernel starting function to collect the start time and the end time of the accelerator kernel starting event; and mounting points are respectively set at the entrance and the exit of the device synchronization function to collect the start time and the end time of the accelerator calculation result synchronization event.

[0029] The memory mapping system call is used to allocate host memory, and by setting mounting points at the entrance and the exit thereof, the start and end times of the host memory allocation can be recorded. For example, an eBPF kernel-level tracking technology kprobe is used to add a mounting point at the Linux kernel nmap system call. The current process is acquired at the SEC ("kprobe / nmap") entrance, and if the PID of the current process belongs to the set P, the event start time starttime is recorded, and in the same way, the event end time endtime is recorded at the SEC ("kretprobe / nmap") return.

[0030] The file system opening call is used to load a model file, and by setting mounting points at the entrance and the exit thereof, the start and end times of the model loading can be recorded. For example, an eBPF kernel-level tracking technology kprobe is used to add a mounting point at the Linux kernel open system call. The current process is acquired at the SEC ("kprobe / open") entrance, and if the PID of the current process belongs to the set P, the event start time starttime is recorded, and in the same way, the event end time endtime is recorded at the SEC ("kretprobe / open") return.

[0031] The allocation of video memory function is used to allocate accelerator video memory, and by setting the mounting point at its entrance and exit, the start and end time of video memory allocation can be recorded. For example, taking CUDA as an example, GPU video memory allocation events are collected. The mounting point is added using the eBPF user space tracking technology uprobe. The current process is obtained at the SEC("uprobe / cudaMalloc") entrance, and if the current process PID belongs to the set P, the event start time starttime is recorded, and similarly, the event end time endtime is recorded at the SEC("uretprobe / cudaMalloc") return.

[0032] The memory copy function is used to transfer data between the host memory and the accelerator video memory, and by setting the mounting point at its entrance and exit, the start and end time of data copying can be recorded. As a feasible implementation, the direction parameter in the memory copy function is used to distinguish the host memory to accelerator video memory copy event and the accelerator video memory to host memory copy event. In specific implementation, the memory copy function usually contains a direction parameter, which is used to indicate the direction of data copying. For example, in CUDA, the fourth parameter cudaMemcpyKind kind of cudaMemcpy function can take the value cudaMemcpyHostToDevice or cudaMemcpyDeviceToHost, which respectively represent the copy from the host memory to the accelerator video memory and the copy from the accelerator video memory to the host memory. By checking this direction parameter, the two different direction memory copy events can be accurately distinguished.

[0033] For the host memory to accelerator video memory copy event, the mounting point is added using the eBPF user space tracking technology uprobe. The fourth parameter cudaMemcpyKind kind of the function is collected at the SEC("uprobe / cudaMemcpy") entrance, and when the parameter is cudaMemcpyHostToDevice and the current process belongs to the set P, the event start time starttime is recorded; similarly, the event end time endtime is recorded at the SEC("uretprobe / cudaMemcpy") return.

[0034] For the accelerator memory to host memory copy event, use the eBPF user space tracking technology uprobe to add a mount point. At the SEC("uprobe / cudaMemcpy") entry of the function, record the event start time starttime when the fourth parameter cudaMemcpyKind kind is cudaMemcpyDeviceToHost and the current process belongs to the set P; similarly, at the SEC("uretprobe / cudaMemcpy") return, record the event end time endtime.

[0035] That is, when the cudaMemcpy function is called, if the direction parameter is cudaMemcpyHostToDevice, the event is recorded as a "host memory to accelerator memory copy event", and if the direction parameter is cudaMemcpyDeviceToHost, the event is recorded as an "accelerator memory to host memory copy event". This way of distinguishing can avoid data confusion and ensure that the collected event data accurately reflects the direction and time of data transmission, providing more detailed information for subsequent performance analysis.

[0036] The kernel launch function (such as cudaLaunchKernel) is used to launch the accelerator kernel, and by setting mount points at its entrance and exit, the start and end times of kernel launch can be recorded. For example, taking CUDA as an example, the kernel launch event is collected. Use the eBPF user space tracking technology uprobe to add a mount point. At the SEC("uprobe / cudaKernelLaunch") entry, get the current process, and if the current process PID belongs to the set P, record the event start time starttime, and at the SEC("uretprobe / cudaKernelLaunch") return, record the event end time endtime.

[0037] The device synchronization function (such as cudaDeviceSynchronize) is used to synchronize the calculation results of the accelerator, and by setting mount points at its entrance and exit, the start and end times of result synchronization can be recorded. For example, taking CUDA as an example, use the eBPF user space tracking technology uprobe to add a mount point at the synchronization function cudaDeviceSynchronize. At the SEC("uprobe / cudaDeviceSynchronize") entry, get the current process, and if the current process PID belongs to the set P, record the event start time starttime; at the SEC("uretprobe / cudaDeviceSynchronize") return, record the event end time endtime.

[0038] It can be seen that through the above mounting points, the time information of each key event in the inference task can be comprehensively collected, providing detailed data for performance analysis.

[0039] As a feasible implementation, the hash value of an event is calculated according to the process identifier and the stack backtracking identifier of the event, and the start time and the end time of the same event are matched according to the hash value of the event at each mounting point.

[0040] In specific implementation, in a multi-process or multi-thread inference task, the process identifier (PID) and the stack backtracking identifier of an event are important bases for distinguishing different events. The process identifier can uniquely identify a process, and the stack backtracking identifier can distinguish different function call paths in the same process through call stack information. By combining the process identifier and the stack backtracking identifier to calculate the hash value, a unique identifier can be generated for each event. When the start time and the end time of an event are collected at the mounting point, the hash value will be used to match the start time and the end time of the same event. For example, assuming that in an inference task, the process identifier of an event is 1234, the stack backtracking identifier is 0x12345678, and the hash value calculated by a specific hash algorithm is 0x98765432. When the start time of the event is recorded, the hash value 0x98765432 will be stored together with the start time; when the end time of the event is recorded, the corresponding start time will be found through the same hash value 0x98765432, so as to accurately match the start time and the end time of the same event. This method can effectively avoid the confusion of event data in a multi-process or multi-thread environment, ensuring that the collected event data is accurate and reliable, and providing a reliable data basis for subsequent performance analysis.

[0041] For example, the collected data is uploaded to the event processing engine through the eBPF map, the map type is BPF_MAP_TYPE_HASH, the structure includes Key and Value, Key is the hash value calculated by combining process identification (PID) and stack trace identification, Value includes Type (event type), starttime (event start time), endtime (event end time). In the above mounting point, the starttime position collects process information pidinfo through bpf_get_current_pid_tgid, obtains the kernel API stack trace identification stackid through bpf_get_stackid, calculates the hash value hash1 through pidinfo+stackid, stores hash1 into key, stores starttime into the start time in the value corresponding to key, and writes the event type. Similarly, in the endtime position, pidinfo and stackid are collected and hash value hash2 is calculated, hash2 is traversed through the map to find hash1 equal to hash2, and endtime is stored into the end time in the value corresponding to key.

[0042] S103: Determine the events corresponding to each reasoning task according to the time range of each reasoning task and the start time and end time of each type of event, and form an event queue of each reasoning task according to the occurrence order of each type of event.

[0043] In this step, by comparing the start and end time of the event with the time range of the reasoning task, it can be determined which events belong to a certain specific reasoning task. For example, if the start time and end time of an event are both within the time range of a certain reasoning task, then this event belongs to the reasoning task. Arranging all events belonging to the same reasoning task in their occurrence order can form an event queue, which reflects the execution process of the reasoning task and helps to analyze the performance of the task and optimize the execution process.

[0044] As a feasible implementation, the method further includes: collecting process identifiers of the inference tasks at a main function entry of the user-mode inference service, and storing the process identifiers into a process set; and accordingly, determining the events corresponding to each inference task according to a time range of each inference task and start times and end times of the events, including: sequentially determining each process identifier in the process set as a target process identifier, and determining a target inference task corresponding to the target process identifier; determining an event as a candidate event of the target inference task, if the process identifier of the event is the target process identifier and the start time and the end time of the event are within the time range of the target inference task; and connecting the start times and the end times of the events in sequence to determine the events corresponding to the target inference task from the candidate events.

[0045] In a specific implementation, process identifiers (PIDs) of inference tasks are collected at a main function entry of a user-mode inference service, and the PIDs are stored into a process set, which can ensure that each inference task can be uniquely identified. In this way, different inference tasks can be distinguished, and data confusion can be avoided. For each target process identifier (i.e., each inference task), candidate events belonging to the inference task can be filtered out by comparing start times and end times of the events with a time range of the inference task. Then, the start times and the end times of the candidate events are connected in sequence to form a complete event sequence. In this way, events corresponding to each inference task can be accurately determined, and an event queue can be formed to provide a clear execution flow for performance analysis.

[0046] As a feasible implementation, the start times and the end times of the events are connected in sequence to determine the events corresponding to the target inference task from the candidate events, including: dividing the candidate events corresponding to the target inference task into different sets according to event types; connecting the start times and the end times of the events in the initialization phase in sequence to determine the events in the initialization phase corresponding to the target inference task in the sets corresponding to the events in the initialization phase, respectively; and connecting the start times and the end times of the events in the inference calculation phase in sequence to determine the events in the inference calculation phase corresponding to the target inference task in the sets corresponding to the events in the inference calculation phase, respectively.

[0047] In a specific implementation, the time range of each event is determined step by step, and events belonging to the target reasoning task are screened according to the time ranges, so as to form a complete event sequence. Specifically, first, the various candidate events are divided into different sets according to the event types, that is, a host memory allocation event set, a model loading event set, an accelerator memory allocation event set, a host memory to accelerator memory copy event set, an accelerator kernel start event set, an accelerator computing result synchronization event set, and an accelerator memory to host memory copy event set. Second, the start time and the end time of the events in the initialization phase are sequentially connected according to the occurrence order of the events in the initialization phase, that is, the order of the host memory allocation event, the model loading event, and the accelerator memory allocation event, so as to determine the events in the initialization phase corresponding to the target reasoning task in the sets corresponding to the events in the initialization phase, that is, to determine the target host memory allocation event, the target model loading event, and the target accelerator memory allocation event corresponding to the target reasoning task. Then, the start time and the end time of the events in the reasoning computation phase are sequentially connected according to the occurrence order of the events in the reasoning computation phase, that is, the order of the host memory to accelerator memory copy event, the accelerator kernel start event, the accelerator computing result synchronization event, and the accelerator memory to host memory copy event, so as to determine the events in the reasoning computation phase corresponding to the target reasoning task in the sets corresponding to the events in the reasoning computation phase, that is, to determine the target host memory to accelerator memory copy event, the target accelerator kernel start event, the target accelerator computing result synchronization event, and the target accelerator memory to host memory copy event corresponding to the target reasoning task.

[0048] As a feasible implementation, the start time and the end time of each type of event in the initialization stage are connected in sequence according to the occurrence order of each type of event in the initialization stage, so as to determine each type of event in the initialization stage corresponding to the target reasoning task in the set corresponding to each type of event in the initialization stage, including: determining the target host memory allocation event corresponding to the target reasoning task in the set corresponding to the host memory allocation event according to the start time of the target reasoning task; determining the target model loading event corresponding to the target reasoning task in the set corresponding to the model loading event according to the end time of the target host memory allocation event and the offset, and the model loading event time range, wherein the model loading event whose start time belongs to the model loading event time range is the target model loading event; determining the target accelerator memory allocation event corresponding to the target reasoning task in the set corresponding to the accelerator memory allocation event according to the end time of the target model loading event and the offset, and the accelerator memory allocation event time range, wherein the accelerator memory allocation event whose start time belongs to the accelerator memory allocation event time range is the target accelerator memory allocation event. According to the occurrence order of each type of event in the reasoning calculation stage, the start time and the end time of each type of event in the reasoning calculation stage are connected in sequence, so as to determine each type of event in the reasoning calculation stage corresponding to the target reasoning task in the set corresponding to each type of event in the reasoning calculation stage, including: determining the target host memory to accelerator memory copy event corresponding to the target reasoning task in the set corresponding to the host memory to accelerator memory copy event according to the end time of the target accelerator memory allocation event and the offset, and the host memory to accelerator memory copy event time range, wherein the host memory to accelerator memory copy event whose start time belongs to the host memory to accelerator memory copy event time range is the target host memory to accelerator memory copy event; determining the target accelerator kernel start event corresponding to the target reasoning task in the set corresponding to the accelerator kernel start event according to the end time of the target host memory to accelerator memory copy event and the offset, and the accelerator kernel start event time range, wherein the accelerator kernel start event whose start time belongs to the accelerator kernel start event time range is the target accelerator kernel start event; determining the target accelerator computation result synchronization event corresponding to the target reasoning task in the set corresponding to the accelerator computation result synchronization event according to the end time of the target accelerator kernel start event and the offset, and the accelerator computation result synchronization event time range, wherein the accelerator computation result synchronization event whose start time belongs to the accelerator computation result synchronization event time range is the target accelerator computation result synchronization event; determining the target accelerator memory to host memory copy event corresponding to the target reasoning task in the set corresponding to the accelerator memory to host memory copy event according to the end time of the target accelerator computation result synchronization event and the offset, and the accelerator memory to host memory copy event time range, wherein the accelerator memory to host memory copy event whose start time belongs to the accelerator memory to host memory copy event time range is the target accelerator memory to host memory copy event.

[0049] In a specific implementation, a host memory allocation event whose start time is the same as the start time of the target inference task is determined as the target host memory allocation event corresponding to the target inference task in the set of host memory allocation events, which is the first event in the inference task initialization phase. Then, according to the end time of the target host memory allocation event and a preset offset (for considering the interval time between events), the time range of the model loading event is determined, and the target model loading event belonging to the time range is screened out from the set of model loading events. Similarly, the target accelerator memory allocation event, the target host memory to accelerator memory copy event, the target accelerator kernel launch event, the target accelerator computation result synchronization event, and the target accelerator memory to host memory copy event are determined in turn. In this way, each event can be correctly identified and sorted according to the actual occurrence order, forming an event queue reflecting the complete execution process of the inference task.

[0050] As a preferred implementation, each event type is assigned a priority according to the event type and the execution flow of the inference task. For example, high priority: accelerator kernel launch event, accelerator computation result synchronization event; medium priority: host memory to accelerator memory copy event, accelerator memory to host memory copy event; low priority: host memory allocation event, model loading event, accelerator memory allocation event. When determining the candidate events corresponding to the target inference task, the size of the time window is dynamically adjusted according to the priority of the event, that is, the offset is negatively related to the priority. For high-priority events, the time window can be appropriately reduced to reduce the possibility of misjudgment. For example, the time window is set to the end time of the event plus a small offset (such as 100 nanoseconds). For low-priority events, the time window can be appropriately expanded to ensure that relevant events are not missed. For example, the time window is set to the end time of the event plus a large offset (such as 500 nanoseconds). It can be seen that by dynamically adjusting the time window, the size of the time window can be flexibly controlled according to the priority of the event, which can more accurately screen out events belonging to the same inference task and reduce misjudgment and omission.

[0051] The task observation method provided by the embodiment of the application first comprehensively covers the cooperation process of the inference task between the host and the accelerator by jointly observing various events executed in the host and the accelerator. The joint observation method not only includes the start and end time of the inference task, but also covers key events in the task initialization stage and the inference calculation stage, thereby providing more comprehensive performance analysis data. Secondly, by forming an event queue according to the occurrence sequence of various events, the execution flow of the inference task can be clearly displayed, and the performance bottleneck and fault point can be easily identified. Finally, the method significantly improves the comprehensiveness and accuracy of task observation through joint observation of events of the host and the accelerator, provides stronger support for performance optimization and fault positioning of the AI inference system, and improves the overall performance and reliability of the system.

[0052] An application embodiment provided by the application is introduced below. In this embodiment, eBPF uprobe is used to observe each inference calculation task event, GPU memory allocation event, memory data copying event, GPU kernel starting event and GPU calculation result synchronization event in the user mode, and record the time consumption of each stage in the user mode. eBPF kprobe is used to observe host memory allocation event and model loading event in the kernel mode, and record the time consumption of each stage in the kernel mode. After interference data is removed through process PID information and stack backtracking, the data is uploaded to an event processing engine. The event processing engine classifies events, associates the same inference calculation task through timestamps, and displays the inference calculation stage time consumption to the user, thereby providing a basis for performance bottleneck analysis.

[0053] Task initialization stage time consumption acquisition: uprobe and uretprobe are used to add mounting points at the user request, which are used to acquire the total time consumption of each inference calculation task. Kprobe and kretprobe are used to add mounting points at the model loading and host memory allocation, which are used to acquire the CPU processing part time consumption. Uprobe and uretprobe are used to add mounting points at the GPU memory allocation, which are used to acquire the GPU memory allocation part time consumption.

[0054] Inference calculation stage time consumption acquisition: uprobe and uretprobe are used to add mounting points at the host memory to GPU memory copying, which are used to acquire the time consumption of this stage. Uprobe and uretprobe are used to add mounting points at the inference calculation kernel, which are used to acquire the starting calculation kernel time consumption. Uprobe and uretprobe are used to add mounting points at the synchronization calculation result, which are used to acquire the inference calculation stage time consumption. Uprobe and uretprobe are used to add mounting points at the GPU memory to host memory copying, which are used to acquire the time consumption of this stage.

[0055] De-noising: The start time of the same function call entry record and the end time of the exit record are associated by stack backtracking identification; the required data is obtained through the direction parameter of data copying between the host memory and the GPU memory; and the non-computing task data is filtered out through the PID of the inference computing task.

[0056] Event processing engine: The data collected in the above two stages is uploaded to the event processing engine through eBPF map. Each piece of data contains three parts of event type, start time and end time. The data is divided into multiple sets according to the event type. First, use the total time set of the inference computing task to divide the multiple event sets into multiple inference task sets. Then, process the event sets in order. The method of sequential processing: traverse each set, associate the data in the time range as an event through the start time and end time of the event, form a CPU&GPU associated event set, and provide it to the user, which provides the basis for performance bottleneck analysis.

[0057] The specific process is shown in Figure 2 Step 1: Collect inference events: The event collection device captures all original events occurring on the CPU and GPU in the inference process in real time through eBPF uprobe / kprobe and other mechanisms.

[0058] Step 2: Divide the set according to the event type: classify the original events according to the event type to form several parallel event sets: A-host memory allocation, B-model loading, C-GPU video memory allocation, D-host memory to GPU video memory copy, E-GPU kernel start, F-GPU calculation result synchronization, and G-GPU video memory to host memory copy.

[0059] Step 3: Cut according to the total time of the inference task: use the total time of the inference task to cut all events again to ensure that each event only belongs to a single inference task, and obtain a “task-level” sub-set.

[0060] Step 4: Traverse all sub-sets according to the order of event occurrence: for each inference task, the engine traverses each type of event sub-set in the order of “host memory allocation→model loading→GPU video memory allocation→host memory to GPU video memory copy→GPU kernel start→GPU calculation result synchronization→GPU video memory to host memory copy”.

[0061] Step 5: Extract the end time of the current event: locate the end time (endtime) of the current event after the current event.

[0062] Step 6: Generate the expected time window of the next type of event: based on the end time (endtime), add a configurable offset (OFFSET_NS, used to tolerate minor time difference) to generate the expected time window of the next type of event.

[0063] Step 7: Determine whether there is an event in the next type of event subset whose start time (starttime) falls within the expected time window, if so, re-enter step 5 to continue to link the next type of event backward, if not, enter step 8.

[0064] Step 8: Form a joint event output.

[0065] It can be seen that the embodiment is based on the eBPF technology, combines uprobe and kprobe, and collects inference calculation events in the user state and the kernel state in the inference calculation process. The events are uploaded to the event processing engine after removing the noise by various means. The processing engine combines the dispersed events to form a CPU and GPU joint observation event. The scheme uses a lightweight new technology to collect inference data at the function level, refines the observation granularity, covers all aspects of event occurrence, has comprehensive observation coverage, reduces context switching, has low performance overhead, and does not need to modify the kernel. The system is stable. The scheme jointly observes the dispersed events, and improves the efficiency of performance bottleneck positioning in the AI scene.

[0066] Next, a task observation device provided by the embodiment of the application is introduced. The task observation device described below can be referred to with the task observation method described above. Referring to Figure 3 , a structural diagram of a task observation device according to an example embodiment is shown.

[0067] The determination module 100 is configured to collect the start time and the end time of each inference task, and determine the time range of each inference task according to the start time and the end time of each inference task.

[0068] The collection module 200 is configured to collect the start time and the end time of each type of event in the initialization stage and the inference calculation stage.

[0069] The association module 300 is configured to determine each type of event corresponding to each inference task according to the time range of each inference task and the start time and the end time of each type of event, and form an event queue of each inference task according to the occurrence order of each type of event.

[0070] The task observation device provided by the embodiment of the application can comprehensively cover the cooperation process of the inference task between the host and the accelerator by jointly observing various events executed in the host and the accelerator. The joint observation method not only includes the start and end time of the inference task, but also covers key events in the task initialization stage and the inference calculation stage, thereby providing more comprehensive performance analysis data. In addition, the execution flow of the inference task can be clearly displayed by forming an event queue according to the occurrence sequence of various events, which facilitates the identification of performance bottlenecks and fault points. Finally, the method significantly improves the comprehensiveness and accuracy of task observation by jointly observing events of the host and the accelerator, provides stronger support for performance optimization and fault positioning of the AI inference system, and improves the overall performance and reliability of the system.

[0071] On the basis of the above-mentioned embodiment, as a preferred implementation manner, the events in the initialization stage include any one or combination of the host memory allocation event, the model loading event, and the accelerator video memory allocation event; the events in the inference calculation stage include any one or combination of the host memory to accelerator video memory copy event, the accelerator kernel start event, the accelerator calculation result synchronization event, and the accelerator video memory to host memory copy event.

[0072] On the basis of the above-mentioned embodiment, as a preferred implementation manner, the determination module 100 is specifically configured to set a mounting point at the entrance and exit of the main function of the user-mode inference service, so as to collect the start time and the end time of each inference task; correspondingly, the collection module 200 is specifically configured to set a mounting point at the entrance and exit of the memory mapping system call, so as to collect the start time and the end time of the host memory allocation event; set a mounting point at the entrance and exit of the file system opening call, so as to collect the start time and the end time of the model loading event; set a mounting point at the entrance and exit of the video memory allocation function, so as to collect the start time and the end time of the accelerator video memory allocation event; set a mounting point at the entrance and exit of the memory copy function, so as to collect the start time and the end time of the host memory to accelerator video memory copy event and the accelerator video memory to host memory copy event; set a mounting point at the entrance and exit of the kernel start function, so as to collect the start time and the end time of the accelerator kernel start event; and set a mounting point at the entrance and exit of the device synchronization function, so as to collect the start time and the end time of the accelerator calculation result synchronization event.

[0073] On the basis of the above-mentioned embodiment, as a preferred implementation manner, the collection module 200 is further configured to distinguish the host memory to accelerator video memory copy event and the accelerator video memory to host memory copy event according to the direction parameter in the memory copy function.

[0074] On the basis of the above-mentioned embodiments, as a preferred implementation manner, the collection module 200 is further configured to calculate a hash value of the event according to the process identification and the stack backtracking identification of the event, and match the start time and the end time of the same event according to the hash value of the event at each mounting point.

[0075] On the basis of the above-mentioned embodiments, as a preferred implementation manner, the determination module 100 is further configured to collect a process identification of the inference task at a main function entrance of the user-mode inference service, and store the process identification into a process set; correspondingly, the association module 300 is specifically configured to: determine each process identification in the process set as a target process identification in sequence, and determine a target inference task corresponding to the target process identification; determine an event as a candidate event corresponding to the target inference task, if the process identification is the target process identification and the start time and the end time of the event are both within a time range of the target inference task; and splice the start time and the end time of each type of event in sequence to determine each type of event corresponding to the target inference task in the candidate event.

[0076] On the basis of the above-mentioned embodiments, as a preferred implementation manner, the association module 300 is specifically configured to: divide the candidate event corresponding to the target inference task into different sets according to the event type; splice the start time and the end time of each type of event in the initialization phase in sequence to determine each type of event in the initialization phase corresponding to the target inference task in the set corresponding to the event in the initialization phase; and splice the start time and the end time of each type of event in the inference calculation phase in sequence to determine each type of event in the inference calculation phase corresponding to the target inference task in the set corresponding to the event in the inference calculation phase.

[0077] On the basis of the above-mentioned embodiments, as a preferred implementation manner, the association module 300 is specifically configured to: determine a target host memory allocation event corresponding to the target inference task in the set corresponding to the event of the host memory allocation according to the start time of the target inference task; determine a target model loading event corresponding to the target inference task in the set corresponding to the event of the model loading according to the end time of the target host memory allocation event and the offset, the target model loading event being a model loading event whose start time belongs to a model loading event time range; and determine a target accelerator video memory allocation event corresponding to the target inference task in the set corresponding to the event of the accelerator video memory allocation according to the end time of the target model loading event and the offset, the target accelerator video memory allocation event being an accelerator video memory allocation event whose start time belongs to an accelerator video memory allocation event time range.

[0078] On the basis of the above-mentioned embodiments, as a preferred implementation, the association module 300 is specifically configured to: determine, according to the end time of the target accelerator video memory allocation event and the offset, an accelerator memory-to-video memory copy event time range, determine, in a set of accelerator memory-to-video memory copy events corresponding to the accelerator memory-to-video memory copy event, an accelerator memory-to-video memory copy event whose start time belongs to the accelerator memory-to-video memory copy event time range as a target accelerator memory-to-video memory copy event corresponding to the target reasoning task; determine, according to the end time of the target accelerator memory-to-video memory copy event and the offset, an accelerator kernel start event time range, determine, in a set of accelerator kernel start events corresponding to the accelerator kernel start event, an accelerator kernel start event whose start time belongs to the accelerator kernel start event time range as a target accelerator kernel start event corresponding to the target reasoning task; determine, according to the end time of the target accelerator kernel start event and the offset, an accelerator calculation result synchronization event time range, determine, in a set of accelerator calculation result synchronization events corresponding to the accelerator calculation result synchronization event, an accelerator calculation result synchronization event whose start time belongs to the accelerator calculation result synchronization event time range as a target accelerator calculation result synchronization event corresponding to the target reasoning task; and determine, according to the end time of the target accelerator calculation result synchronization event and the offset, an accelerator video memory-to-host memory copy event time range, determine, in a set of accelerator video memory-to-host memory copy events corresponding to the accelerator video memory-to-host memory copy event, an accelerator video memory-to-host memory copy event whose start time belongs to the accelerator video memory-to-host memory copy event time range as a target accelerator video memory-to-host memory copy event corresponding to the target reasoning task.

[0079] As to the apparatus in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0080] Embodiments of the present application also provide an electronic device, Figure 4 To show the structure of an electronic device according to an exemplary embodiment, as shown in Figure 4 The electronic device includes: a communication interface 1 capable of information interaction with other devices such as network devices; a processor 2 connected with the communication interface 1 to realize information interaction with other devices, for running a computer program, performing the task observation method provided by one or more technical solutions described above. And the computer program is stored on the memory 3.

[0081] Of course, in actual application, each component in the electronic device is coupled together through a bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 4 also includes a power bus, a control bus and a state signal bus. However, in order to clearly illustrate, all kinds of buses are marked as bus system 4 in Figure 4 .

[0082] The memory 3 in embodiments of the application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer programs for operating on the electronic device.

[0083] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), direct rambus random access memory (DRRAM).The memory 3 described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0084] The method disclosed in the embodiments of the present application can be applied in the processor 2 or implemented by the processor 2. The processor 2 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by integrated logic circuits of hardware in the processor 2 or instructions in the form of software. The processor 2 described above can be a general processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the execution can be directly completed by a hardware decoding processor or a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 3. The processor 2 reads the program in the memory 3 and combines the hardware to complete the steps of the above method.

[0085] The processor 2 executes the program to implement the corresponding flow in each method of the embodiments of the present application. For the sake of brevity, it will not be repeated here.

[0086] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is arranged to execute the steps in any of the above task observation method embodiments when running.

[0087] In an exemplary embodiment, the above computer readable storage medium can include, but not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic or optical disk, and various media that can store computer programs.

[0088] The embodiments of the present application also provide a computer program product, which includes a computer program. The computer program is executed by the processor 2 to implement the steps in any of the above task observation method embodiments.

[0089] The embodiments of the present application also provide another computer program product, which includes a non-volatile computer readable storage medium. The non-volatile computer readable storage medium stores a computer program. The computer program is executed by the processor 2 to implement the steps in any of the above task observation method embodiments.

[0090] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0091] The foregoing has provided a detailed description of the mission observation system, method, apparatus, equipment, medium, and product provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only intended to aid in understanding the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A task observation method characterized by comprising: The method comprises the following steps: collecting start time and end time of each inference task, and determining time range of each inference task according to the start time and the end time of each inference task; collecting start time and end time of each event in an initialization stage and a reasoning calculation stage in a life cycle of each inference task; wherein, the events in the initialization stage comprise any one or combination of host memory allocation event, model loading event, and accelerator memory allocation event, and the events in the reasoning calculation stage comprise any one or combination of host memory to accelerator memory copy event, accelerator kernel start event, accelerator calculation result synchronization event, and accelerator memory to host memory copy event; determining each event corresponding to each inference task according to the time range of each inference task and the start time and the end time of each event, and forming an event queue of each inference task according to the occurrence order of each event; wherein, the method further comprises: collecting process identifier of the inference task at a main function entrance of the user-mode inference service, and storing the process identifier into a process set; correspondingly, determining each event corresponding to each inference task according to the time range of each inference task and the start time and the end time of each event comprises: determining each process identifier in the process set as target process identifier in sequence, and determining target inference task corresponding to the target process identifier; determining an event whose process identifier is the target process identifier and whose start time and end time are both within the time range of the target inference task as candidate event corresponding to the target inference task; sequentially connecting the start time and the end time of each event at the beginning and the end to determine each event corresponding to the target inference task in the candidate event according to the occurrence order of each event; wherein, the method further comprises: calculating hash value of the event according to the process identifier and the stack backtracking identifier of the event, and matching the start time and the end time of the same event according to the hash value of the event at each mounting point.

2. The task observation method according to claim 1, wherein collecting start time and end time of each inference task comprises: setting mounting points at a main function entrance and exit of the user-mode inference service to collect the start time and the end time of each inference task; correspondingly, collecting start time and end time of each event in the initialization stage and the reasoning calculation stage comprises: setting mounting points at an entrance and exit of a memory mapping system call to collect the start time and the end time of the host memory allocation event; setting mounting points at an entrance and exit of an open file system call to collect the start time and the end time of the model loading event; setting mounting points at an entrance and exit of an accelerator memory allocation function to collect the start time and the end time of the accelerator memory allocation event; setting mounting points at an entrance and exit of a memory copy function to collect the start time and the end time of the host memory to accelerator memory copy event and the accelerator memory to host memory copy event; setting mounting points at an entrance and exit of a kernel start function to collect the start time and the end time of the accelerator kernel start event; Mount points are set at the entrance and exit of the device synchronization function to collect the start time and end time of the accelerator computing result synchronization event.

3. The task observation method according to claim 2, wherein Further comprising: According to the direction parameter in the memory copy function, distinguish the host memory to accelerator memory copy event and the accelerator memory to host memory copy event.

4. The task observation method according to claim 1, wherein According to the occurrence order of each type of event, the start time and end time of each type of event are connected at the beginning and end, so as to determine the events corresponding to the target reasoning task in the candidate events, including: According to the event type, the candidate events corresponding to the target reasoning task are divided into different sets; According to the occurrence order of each type of event in the initialization phase, the start time and end time of each type of event in the initialization phase are connected at the beginning and end, so as to determine the events corresponding to the target reasoning task in the initialization phase in the set corresponding to each type of event in the initialization phase, respectively. According to the occurrence order of each type of event in the reasoning calculation phase, the start time and end time of each type of event in the reasoning calculation phase are connected at the beginning and end, so as to determine the events corresponding to the target reasoning task in the reasoning calculation phase in the set corresponding to each type of event in the reasoning calculation phase, respectively.

5. The task observation method according to claim 4, wherein According to the occurrence order of each type of event in the initialization phase, the start time and end time of each type of event in the initialization phase are connected at the beginning and end, so as to determine the events corresponding to the target reasoning task in the initialization phase in the set corresponding to each type of event in the initialization phase, respectively, including: According to the start time of the target reasoning task, determine the target host memory allocation event corresponding to the target reasoning task in the set corresponding to the host memory allocation event; According to the end time of the target host memory allocation event and the offset, determine the model loading event time range, and determine the model loading event whose start time belongs to the model loading event time range as the target model loading event corresponding to the target reasoning task in the set corresponding to the model loading event; According to the end time of the target model loading event and the offset, determine the accelerator memory allocation event time range, and determine the accelerator memory allocation event whose start time belongs to the accelerator memory allocation event time range as the target accelerator memory allocation event corresponding to the target reasoning task in the set corresponding to the accelerator memory allocation event.

6. The task observation method according to claim 5, wherein According to the occurrence order of each type of event in the reasoning calculation phase, the start time and end time of each type of event in the reasoning calculation phase are connected at the beginning and end, so as to determine the events corresponding to the target reasoning task in the reasoning calculation phase in the set corresponding to each type of event in the reasoning calculation phase, respectively, including: According to the end time of the target accelerator memory allocation event and the offset, determine the host memory to accelerator memory copy event time range, and determine the host memory to accelerator memory copy event whose start time belongs to the host memory to accelerator memory copy event time range as the target host memory to accelerator memory copy event corresponding to the target reasoning task in the set corresponding to the host memory to accelerator memory copy event. determine an accelerator kernel start event time range according to the end time of the target host memory to accelerator memory copy event and the offset, determine, in a set of accelerator kernel start events corresponding to the accelerator kernel start event, an accelerator kernel start event whose start time belongs to the accelerator kernel start event time range as a target accelerator kernel start event corresponding to the target inference task; determine an accelerator computation result synchronization event time range according to the end time of the target accelerator kernel start event and the offset, determine, in a set of accelerator computation result synchronization events corresponding to the accelerator computation result synchronization event, an accelerator computation result synchronization event whose start time belongs to the accelerator computation result synchronization event time range as a target accelerator computation result synchronization event corresponding to the target inference task; determine an accelerator memory to host memory copy event time range according to the end time of the target accelerator computation result synchronization event and the offset, determine, in a set of accelerator memory to host memory copy events corresponding to the accelerator memory to host memory copy event, an accelerator memory to host memory copy event whose start time belongs to the accelerator memory to host memory copy event time range as a target accelerator memory to host memory copy event corresponding to the target inference task.

7. An electronic device, comprising: comprise: a memory for storing a computer program; a processor for implementing the steps performed by the task observation method according to any one of claims 1 to 6 when executing the computer program.

Images