RTOS remote debugging and kernel tracking method, device, equipment and storage medium

Abstract

The application relates to an RTOS remote debugging and kernel tracking method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: after the RTOS device is powered on, initializing a memory double ring buffer, loading a non-invasive static tracking point sandbox, registering a kernel event dynamic probe on demand, executing on-demand mounting of a kernel event collection point through a collection function and a tracking overhead real-time self-checking function; performing local filtering sampling on the kernel events captured through the kernel event dynamic probe, and writing the qualified kernel events screened out into a first-level high-speed interrupt cache in the memory double ring buffer through light coding; completing TLS handshaking and identity authentication with a remote server through a light debugging agent carried thereon, so that a secure long connection between the remote server and the RTOS device is established, and a tracking binary stream in the first-level high-speed interrupt cache is reported to the remote server in blocks. According to the method, the invasiveness of RTOS debugging and kernel tracking can be reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for remote debugging and kernel tracing of an RTOS. Background Technology

[0002] With the rapid popularization of embedded scenarios such as Industrial IoT, smart wearables, automotive terminals, and edge control devices, real-time operating systems (RTOS) such as FreeRTOS, RT-Thread, uC / OS, and HarmonyOS Lite have become the core scheduling platform for small, resource-constrained MCU (Microcontroller Unit) devices. During the device R&D, debugging, and mass production line troubleshooting phases, developers need to collect the entire runtime trajectory of the RTOS kernel scheduling, interrupt response, and memory allocation to locate timing-related hidden faults such as task blocking, priority inversion, stack overflow, memory leaks, and interrupt storms. Kernel tracing and remote debugging tools are key support methods for ensuring the stability of embedded devices.

[0003] Currently, the mainstream RTOS debugging and kernel tracing solutions in the industry are divided into three categories: local hardware debugging, local software tracing, and simple remote debugging. However, each solution has obvious shortcomings in terms of real-time performance, intrusiveness, remote operation and maintenance, resource consumption, security management, and automated fault analysis, making it difficult to adapt to the online operation and maintenance scenarios of mass-produced large-scale terminals. For example, traditional kernel tracing methods are highly invasive, severely disrupting the real-time scheduling sequence of the RTOS and making fault reproduction difficult. Traditional local debugging methods include serial port printf (print formatted) log printing, kernel static hooks, and JTAG (Joint Test Action Group) hardware breakpoints. Serial port printing continuously preempts the system bus and task time slices, changing the original task scheduling sequence and easily masking occasional deadlocks and priority inversions that only occur under specific timing conditions. Kernel static hooks require modifying the RTOS kernel source code and recompiling the entire firmware, making porting to multiple models and versions of RTOS devices extremely costly and impossible to temporarily enable tracing on mass-produced terminals where firmware upgrades are not allowed. JTAG hardware breakpoint debugging forcibly pauses the CPU (Central Processing Unit) operation, interrupting continuous business data flow, and can only capture static instantaneous states. It cannot fully record dynamic abnormal behaviors such as millisecond-level continuous scheduling glitches and high-frequency interrupt overloads, making it difficult to locate occasional stability problems. Therefore, among traditional technologies, RTOS debugging and kernel tracing are highly invasive. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for remote debugging and kernel tracing of RTOS that can reduce the intrusiveness of RTOS debugging and kernel tracing, in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for remote debugging and kernel tracing of an RTOS, including:

[0006] After the RTOS device is powered on, the memory dual-ring buffer is initialized, the preset non-intrusive static trace point sandbox is loaded, the kernel event dynamic probe is registered as needed, and the kernel event collection point is mounted as needed through the preset collection function and the real-time self-test function of the trace overhead.

[0007] Local filtering and sampling processing is performed on the kernel events captured by the kernel event dynamic probe. The qualified kernel events selected are written into the first-level high-speed interrupt cache in the dual-ring buffer of memory using lightweight encoding. The first-level high-speed interrupt cache stores the trace binary stream formed by the qualified kernel events after lightweight encoding.

[0008] The system completes TLS handshake and authentication with the remote server through the lightweight debugging agent to establish a secure long connection with the remote server, and reports the trace binary stream in the first-level high-speed interrupt cache to the remote server in blocks.

[0009] The remote server is used to concurrently receive tracking data packets from multiple RTOS devices based on the segmentation of the tracking binary stream via multiple connection coroutines; to perform verification, fragmentation, and reassembly of the tracking data packets, and to write the reassembled binary events into a time-series database for storage; to load the firmware symbol table of the corresponding RTOS device, to parse the binary events stored in the time-series database, to call built-in calculation functions to generate scheduling performance quantification indicators in batches, and to perform anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly tags; to input the scheduling performance quantification indicators and the anomaly tags into a preset fault rule matching inference model to obtain the corresponding fault root cause, and to generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

[0010] In one embodiment, initializing the memory dual-ring buffer after the RTOS device is powered on includes:

[0011] After the RTOS device is powered on, two independent cache memory blocks are statically partitioned in the RTOS heap of the RTOS device to obtain a dual-ring buffer memory.

[0012] The two independent cache memory blocks include a first-level high-speed interrupt cache and a second-level persistent asynchronous cache. The first-level high-speed interrupt cache is used for zero-copy write events of interrupt context, and the second-level persistent asynchronous cache is used for low-priority tracking task asynchronous transport events. The cache structure uses atomic variables as read and write indexes.

[0013] In one embodiment, the preset non-intrusive static tracking point sandbox only reserves empty jump entry points and provides two sets of non-blocking APIs to support dynamic addition and deletion of acquisition probes via remote commands at runtime; when executing logic, the preset non-intrusive static tracking point sandbox only performs data copying and atomic index auto-increment, and has three built-in components; the three components include: a global probe hash chain, an atomic read / write two-level circular cache, and a high-precision timing read assembly function.

[0014] In one embodiment, the step of performing local filtering sampling on kernel events captured by the kernel event dynamic probe, and writing the selected qualified kernel events into the first-level high-speed interrupt cache in the dual-ring memory buffer using lightweight encoding, includes:

[0015] When the mounted trace point probe is triggered, the original event fields of the captured kernel event are collected through the preset non-intrusive static trace point sandbox based on atomic write operations; the original event fields include timestamp, event type enumeration, task identifier, interrupt number, stack remaining bytes, and memory operation size;

[0016] The system uses a built-in three-layer serial filtering function and selects qualified kernel events from the captured kernel events based on the original event fields. The three-layer serial filtering function includes a task priority filtering function, a high-frequency event downsampling function, and a task whitelist filtering function.

[0017] The qualified kernel events are encoded using a preset encoding module and written into the first-level high-speed interrupt cache in the dual-ring buffer of memory.

[0018] In one embodiment, the method further includes:

[0019] When it is determined that the write index of the first-level high-speed interrupt cache will catch up with the read index, the overflow protection logic is triggered, a global overflow flag is written in the header of the event packet, and it is reported to the remote server. After receiving the global overflow flag, the remote server will prompt the developer that the current cache capacity is insufficient and issue a remote instruction to the RTOS device to expand the second-level persistent asynchronous cache.

[0020] Based on the lowest priority transport task created during the RTOS device initialization, the system reads valid events from the first-level high-speed interrupt cache in a non-blocking loop and copies them in batches to the second-level persistent asynchronous cache.

[0021] In one embodiment, the block reporting of the trace binary stream in the Level 1 high-speed interrupt cache to the remote server includes:

[0022] Extract the tracking binary stream from the first-level high-speed interrupt cache;

[0023] Determine the target segmentation method that matches the tracking binary stream, and segment the tracking binary stream according to the target segmentation method to obtain the corresponding tracking data packets;

[0024] The tracking data packet is reported to the remote server.

[0025] Secondly, this application also provides an RTOS remote debugging and kernel tracing device, comprising:

[0026] The function execution module is used to initialize the memory double ring buffer after the RTOS device is powered on, load the preset non-intrusive static trace point sandbox, register the kernel event dynamic probes as needed, and perform on-demand mounting of kernel event collection points through preset collection functions and real-time self-check functions for trace overhead.

[0027] The event encoding module is used to perform local filtering and sampling processing on the kernel events captured by the kernel event dynamic probe, and to write the qualified kernel events selected into the first-level high-speed interrupt cache in the dual-ring buffer of memory using lightweight encoding; the first-level high-speed interrupt cache stores the trace binary stream formed by the qualified kernel events after lightweight encoding;

[0028] The connection establishment module is used to complete TLS handshake and identity authentication with the remote server through the built-in lightweight debugging agent in order to establish a secure long connection with the remote server, and to report the trace binary stream in the first-level high-speed interrupt cache to the remote server in blocks;

[0029] The remote server is used to concurrently receive tracking data packets from multiple RTOS devices based on the segmentation of the tracking binary stream via multiple connection coroutines; to perform verification, fragmentation, and reassembly of the tracking data packets, and to write the reassembled binary events into a time-series database for storage; to load the firmware symbol table of the corresponding RTOS device, to parse the binary events stored in the time-series database, to call built-in calculation functions to generate scheduling performance quantification indicators in batches, and to perform anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly tags; to input the scheduling performance quantification indicators and the anomaly tags into a preset fault rule matching inference model to obtain the corresponding fault root cause, and to generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

[0030] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0031] After the RTOS device is powered on, the memory dual-ring buffer is initialized, the preset non-intrusive static trace point sandbox is loaded, the kernel event dynamic probe is registered as needed, and the kernel event collection point is mounted as needed through the preset collection function and the real-time self-test function of the trace overhead.

[0032] Local filtering and sampling processing is performed on the kernel events captured by the kernel event dynamic probe. The qualified kernel events selected are written into the first-level high-speed interrupt cache in the dual-ring buffer of memory using lightweight encoding. The first-level high-speed interrupt cache stores the trace binary stream formed by the qualified kernel events after lightweight encoding.

[0033] The system completes TLS handshake and authentication with the remote server through the lightweight debugging agent to establish a secure long connection with the remote server, and reports the trace binary stream in the first-level high-speed interrupt cache to the remote server in blocks.

[0034] The remote server is used to concurrently receive tracking data packets from multiple RTOS devices based on the segmentation of the tracking binary stream via multiple connection coroutines; to perform verification, fragmentation, and reassembly of the tracking data packets, and to write the reassembled binary events into a time-series database for storage; to load the firmware symbol table of the corresponding RTOS device, to parse the binary events stored in the time-series database, to call built-in calculation functions to generate scheduling performance quantification indicators in batches, and to perform anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly tags; to input the scheduling performance quantification indicators and the anomaly tags into a preset fault rule matching inference model to obtain the corresponding fault root cause, and to generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

[0035] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0036] After the RTOS device is powered on, the memory dual-ring buffer is initialized, the preset non-intrusive static trace point sandbox is loaded, the kernel event dynamic probe is registered as needed, and the kernel event collection point is mounted as needed through the preset collection function and the real-time self-test function of the trace overhead.

[0037] Local filtering and sampling processing is performed on the kernel events captured by the kernel event dynamic probe. The qualified kernel events selected are written into the first-level high-speed interrupt cache in the dual-ring buffer of memory using lightweight encoding. The first-level high-speed interrupt cache stores the trace binary stream formed by the qualified kernel events after lightweight encoding.

[0038] The system completes TLS handshake and authentication with the remote server through the lightweight debugging agent to establish a secure long connection with the remote server, and reports the trace binary stream in the first-level high-speed interrupt cache to the remote server in blocks.

[0039] The remote server is used to concurrently receive tracking data packets from multiple RTOS devices based on the segmentation of the tracking binary stream via multiple connection coroutines; to perform verification, fragmentation, and reassembly of the tracking data packets, and to write the reassembled binary events into a time-series database for storage; to load the firmware symbol table of the corresponding RTOS device, to parse the binary events stored in the time-series database, to call built-in calculation functions to generate scheduling performance quantification indicators in batches, and to perform anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly tags; to input the scheduling performance quantification indicators and the anomaly tags into a preset fault rule matching inference model to obtain the corresponding fault root cause, and to generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

[0040] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0041] After the RTOS device is powered on, the memory dual-ring buffer is initialized, the preset non-intrusive static trace point sandbox is loaded, the kernel event dynamic probe is registered as needed, and the kernel event collection point is mounted as needed through the preset collection function and the real-time self-test function of the trace overhead.

[0042] Local filtering and sampling processing is performed on the kernel events captured by the kernel event dynamic probe. The qualified kernel events selected are written into the first-level high-speed interrupt cache in the dual-ring buffer of memory using lightweight encoding. The first-level high-speed interrupt cache stores the trace binary stream formed by the qualified kernel events after lightweight encoding.

[0043] The system completes TLS handshake and authentication with the remote server through the lightweight debugging agent to establish a secure long connection with the remote server, and reports the trace binary stream in the first-level high-speed interrupt cache to the remote server in blocks.

[0044] The remote server is used to concurrently receive tracking data packets from multiple RTOS devices based on the segmentation of the tracking binary stream via multiple connection coroutines; to perform verification, fragmentation, and reassembly of the tracking data packets, and to write the reassembled binary events into a time-series database for storage; to load the firmware symbol table of the corresponding RTOS device, to parse the binary events stored in the time-series database, to call built-in calculation functions to generate scheduling performance quantification indicators in batches, and to perform anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly tags; to input the scheduling performance quantification indicators and the anomaly tags into a preset fault rule matching inference model to obtain the corresponding fault root cause, and to generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

[0045] The aforementioned RTOS remote debugging and kernel tracing method, apparatus, computer device, computer-readable storage medium, and computer program product, after the RTOS device is powered on, initialize a dual-ring buffer in memory, load a preset non-intrusive static trace point sandbox, register dynamic kernel event probes as needed, and execute on-demand mounting of kernel event collection points through preset acquisition functions and real-time self-checking functions for tracing overhead. Next, local filtering and sampling processing is performed on the kernel events captured by the dynamic kernel event probes. Qualified kernel events are lightweight-encoded and written to the first-level high-speed interrupt buffer in the dual-ring buffer in memory. The first-level high-speed interrupt buffer stores the trace binary stream formed by the lightweight encoding of qualified kernel events. Finally, a lightweight debugging agent is used to complete a TLS handshake and authentication with the remote server to establish a secure long-term connection with the remote server. The system then reports the trace binary stream in the first-level high-speed interrupt buffer to a remote server in chunks. The remote server concurrently receives trace data packets from multiple RTOS devices based on chunks of the trace binary stream via multiple connection coroutines. The trace data packets are verified, fragmented, and reassembled. The reassembled binary events are written to a time-series database for storage. The firmware symbol table of the corresponding RTOS device is loaded, the binary events stored in the time-series database are parsed, and built-in calculation functions are called to generate scheduling performance quantification indicators in batches. Anomalies are identified in the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly labels. The scheduling performance quantification indicators and anomaly labels are input into a preset fault rule matching inference model to obtain the corresponding fault root cause and generate visualized structured data. The visualized structured data includes a time-series timeline view and a performance statistics view.In this way, by pre-installing collection points in a non-intrusive static tracing sandbox, kernel event collection and mounting can be completed without modifying the RTOS kernel's native source code or pausing the processor kernel. This completely avoids the highly invasive problems associated with traditional methods such as printf printing, JTAG breakpoint debugging, and kernel source code instrumentation, effectively preventing tracing and collection operations from disturbing the RTOS kernel's native scheduling timing and running state. Simultaneously, through lightweight collection and processing logic using local filtering sampling, lightweight encoding, and dual-ring cache hierarchical storage, CPU overhead and system latency during kernel event collection, encoding, and storage are significantly reduced. This enables real-time self-checking and controllable tracing overhead, ensuring the RTOS system's performance. It ensures real-time performance and operational stability. Furthermore, relying on a remote debugging architecture based on secure long-connection block reporting, server-side concurrent parsing, and intelligent fault reasoning, it achieves non-intrusive kernel event collection, reliable transmission, accurate analysis, and visualization. It eliminates the need for high-frequency debugging operations and manual intervention on the device, greatly reducing the operational intrusion and resource consumption overhead of RTOS kernel tracing and device debugging. It effectively solves the problems of traditional debugging methods that easily cause system timing disorders, large debugging disturbances, and inability to adapt to long-term online tracing and debugging. It significantly reduces the intrusion of RTOS debugging and kernel tracing, while greatly improving the real-time performance, stability, and debugging accuracy of RTOS system kernel tracing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a diagram illustrating the application environment of an RTOS remote debugging and kernel tracing method in one embodiment.

[0048] Figure 2 This is a flowchart illustrating the RTOS remote debugging and kernel tracing method in another embodiment;

[0049] Figure 3 This is a block diagram of a remote debugging and kernel tracing device for an RTOS in one embodiment;

[0050] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] Current mainstream RTOS debugging and kernel tracing solutions in the industry, besides having the drawback of highly intrusive kernel tracing methods, also suffer from the following shortcomings:

[0053] First, the tracking data only supports local parsing. Mass-produced devices without debugging interfaces cannot remotely collect kernel traces, and occasional offline faults cannot be reproduced. Mainstream professional tracking tools such as Tracealyzer and SystemView rely on hardware probes such as J-Link and ST-Link to directly connect to the device, which is only suitable for debugging laboratory prototypes. Mass-produced embedded terminals generally eliminate debugging probe interfaces to control hardware costs. When online devices experience crashes, freezes, or disconnections, only a small amount of text logs can be stored, losing complete task switching, interrupt timing, and memory change processes. At the same time, the local tracking buffer capacity of the device is limited by the extremely small RAM (Random Access Memory) resources of the MCU. Massive tasks and interrupt events can easily cause buffer overflows, and critical abnormal events are directly discarded. Existing tools do not have a unified remote data aggregation service, and distributed terminals in multiple production lines and regions cannot achieve parallel batch tracking and debugging. On-site fault diagnosis must be carried out offline by disassembling the device, resulting in extremely low operation and maintenance efficiency.

[0054] Secondly, the lack of dynamic load control mechanism in tracking and acquisition leads to computing power and power consumption overload issues for resource-constrained MCUs during long-term operation. Many low-end Cortex-M0 / M3 MCUs, powered by batteries and with limited computing power, possess only tens of KB of memory and low-speed processing units. Existing tracking solutions default to continuously collecting all kernel events without tiered filtering or dynamic sampling logic. Furthermore, existing tools do not support on-demand start / stop tracking; prolonged acquisition significantly increases device power consumption, shortens battery life, and consumes computing resources for control algorithms and business logic, potentially triggering new operational failures.

[0055] Third, the remote debugging link lacks a complete security isolation mechanism, posing risks of firmware and kernel information leakage and unauthorized manipulation to mass-produced online devices. Existing simple remote debugging methods are mostly based on bare TCP (Transmission Control Protocol), serial port pass-through, and unauthenticated GDB (GNU Debugger) remote protocols, without hierarchical permission control and transmission encryption. The raw data stream contains core sensitive firmware information such as task handles, memory addresses, and function symbols, and plaintext transmission can easily lead to intellectual property leakage. At the same time, existing solutions lack a complete operation auditing system, and all remote debugging operations are not recorded, making it impossible to trace the source of operations in case of unauthorized access or device malfunction.

[0056] Fourth, the kernel's raw trace data only provides the underlying timestamps and raw digital IDs (identifiers), lacking automated quantification and root cause analysis capabilities, making manual parsing extremely inefficient. Existing trace tools can only convert raw binary event data into basic time-series text, without built-in standardized performance calculation functions and fault diagnosis models. Developers need to manually translate task IDs and memory addresses one by one according to the firmware ELF (Executable and Linkable Format) symbol table, and manually calculate performance indicators such as task blocking time, CPU load, interrupt time, and stack water level. Locating a single deadlock or memory leak often takes several hours. Furthermore, there is no automatic anomaly alarm mechanism, and threshold exceedances and timing anomalies hidden in massive trace data require manual frame-by-frame investigation, resulting in extremely high labor costs in large-scale equipment batch maintenance scenarios.

[0057] Therefore, to address at least one of the aforementioned technical problems, there is an urgent need to provide a low-intrusion, lightweight RTOS remote debugging server and kernel tracing solution that supports remote aggregation, is secure and controllable, and possesses automated indicator calculation and fault analysis capabilities. Based on this, this application provides an RTOS remote debugging and kernel tracing method that can significantly reduce the invasiveness of RTOS debugging and kernel tracing, while greatly improving the real-time performance, stability, and debugging accuracy of RTOS system kernel tracing.

[0058] In one exemplary embodiment, such as Figure 1 As shown, a method for remote debugging and kernel tracing of an RTOS is provided. Taking the application of this method to an RTOS device (an embedded hardware device equipped with an RTOS) as an example, the method includes the following steps S101 to S103. Wherein:

[0059] Step S101: After the RTOS device is powered on, initialize the memory dual-ring buffer, load the preset non-intrusive static tracking point sandbox, register the kernel event dynamic probe as needed, and execute the on-demand mounting of kernel event acquisition points through the preset acquisition function and tracking overhead real-time self-test function.

[0060] Step S102: Perform local filtering and sampling processing on the kernel events captured by the kernel event dynamic probe, and write the qualified kernel events selected into the first-level high-speed interrupt buffer in the dual-ring buffer of memory using lightweight encoding; the first-level high-speed interrupt buffer stores the trace binary stream formed by the qualified kernel events after lightweight encoding.

[0061] Step S103: The lightweight debugging agent is used to complete the TLS (Transport Layer Security) handshake and authentication with the remote server to establish a secure long connection with the remote server, and the trace binary stream in the first-level high-speed interrupt cache is reported to the remote server in blocks.

[0062] The remote server is used to concurrently receive trace data packets from multiple RTOS devices based on trace binary stream chunks via multiple connection coroutines; it verifies, fragments, and reassembles the trace data packets, writes the reassembled binary events into a time-series database for storage; it loads the firmware symbol table of the corresponding RTOS device, parses the binary events stored in the time-series database, calls built-in calculation functions to generate scheduling performance quantification metrics in batches, and performs anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly labels; it inputs the scheduling performance quantification metrics and anomaly labels into a preset fault rule matching inference model to obtain the corresponding fault root cause and generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

[0063] For example, after the RTOS device is powered on, the memory dual-ring buffer is initialized, including: after the RTOS device is powered on, two independent cache memory blocks are statically partitioned in the RTOS heap of the RTOS device to obtain the memory dual-ring buffer; wherein, the two independent cache memory blocks include a level-one high-speed interrupt cache and a level-two persistent asynchronous cache. The level-one high-speed interrupt cache is used for zero-copy write events of interrupt context, and the level-two persistent asynchronous cache is used for low-priority tracking task asynchronous transport events; the cache structure uses atomic variables as read and write indexes.

[0064] For example, the preset non-intrusive static tracking point sandbox only reserves empty jump entry points and provides two sets of non-blocking APIs (Application Programming Interfaces) to support dynamic addition and deletion of acquisition probes via remote commands at runtime; when executing logic, the preset non-intrusive static tracking point sandbox only performs data copying and atomic index auto-increment, and has three built-in components; the three components include: a global probe hash chain, an atomic read / write two-level circular cache, and a high-precision timing read assembly function.

[0065] For example, local filtering and sampling processing is performed on kernel events captured by dynamic kernel event probes. Qualified kernel events are then written to a first-level high-speed interrupt cache in a dual-ring buffer using lightweight encoding. This process includes: when a mounted tracepoint probe is triggered, the original event fields of the captured kernel events are collected using an atomic write operation through a pre-defined non-intrusive static tracepoint sandbox; the original event fields include timestamp, event type enumeration, task identifier, interrupt number, remaining stack bytes, and memory operation size; qualified kernel events are selected from the captured kernel events using a built-in three-layer serial filtering function based on the original event fields; the three-layer serial filtering function includes a task priority filtering function, a high-frequency event downsampling function, and a task whitelist filtering function; and qualified kernel events are encoded using a pre-defined encoding module and written to a first-level high-speed interrupt cache in a dual-ring buffer.

[0066] For example, the RTOS remote debugging and kernel tracing method provided in this application further includes: when it is determined that the write index of the first-level high-speed interrupt cache will catch up with the read index, triggering overflow protection logic, writing a global overflow flag in the event packet header, and reporting it to the remote server, so that after the remote server receives the global overflow flag, it prompts the developer that the current cache capacity is insufficient, and issues a remote instruction to the RTOS device to expand the second-level persistent asynchronous cache; according to the lowest priority transfer task created during the initialization of the RTOS device, reading the valid events in the first-level high-speed interrupt cache in a non-blocking loop, and copying them in batches to the second-level persistent asynchronous cache.

[0067] For example, reporting the trace binary stream in the Level 1 high-speed interrupt buffer to a remote server in chunks includes: extracting the trace binary stream from the Level 1 high-speed interrupt buffer; determining a target chunking method matching the trace binary stream; chunking the trace binary stream according to the target chunking method to obtain corresponding trace data packets; and reporting the trace data packets to the remote server. For example, the current communication channel status can be obtained, the corresponding target chunking method can be determined based on the current communication channel status, and the trace binary stream can be chunked according to the target chunking method to obtain trace data packets. Alternatively, the corresponding target chunking method can be determined based on the current communication channel status and the data volume of the trace binary stream to be sent.

[0068] It should be noted that the RTOS remote debugging and kernel tracing method provided in this application relies on non-intrusive dynamic tracepoints, self-developed lightweight Protobuf encoding, layered remote transmission architecture, built-in quantization calculation functions, and a hierarchical security instruction control end-to-end design. This not only solves the underlying defects of traditional debugging tools, such as strong intrusion, disruption of RTOS real-time timing, support for only local hardware debugging, high acquisition overhead, and large bandwidth and memory consumption, but also addresses the pain point of low efficiency in manual log parsing through cloud-based automated indicator calculation and intelligent fault reasoning visualization. At the same time, it builds a security system of RBAC (Role-Based Access Control) hierarchical permissions and full operation auditing to avoid the risk of remote control of mass-produced equipment. The whole solution has extremely low hardware resource consumption, can be quickly ported to multiple types of RTOS, and takes into account both laboratory debugging and large-scale online mass production equipment remote fault diagnosis scenarios. It has significant industry advantages in five dimensions: real-time performance, lightweight design, remote operation and maintenance, security control, and automated fault location.

[0069] In the aforementioned RTOS remote debugging and kernel tracing method, after the RTOS device is powered on, a dual-ring buffer in memory is initialized, a preset non-intrusive static trace point sandbox is loaded, and dynamic kernel event probes are registered as needed. Through preset acquisition functions and real-time self-checking functions for tracing overhead, the kernel event acquisition points are mounted as needed. Next, local filtering and sampling processing is performed on the kernel events captured by the dynamic kernel event probes. Qualified kernel events are lightweight-encoded and written to the first-level high-speed interrupt buffer in the dual-ring buffer. The first-level high-speed interrupt buffer stores the trace binary stream formed by the lightweight encoding of qualified kernel events. Finally, a lightweight debugging agent is used to complete a TLS handshake and authentication with the remote server to establish a secure long-term connection, and the first-level high-speed interrupt buffer is reported in blocks. The system transmits the trace binary stream to a remote server. The remote server concurrently receives trace data packets from multiple RTOS devices, obtained by segmenting the trace binary stream, via multiple connection coroutines. The trace data packets are verified, fragmented, and reassembled. The reassembled binary events are written to a time-series database for storage. The system loads the firmware symbol table of the corresponding RTOS device, parses the binary events stored in the time-series database, calls built-in calculation functions to generate batch scheduling performance quantification metrics, and performs anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly labels. The scheduling performance quantification metrics and anomaly labels are input into a preset fault rule matching inference model to determine the corresponding root cause of the fault and generate visualized structured data. The visualized structured data includes a time-series timeline view and a performance statistics view. In this way, by pre-installing collection points in a non-intrusive static tracing sandbox, kernel event collection and mounting can be completed without modifying the RTOS kernel's native source code or pausing the processor kernel. This completely avoids the highly invasive problems associated with traditional methods such as printf printing, JTAG breakpoint debugging, and kernel source code instrumentation, effectively preventing tracing and collection operations from disturbing the RTOS kernel's native scheduling timing and running state. Simultaneously, through lightweight collection and processing logic using local filtering sampling, lightweight encoding, and dual-ring cache hierarchical storage, CPU overhead and system latency during kernel event collection, encoding, and storage are significantly reduced. This enables real-time self-checking and controllable tracing overhead, ensuring the RTOS system's performance. It ensures real-time performance and operational stability. Furthermore, relying on a remote debugging architecture based on secure long-connection block reporting, server-side concurrent parsing, and intelligent fault reasoning, it achieves non-intrusive kernel event collection, reliable transmission, accurate analysis, and visualization. It eliminates the need for high-frequency debugging operations and manual intervention on the device, greatly reducing the operational intrusion and resource consumption overhead of RTOS kernel tracing and device debugging. It effectively solves the problems of traditional debugging methods that easily cause system timing disorders, large debugging disturbances, and inability to adapt to long-term online tracing and debugging. It significantly reduces the intrusion of RTOS debugging and kernel tracing, while greatly improving the real-time performance, stability, and debugging accuracy of RTOS system kernel tracing.

[0070] To more clearly illustrate the RTOS remote debugging and kernel tracing method provided in this application, a specific embodiment is used below to describe the method. In an exemplary embodiment, such as Figure 2 As shown, this application also proposes another method for remote debugging and kernel tracing of an RTOS (i.e., an RTOS remote debugging server and kernel tracing method), including the following steps S1 to S7. Wherein:

[0071] Step S1: The device powers on and initializes the dual-ring buffer in memory, loads the preset non-intrusive static tracking point sandbox, registers dynamic kernel event probes as needed, and realizes the on-demand mounting of kernel event collection points such as task switching, interrupts, and memory through the acquisition function and the real-time self-check function of tracking overhead.

[0072] In this embodiment, the storage system of the dual-ring buffer in memory (such as a two-level ring cache) is as follows: After the RTOS device starts up, two independent cache memory blocks are statically divided in the RTOS heap, including a first-level high-speed interrupt cache for zero-copy write events of interrupt context; and a second-level persistent asynchronous cache for asynchronous event handling by low-priority tracking tasks. The cache structure uses atomic variables as read and write indexes, completely abandoning mutexes and semaphores to avoid the tracking logic from preempting the time slice of real-time tasks; and the cache structure has a built-in 64-bit event sequence number, event type flag, CRC (Cyclic Redundancy Check) check bit, and overflow flag bit. When the first-level cache write index catches up with the read index, the overflow flag is automatically triggered, and low-priority events of idle tasks are discarded first, while interrupt and business task exception events are retained.

[0073] In this embodiment, the preset non-intrusive static tracking point sandbox specifically controls the total Flash (Flash Memory) usage to within 1.2KB. The sandbox only reserves empty jump entry points and includes two sets of non-blocking APIs, trace_reg_probe() and trace_unreg_probe(), which support dynamic addition and deletion of acquisition probes via remote commands at runtime. The sandbox execution logic only performs data copying and atomic index auto-increment, and the execution time of a single probe is less than 0.3 microseconds, minimizing acquisition overhead.

[0074] In this embodiment, probes are divided into two categories: basic kernel probes and custom business probes. Basic probes are pre-configured with six core data collection points: task switching (sched_switch), interrupt entry (irq_entry), interrupt exit (irq_exit), memory allocation (malloc), memory release (free), and stack level detection. Custom business probes are user-defined based on APIs and are used to collect business algorithm execution events. During device initialization, only interrupt and task switching probes are mounted by default; other probes are unmounted by default but can be dynamically mounted via subsequent remote debugging commands to reduce computational power consumption during idle data collection.

[0075] In this embodiment, the ARM Cortex-M NOP null instruction pre-embedding mechanism is adopted, which inserts a fixed length of null operation assembly placeholders at a fixed position in the RTOS native function. After compilation, only the machine code without operation is filled, without changing the original registers, stack, and program execution flow. By default, there is no acquisition overhead.

[0076] In this embodiment, the sandbox is an independent Flash assembly code (total size ≤ 1.2KB), completely independent of RTOS scheduling and interrupt stack management logic. As an independent tracing execution carrier, all event acquisition, timestamp reading, and cache writing logic are encapsulated inside the sandbox. The kernel only retains the NOP (No Operation) anchor point entry, achieving complete decoupling between the kernel and the tracing code, and achieving the standard of non-intrusive kernel.

[0077] In this embodiment, the sandbox has three main components: a global probe hash list (struct trace_probe_list), where the key is the event ID (task switching / interruption / malloc, etc.) and the value is the address of the acquisition callback function; an atomic read / write two-level circular cache, used for binary storage of events, which does not rely on RTOS mutex locks; and a high-precision timing read assembly function, which directly reads the hardware timer register without system call overhead.

[0078] In this embodiment, when the CPU executes the pre-embedded TRACE_ANCHOR()NOP anchor point, it automatically jumps to the sandbox unified distribution entry point. The sandbox quickly checks the table using atomic lookup to determine whether there is a valid collection hook for the current event ID: if it exists, the event collection logic is executed; if it does not exist, it directly returns to the original function to continue executing the business, realizing on-demand start and stop of events, and zero overhead for events not enabled.

[0079] In this embodiment, the acquisition function is implemented using a dual-channel timing read function combining a set timer and SysTick (System Tick) to mask the differences in underlying registers of different MCUs and output a unified 64-bit microsecond timestamp, eliminating time jitter caused by single timing system calls. The interrupt context directly reads the hardware timer register, and the task context reads the SysTick count value and converts it to microseconds. All events uniformly carry absolute timestamps, ensuring no deviation in timing alignment with the remote server. A set tracking overhead real-time self-check function calculates the average execution time of all probes every 100ms and calculates the overall CPU utilization of the tracking logic. If the utilization rate exceeds a preset 3% threshold, a probe lightweighting strategy is automatically triggered: temporarily unloading low-priority task acquisition probes and increasing the event sampling interval to prevent the tracking logic from crowding out the control algorithm and business task computing power, avoiding the induction of new faults such as priority inversion and task stalling.

[0080] In this embodiment, without modifying the native RTOS kernel source code or disrupting real-time scheduling, an independent, lock-free two-level memory ring cache storage system is constructed. A miniature Tracepoint assembly sandbox that occupies only a small amount of Flash resources is loaded, providing standardized APIs for dynamic probe registration and unloading. High-precision microsecond-level timing acquisition functions and real-time self-test functions for tracking overhead are integrated synchronously, enabling on-demand mounting of kernel event acquisition points such as task switching, interrupts, and memory. This fundamentally solves the existing technical defects of traditional static hooks, such as strong invasiveness, high porting costs, and debugging disruption of business timing.

[0081] Step S2: Perform local filtering sampling on kernel events captured by the kernel probe, and write qualified events into the first-level circular cache using Protobuf lightweight encoding.

[0082] In this embodiment, step S2 specifically includes the following steps:

[0083] Step 21: When the mounted Tracepoint probe is triggered, the sandbox collects the original event fields through atomic write operations, including timestamp, event type enumeration, task unique ID, interrupt number, remaining stack bytes, memory operation size, etc. All data is written in-place to the register and then copied to the buffer. In this way, the interrupt context does not call the RTOS blocking API at all, and all data is written in-place to the register and then copied to the buffer, eliminating the risk of interrupt blocking.

[0084] Step 22: Using a built-in three-layer serial filtering function, events are filtered sequentially to delete invalid data. Specifically, idle and low-priority background task events below the preset task priority threshold are automatically discarded. Then, sampling windows are set for high-frequency events such as system ticks and idle loops, and single sampling records are retained at preset intervals to filter out duplicate and worthless time-series data. Finally, business task events are filtered through a task whitelist. For example, only core control tasks registered in the configuration file are collected, and other ordinary business task events are directly discarded.

[0085] Step 23: The filtered structured kernel event information is encoded by an embedded minimalist Protobuf encoding module that retains only three basic serialization capabilities: unsigned fixed-length integers, enumerated event types, and fixed short identifiers. After encoding, the information is directly written to the first-level circular buffer.

[0086] Furthermore, this embodiment also includes the following steps:

[0087] Step 24: When it is determined that the write index of the first-level cache will catch up with the read index, the overflow protection logic is triggered. That is, the priority of all events in the cache is automatically identified, and idle tasks and low-priority tick events are discarded in batches, while only interrupt, memory exception and core business task events are retained. At the same time, a global overflow flag is written to the header of the event packet and reported to the remote server. After receiving the flag, the server prompts the developer that the current cache capacity is insufficient and supports remote issuance of instructions to expand the second-level persistent cache.

[0088] Step 25: Based on device initialization, create the lowest priority trace_transfer_task transfer task, read valid events from the first-level cache in a non-blocking loop, and copy them in batches to the second-level persistent cache; set the maximum number of bytes to copy in a single transfer operation; if the first-level cache is empty, the task automatically enters the RTOS delayed blocking state, completely releasing CPU computing power.

[0089] In step S3, the device-side lightweight debugging agent completes the TLS handshake and identity authentication. After successful authentication, a long connection is established and the binary stream is reported in the cache in chunks.

[0090] In this embodiment, step S3 specifically includes the following steps:

[0091] Step 31: Create a separate low-priority task named debug_agent_task, which internally encapsulates a dual-protocol client, including a lightweight MQTT (Message Queuing Telemetry Transport) client adapted for wireless narrowband IoT devices, a TCP streaming client adapted for high-speed Ethernet devices, and a serial port pass-through module adapted for older devices without a network port. Upon power-up, the hardware channel status is automatically detected, with Ethernet being the preferred choice. When there is no wired network, it automatically switches to NB-IoT, and if there is no wireless module, it enables the local serial port pass-through mode. In this embodiment, the three channels share the same set of event fragmentation encapsulation logic.

[0092] Step 32: When establishing a connection with the remote server using the device's SN (Serial Number) and asymmetric encryption device certificate, a two-way TLS 1.3 handshake is performed. The device verifies the server's root certificate, and the server verifies the device's unique certificate. After the handshake is completed, the device uploads the SN and firmware version number. The server matches the preset RBAC permission label based on the SN, which is divided into three levels of permissions: read-only tracking, read-write debugging, and device maintenance. If the authentication is successful, step 33 is executed. If the authentication fails, the TCP / MQTT long connection is directly disconnected, and all data reporting and command interaction are rejected.

[0093] Step 33: Read the trace events in the secondary cache in a loop, and package the secondary cache events into fixed fragments according to a preset number of bytes (e.g., 64). Add the device SN, fragment sequence number, 16-bit CRC check code and total fragment identifier to the header of each fragment, and assemble them into a standard data packet that can be transmitted remotely, thus completing the preprocessing of the reported data. After the fragmented data packets are assembled, they are reliably delivered based on the channel protocol. MQTT adopts QoS level 1 and TCP adopts an acknowledgment mechanism.

[0094] Step 33 also includes step 331, where when the network connection drops due to fluctuations, the proxy locally caches unuploaded fragment data within a preset time period. After reconnecting to the server, the data is resumed based on the fragment sequence number, without losing the kernel tracking events in the cache. After receiving the fragment, the server verifies the integrity of the data packet using CRC. If a fragment is damaged, a retransmission command is issued to ensure that the tracking data stream is complete and not lost.

[0095] Step 34: Based on the data reporting, start the instruction listening logic in parallel loop, continuously subscribe to the server's downlink instruction topics, and after receiving the binary instruction packet, first match the local RBAC permission configuration to intercept high-risk unauthorized operations. For legitimate instructions, distinguish between three categories: tracking and control, kernel query, and operation and maintenance, and call the corresponding API of the Tracepoint sandbox to execute them respectively. After the operation is completed, generate a receipt packet with timestamp and instruction sequence number and send it back to the server to synchronize the current tracking status of the device.

[0096] In this embodiment, step S3 further includes the following steps:

[0097] Step 35: Read the battery voltage acquisition register cyclically and dynamically adjust the segment reporting cycle according to the preset voltage threshold. When the voltage is normal, maintain a reporting interval of 100ms. When the voltage is low, extend the interval to 1 second and only report abnormal events. When the battery is extremely low, directly pause wireless reporting and only cache locally. Adaptively balance the tracking acquisition and device power consumption throughout the process.

[0098] In this embodiment, relying on an independent low-priority device debugging agent task, the TLS1.3 bidirectional encrypted link handshake and device RBAC authentication process are implemented. It is compatible with three types of transmission channels: Ethernet TCP, NB-IoT / MQTT, and serial port transparent transmission. The binary tracking data in the secondary cache is fragmented and encapsulated, CRC checked, and reliably retransmitted. It also listens for remote debugging control commands issued by the server and integrates battery power consumption adaptive reporting logic. This solves the technical defects of traditional remote debugging plaintext transmission, lack of device authentication, incompatibility of multiple transmission channels, and large battery life loss of wireless low-power terminals.

[0099] Step S4: The remote server uses multiple connection coroutines to concurrently receive tracking packets from multiple devices, verify the fragmentation and reassembly, and then write them into the time-series database for storage.

[0100] In this embodiment, step S4 includes the following steps:

[0101] Step 41: Start the four major service modules: access gateway layer, parsing layer, indicator calculation layer, and front-end visualization layer; create a Go coroutine concurrent access pool and initialize the Redis hot cache, InfluxDB time series library, and MySQL relational library connection pool; preload the ELF file index table of all mass-produced device firmware.

[0102] Step 42: After the gateway receives the fragmented packet uploaded by the device, it first verifies the 16-bit CRC code. If the verification fails, it directly issues a fragment retransmission command. If it passes, it performs memory reassembly based on the fragment sequence number. Fragments from the same device are stored in a temporary memory buffer. After receiving a complete data packet and marking it, they are merged into a complete binary event stream. If the reassembly fails to collect all fragments within the preset time, the temporary buffer is automatically cleared and a dirty data log is recorded to avoid continuous memory occupation.

[0103] Step 43: The reconstructed structured tracking data is stored in a hierarchical manner. Real-time hot data is stored in a Redis memory cache for real-time streaming display on the front-end page. Long-term full time-series data is written to the InfluxDB time-series database, with device SN and timestamp as partition indexes, and automatically split into tables according to the storage period. Cross-time period historical trajectory query is supported. Device basic information, permission configuration, and audit operation logs are synchronously stored in a MySQL relational database for persistent retention.

[0104] In this embodiment, a coroutine pool architecture is used to manage the concurrent connections of tens of thousands of embedded devices. It connects to the MQTT / TCP gateway to receive fragmented tracking data packets, completes CRC verification, fragment reassembly, and device session context binding, and stores the parsed structured tracking data in layers in the Redis hot cache and the InfluxDB time-series database. The session state persistence mechanism solves the problems of the limited number of devices that can be connected by traditional single-threaded servers and the lack of layered storage of tracking data.

[0105] In this embodiment, in order to address the technical pain point of device offline loss of session context, step S4 further includes the following steps:

[0106] Step 44: The server detects the device's heartbeat reporting packets according to the preset detection cycle. If no data is reported within the preset time, the device is automatically marked as offline, and the currently unprocessed fragment cache index and tracking configuration are saved. After the device reconnects, the server automatically matches the historical session context, restores the breakpoint resume transmission and real-time data stream push, without having to reissue the tracking configuration command.

[0107] In step S5, the server loads the corresponding firmware symbol table, parses binary events, calls the built-in calculation function to generate scheduling performance quantification indicators in batches, and automatically marks various timing and memory abnormal events according to the preset threshold judgment model.

[0108] In this embodiment, step S5 includes the following steps:

[0109] Step 51: The received device tracking data stream is processed by matching the device firmware ELF file pre-stored in the server with the device SN number to the corresponding firmware package; the ELF file is parsed to construct two global hash mapping tables: (task ID → task name) and (memory virtual address → function name / variable name), replacing the numeric identifiers in the original binary stream, and outputting a structured event object with readable text.

[0110] Step 52: Call the built-in calculation function to automatically generate quantitative indicators by batch traversing the parsed event sequence, including task blocking delay, CPU load, interrupt time, stack level, and heap memory usage.

[0111] In one embodiment, the indicators are determined using the following calculation function:

[0112] Task blocking delay function: T block =T wake -T run_start t; Maximum, minimum, and average blocking time for a single task.

[0113] CPU load calculation function: It accumulates the effective execution time of all tasks within a preset sliding time window, and divides the total window duration to obtain the real-time CPU usage percentage.

[0114] Interrupt execution function: irq cost =irq exit_time -irq enter_time ; Statistically analyze the execution time of a single interrupt and filter out interrupts that have been blocked for more than 1ms.

[0115] Stack water level statistics function: Iterates through all stack detection events, records the minimum remaining stack space during the execution of each task, and predicts the risk of stack overflow.

[0116] Heap memory statistics functions: paired with malloc / free events, to calculate the current heap usage, the maximum single memory allocation, and the total number of bytes leaked and not released in real time.

[0117] Step 53: Compare each indicator with the preset fault threshold matrix row by row, and automatically attach a unique abnormal label to the events that exceed the standard. Simultaneously record the values ​​exceeding the standard and store them in an independent abnormal index table to quickly distinguish between normal and fault events.

[0118] Step 54: Group and aggregate events and metrics based on three dimensions: device SN, task ID, and interrupt number. Generate aggregated datasets such as task delay histogram data, interrupt time statistics, and heap memory change curves, and write them to InfluxDB to provide a data source for subsequent visualization statistics views.

[0119] Step S6: Based on the quantitative indicators and anomaly labels determined in step S5, execute the fault rule matching inference model to automatically determine the root cause of the fault and generate two types of visualized structured data: a time series timeline view and a performance statistics view.

[0120] In this embodiment, the event set with abnormal labels and complete quantitative indicators output in step S5 is read, loaded into a preset multi-condition fault matching rule base for logical reasoning, and four typical faults—interruption storm, stack overflow risk, heap memory leak, and priority inversion—are determined by combining multiple indicators. The fault occurrence time, involved tasks / interruptions, excessive values, and code optimization suggestions are output synchronously.

[0121] In one embodiment, a long WebSocket connection is maintained between the service and the front end. Online devices push the latest visualization data every second to achieve real-time monitoring. When users query historical time periods, InfluxDB time-series data is read in batches and a complete view dataset is returned at once, supporting real-time debugging and historical playback requirements respectively. When the front end clicks on the abnormal color block or statistical interval in the view, a filtering instruction is sent to the back end. The server retrieves all related tasks and interruption events in the same time window and returns the fault context time series. It supports one-click filtering of all similar abnormal records by fault type and device number.

[0122] In this embodiment, to address the pain points of existing simple remote debugging systems lacking hierarchical access control, allowing any access terminal to issue high-risk commands such as shutdown and memory read / write, posing risks of malicious intrusion and firmware tampering to mass-produced industrial and vehicle terminals, lacking a unified route for issuing remote debugging commands, causing crosstalk and chaotic management due to concurrent commands from multiple devices, and the inability of the device to dynamically adjust tracking and acquisition strategies online, necessitating a restart / re-burning of the firmware, the following steps are also set up.

[0123] In step S7, the front end sends a debugging control command, which is then authenticated by the server and forwarded to the corresponding device agent. The device then dynamically adjusts the tracking parameters or performs debugging operations.

[0124] In this embodiment, step S7 includes the following steps:

[0125] Step 71: The three types of instructions sent from the front-end page—tracking control, kernel query, and operation and maintenance—are uniformly entered into the permission interception module. The system reads the device permission tags bound to the currently logged-in user. Read-only accounts intercept high-risk instructions such as memory read / write and device restart. If the permissions do not match, a pop-up window will be displayed and the instructions will not be distributed further. If the permissions match, the compliant instructions will be sent to the next process.

[0126] Step 72: The compliant instruction carries the target device SN into the routing and distribution module, supporting two modes: precise delivery to a single device or batch delivery to multiple production lines. The server matches the corresponding coroutine session of the device, adds a unique serial number to the instruction, encapsulates it into an MQTT / TCP downlink data packet, and sends it to the terminal.

[0127] Step 73: The device debug_agent receives downlink command packets, parses the serial number and command type for classification and execution, traces control commands by calling the Tracepoint sandbox trace_reg / trace_unreg API to dynamically add or remove trace events, and uses kernel query commands to read TCB and heap memory snapshots in real time. High-risk operation and maintenance commands require secondary confirmation from the device before execution.

[0128] Step 74: After the device completes the operation corresponding to the instruction, it generates a receipt packet with the same serial number and execution success / failure status and uploads it to the server. After receiving the receipt, the server synchronously updates the front-end operation panel to ensure that the front-end and back-end device tracking configuration and kernel status are completely synchronized.

[0129] Step 75: Extract the operator's account, operation time, target device SN, original instruction content, and device execution receipt result, and write them completely into a non-deletable MySQL audit data table. The accompanying log retrieval page supports tracing all remote debugging operations by personnel, device, and time, meeting security audit standards.

[0130] This embodiment completes permission verification, routing distribution, and device downlink forwarding of front-end debugging operation commands. It distinguishes between three types of commands: tracking control, kernel query, and device maintenance. It establishes a device command execution receipt and full-process operation audit storage mechanism, strictly intercepts high-risk operations based on RBAC hierarchical permission, and ensures that all remote debugging behaviors are fully traceable. This solves the security risks of mass production equipment caused by traditional remote debugging's lack of permission control, lack of operation logs, and arbitrary issuance of high-risk commands.

[0131] The above embodiments can achieve the following technical effects: (1) By setting up a micro static Tracepoint sandbox in RTOS to replace the hard-coded global hook, no kernel source code needs to be modified. At runtime, task switching, interrupt, and memory allocation probes are dynamically mounted / unmounted, and events are collected non-intrusively; (2) The device-side dual-level circular buffer (high-speed real-time small cache and low-speed persistent large cache) is used to sample and reduce load based on event type, task priority, and time window, and low-value idle events are automatically discarded; (3) The edge device debugging agent is combined with the cloud aggregation service in a dual-level manner. Based on the MQTT lightweight protocol, the Protobuf serialized binary trace stream is transmitted, and multiple devices can access it concurrently; (4) TLS1.3 encrypted link, RBAC hierarchical debugging permissions, and operation log persistence distinguish between three types of permissions: read-only trace, read-write debugging, and device shutdown; (5) The server has built-in time-series quantization calculation function and fault judgment rule model to automatically calculate CPU load, task blocking time, stack high water level, and automatically mark abnormal events in red and output root cause suggestions; (6) Supports Ethernet TCP, 4G / NB-IoT Three transmission channels, including MQTT and serial port pass-through, are compatible with wired / wireless mass production terminals; (7) The server preloads the firmware ELF symbol table and automatically converts the original TCB handle and memory address into task name and function name to generate an interactive timing visualization view.

[0132] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0133] Based on the same inventive concept, this application also provides an RTOS remote debugging and kernel tracing device for implementing the aforementioned RTOS remote debugging and kernel tracing method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more RTOS remote debugging and kernel tracing device embodiments provided below can be found in the limitations of the RTOS remote debugging and kernel tracing method described above, and will not be repeated here.

[0134] In one exemplary embodiment, such as Figure 3 As shown, an RTOS remote debugging and kernel tracing device is provided, including: a function execution module 301, an event encoding module 302, and a link establishment module 303, wherein:

[0135] The function execution module 301 is used to initialize the memory double ring buffer after the RTOS device is powered on, load the preset non-intrusive static tracking point sandbox, register the kernel event dynamic probe on demand, and perform on-demand mounting of kernel event acquisition points through preset acquisition functions and real-time self-test functions for tracking overhead.

[0136] The event encoding module 302 is used to perform local filtering and sampling processing on kernel events captured by the kernel event dynamic probe, and to write the qualified kernel events selected into a first-level high-speed interrupt buffer in a dual-ring buffer in memory using lightweight encoding; the first-level high-speed interrupt buffer stores the trace binary stream formed by the qualified kernel events after lightweight encoding.

[0137] The connection establishment module 303 is used to complete the TLS handshake and authentication with the remote server through the built-in lightweight debugging agent in order to establish a secure long connection with the remote server, and to report the trace binary stream in the first-level high-speed interrupt cache to the remote server in blocks.

[0138] The remote server is used to concurrently receive trace data packets from multiple RTOS devices based on trace binary stream chunks via multiple connection coroutines; it verifies, fragments, and reassembles the trace data packets, writes the reassembled binary events into a time-series database for storage; it loads the firmware symbol table of the corresponding RTOS device, parses the binary events stored in the time-series database, calls built-in calculation functions to generate scheduling performance quantification metrics in batches, and performs anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly labels; it inputs the scheduling performance quantification metrics and anomaly labels into a preset fault rule matching inference model to obtain the corresponding fault root cause and generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

[0139] In an exemplary embodiment, the function execution module 301 is further configured to statically partition two independent cache memory blocks in the RTOS heap of the RTOS device after the RTOS device is powered on, thereby obtaining a dual-ring buffer memory; wherein, the two independent cache memory blocks include a first-level high-speed interrupt cache and a second-level persistent asynchronous cache, the first-level high-speed interrupt cache is used for interrupt context zero-copy write events, and the second-level persistent asynchronous cache is used for low-priority tracking task asynchronous transport events; the cache structure uses atomic variables as read and write indexes.

[0140] In an exemplary embodiment, the preset non-intrusive static tracking point sandbox only reserves empty jump entry points and provides two sets of non-blocking APIs to support dynamic addition and deletion of acquisition probes via remote commands at runtime. When executing logic, the preset non-intrusive static tracking point sandbox only performs data copying and atomic index auto-increment, and has three built-in components. The three components include: a global probe hash chain, an atomic read / write two-level circular cache, and a high-precision timing read assembly function.

[0141] In an exemplary embodiment, the event encoding module 302 is further configured to, when the mounted trace point probe is triggered, collect the original event fields of the captured kernel events based on atomic write operations through a preset non-intrusive static trace point sandbox; the original event fields include timestamp, event type enumeration, task identifier, interrupt number, stack remaining bytes, and memory operation size; and filter qualified kernel events from the captured kernel events based on the original event fields using a built-in three-layer serial filtering function; the three-layer serial filtering function includes a task priority filtering function, a high-frequency event downsampling function, and a task whitelist filtering function; and encode the qualified kernel events through a preset encoding module and write them into the first-level high-speed interrupt cache in the memory double-ring buffer.

[0142] In an exemplary embodiment, the RTOS remote debugging and kernel tracing device further includes a batch copy module, which is used to trigger overflow protection logic when it is determined that the write index of the first-level high-speed interrupt cache will catch up with the read index, write a global overflow flag in the event packet header, and report it to the remote server. After receiving the global overflow flag, the remote server will prompt the developer that the current cache capacity is insufficient and issue a remote instruction to the RTOS device to expand the second-level persistent asynchronous cache. According to the lowest priority transfer task created during the initialization of the RTOS device, the module reads the valid events in the first-level high-speed interrupt cache in a non-blocking loop and copies them in batches to the second-level persistent asynchronous cache.

[0143] In an exemplary embodiment, the connection establishment module 303 is further configured to extract the tracking binary stream from the first-level high-speed interrupt cache; determine the target segmentation method matching the tracking binary stream; segment the tracking binary stream according to the target segmentation method to obtain the corresponding tracking data packet; and report the tracking data packet to the remote server.

[0144] The modules in the aforementioned RTOS remote debugging and kernel tracing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0145] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for remote debugging and kernel tracing of an RTOS. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0146] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0147] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0148] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0149] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0152] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for remote debugging and kernel tracing in an RTOS, characterized in that, The method includes: After the RTOS device is powered on, the memory dual-ring buffer is initialized, the preset non-intrusive static trace point sandbox is loaded, the kernel event dynamic probe is registered as needed, and the kernel event collection point is mounted as needed through the preset collection function and the real-time self-test function of the trace overhead. Local filtering and sampling processing is performed on the kernel events captured by the kernel event dynamic probe. The qualified kernel events selected are written into the first-level high-speed interrupt cache in the dual-ring buffer of memory using lightweight encoding. The first-level high-speed interrupt cache stores the trace binary stream formed by the qualified kernel events after lightweight encoding. The system completes TLS handshake and authentication with the remote server through the lightweight debugging agent to establish a secure long connection with the remote server, and reports the trace binary stream in the first-level high-speed interrupt cache to the remote server in blocks. The remote server is used to concurrently receive tracking data packets from multiple RTOS devices based on the segmentation of the tracking binary stream via multiple connection coroutines; to perform verification, fragmentation, and reassembly of the tracking data packets, and to write the reassembled binary events into a time-series database for storage; to load the firmware symbol table of the corresponding RTOS device, parse the binary events stored in the time-series database, call built-in calculation functions to generate scheduling performance quantification indicators in batches, and to perform anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly tags; to input the scheduling performance quantification indicators and the anomaly tags into a preset fault rule matching inference model to obtain the corresponding fault root cause and generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

2. The method according to claim 1, characterized in that, The initialization of the dual-ring memory buffer after the RTOS device is powered on includes: After the RTOS device is powered on, two independent cache memory blocks are statically partitioned in the RTOS heap of the RTOS device to obtain a dual-ring buffer memory. The two independent cache memory blocks include a first-level high-speed interrupt cache and a second-level persistent asynchronous cache. The first-level high-speed interrupt cache is used for zero-copy write events of interrupt context, and the second-level persistent asynchronous cache is used for low-priority tracking task asynchronous transport events. The cache structure uses atomic variables as read and write indexes.

3. The method according to claim 1, characterized in that, The preset non-intrusive static tracking point sandbox only reserves empty jump entry points and provides two sets of non-blocking APIs to support the dynamic addition and deletion of acquisition probes via remote commands at runtime. The preset non-intrusive static tracking point sandbox execution logic only performs data copying and atomic index auto-increment, and has three built-in components: a global probe hash chain, an atomic read / write two-level circular cache, and a high-precision timing read assembly function.

4. The method according to claim 1, characterized in that, The process of performing local filtering and sampling on kernel events captured by the kernel event dynamic probe, and writing the selected qualified kernel events into the first-level high-speed interrupt cache in the dual-ring buffer of memory using lightweight encoding, includes: When the mounted trace point probe is triggered, the original event fields of the captured kernel event are collected through the preset non-intrusive static trace point sandbox based on atomic write operations; the original event fields include timestamp, event type enumeration, task identifier, interrupt number, stack remaining bytes, and memory operation size; The system uses a built-in three-layer serial filtering function and selects qualified kernel events from the captured kernel events based on the original event fields. The three-layer serial filtering function includes a task priority filtering function, a high-frequency event downsampling function, and a task whitelist filtering function. The qualified kernel events are encoded using a preset encoding module and written into the first-level high-speed interrupt cache in the dual-ring buffer of memory.

5. The method according to claim 4, characterized in that, The method further includes: When it is determined that the write index of the first-level high-speed interrupt cache will catch up with the read index, the overflow protection logic is triggered, a global overflow flag is written in the header of the event packet, and it is reported to the remote server. After receiving the global overflow flag, the remote server will prompt the developer that the current cache capacity is insufficient and issue a remote instruction to the RTOS device to expand the second-level persistent asynchronous cache. Based on the lowest priority transport task created during the RTOS device initialization, the system reads valid events from the first-level high-speed interrupt cache in a non-blocking loop and copies them in batches to the second-level persistent asynchronous cache.

6. The method according to any one of claims 1 to 5, characterized in that, The segmented reporting of the trace binary stream in the first-level high-speed interrupt cache to the remote server includes: Extract the tracking binary stream from the first-level high-speed interrupt cache; Determine the target segmentation method that matches the tracking binary stream, and segment the tracking binary stream according to the target segmentation method to obtain the corresponding tracking data packets; The tracking data packet is reported to the remote server.

7. A remote debugging and kernel tracing device for an RTOS, characterized in that, The device includes: The function execution module is used to initialize the memory double ring buffer after the RTOS device is powered on, load the preset non-intrusive static trace point sandbox, register the kernel event dynamic probes as needed, and perform on-demand mounting of kernel event collection points through preset collection functions and real-time self-check functions for trace overhead. The event encoding module is used to perform local filtering and sampling processing on the kernel events captured by the kernel event dynamic probe, and to write the qualified kernel events selected into the first-level high-speed interrupt cache in the dual-ring buffer of memory using lightweight encoding; the first-level high-speed interrupt cache stores the trace binary stream formed by the qualified kernel events after lightweight encoding; The connection establishment module is used to complete TLS handshake and identity authentication with the remote server through the built-in lightweight debugging agent in order to establish a secure long connection with the remote server, and to report the trace binary stream in the first-level high-speed interrupt cache to the remote server in blocks; The remote server is used to concurrently receive tracking data packets from multiple RTOS devices based on the segmentation of the tracking binary stream via multiple connection coroutines; to perform verification, fragmentation, and reassembly of the tracking data packets, and to write the reassembled binary events into a time-series database for storage; to load the firmware symbol table of the corresponding RTOS device, parse the binary events stored in the time-series database, call built-in calculation functions to generate scheduling performance quantification indicators in batches, and to perform anomaly identification on the parsed kernel events according to a preset threshold judgment model to obtain corresponding anomaly tags; to input the scheduling performance quantification indicators and the anomaly tags into a preset fault rule matching inference model to obtain the corresponding fault root cause and generate visualized structured data; the visualized structured data includes a time-series timeline view and a performance statistics view.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images