Method and apparatus for detecting an overflow condition in a kernel stack during operating system development

Abstract

One embodiment of the present invention provides a system with a mechanism that facilitates determining whether a kernel stack will overflow during development of an operating system. During execution of the operating system, the operating system executes a procedure. Upon entering the procedure, after the current stack pointer has been automatically adjusted to provide stack space for the procedure, the system retrieves the current stack pointer for the process, and determines the stack limit for the process. The system then compares the current stack pointer to the stack limit to determine if the process will cause, or is likely to cause a stack overflow. If a stack overflow will occur, or is likely to occur, the system indicates the existence of a stack overflow condition.

Description

BACKGROUND[0001]1. Field of the Invention[0002]The present invention relates to the process of developing operating systems for computers. More specifically, the present invention relates to a method and an apparatus for detecting a kernel stack overflow condition during development of an operating system.[0003]2. Related Art[0004]The problem of kernel stack overflow arises frequently during operating system development. As kernel code makes procedure calls, each successive procedure call allocates a stack frame on the kernel stack. Upon completion of a procedure call, the stack frame for the procedure call is removed from the top of the kernel stack. When procedure calls become deeply nested, a large amount of stack space can potentially be used. In some instances, so much stack space is used that the stack grows beyond its allocated space and stack overflow results. During a stack overflow, the system starts overwriting information in memory located past the end of the stack. Oper...

AI Technical Summary

Benefits of technology

[0009]One embodiment of the present invention provides a system with a mechanism that facilitates determining whether a kernel stack will overflow during development of an operating system. During execution of the operating system, the operating system executes a procedure. Upon entering the procedure, after the current stack pointer has been automatically adjusted to provide stack space for the procedure, the system retrieves the current stack pointer for the process, and determines the stack limit for the process. The system then compares the current stack pointer to the stack limit to determine if the process will cause, or is likely to cause a stack overflow. If a stack overflow will occur, or is likely to occur, the system indicates the existence of a stack overflow condition.

Problems solved by technology

The problem of kernel stack overflow arises frequently during operating system development.

When procedure calls become deeply nested, a large amount of stack space can potentially be used.

In some instances, so much stack space is used that the stack grows beyond its allocated space and stack overflow results.

Operating systems are typically designed to handle stack overflow conditions, but stack overflow conditions are not always detected and handled successfully, especially during the operating system development process, when the kernel code is not stable.

Stack overflows of this nature can be extremely difficult to diagnose, especially because they can randomly corrupt memory being used for unrelated purposes.

However, this technique can greatly complicate hardware development and can potentially degrade system performance.

While sometimes useful, this technique does not immediately provide notification when the stack pointer exceeds its boundaries.

Hence, the overflow can potentially corrupt critical data before the stack overflow is detected, making it hard to diagnose the problem.

Images