System desktop recording method and system based on IPKVM

Abstract

The embodiment of the invention provides a system desktop recording method and system based on an IPKVM, and relates to the technical field of desktop recording technologies. The method comprises the following steps: establishing association configuration between IPKVM equipment and target equipment, and generating an equipment mapping table; the IPKVM device obtains an original video stream from the target device, converts the original video stream into digital video frames, and associates the video frames in the same time window with the control signal according to a timestamp to generate a synchronous data packet; performing adaptive compression coding on the video frames in the synchronous data packet; determining the dynamic complexity of the current frame based on the inter-frame pixel change rate, and selecting a compression mode and adjusting coding parameters according to the dynamic complexity; controlling the starting and stopping of the recording task according to the recording trigger condition; and packaging the compressed video stream data and the corresponding control instruction data to generate a recording file. According to the method and the device, the problem of fuzzy character recording is solved, and the effect of improving the character recording definition is further achieved.

Description

Technical Field

[0001] This invention relates to the field of desktop recording technology, and more specifically, to a system desktop recording method and system based on IPKVM. Background Technology

[0002] In the operation and maintenance management of data centers and enterprise server rooms, IPKVM (KVM over IP) technology is the core means to achieve remote control. Maintenance personnel use IPKVM to remotely access servers to perform troubleshooting, system installation, and other operations. To meet the needs of compliance auditing and fault review, recording the desktop screen during the operation and maintenance process is crucial.

[0003] However, existing IPKVM recording solutions have significant technical flaws. For example, traditional general video compression standards (such as the basic configuration of H.264 / H.265) are mainly designed for natural scene videos and have not been optimized for the characteristics of desktop video streams, such as "high proportion of static images, dense text lines, and local dynamic changes". This results in blurred text edges at low bitrates, which seriously affects audit readability, or generates huge storage redundancy when trying to maintain clarity. Summary of the Invention

[0004] This invention provides a system desktop recording method and system based on IPKVM, which at least solves the problem of blurry text recording in related technologies.

[0005] According to an embodiment of the present invention, a system desktop recording method based on IPKVM is provided, comprising: Establish the association configuration between the IPKVM device and the target device, and generate a device mapping table; The IPKVM device acquires the raw video stream from the target device and converts it into digital video frames. At the same time, it captures the control signals sent by the remote operation and maintenance terminal, and stamps the digital video frames and the control signals with millisecond-level timestamps based on a unified clock source. Based on the timestamps, it associates the video frames and control signals within the same time window to generate synchronization data packets. Adaptive compression coding is performed on the video frames in the synchronization data packet, the adaptive compression coding including calculating the inter-frame pixel change rate of the current video frame relative to the reference frame; The dynamic complexity of the current frame is determined based on the inter-frame pixel change rate, and the compression mode and encoding parameters are selected and adjusted according to the dynamic complexity; wherein, the encoding parameters include frame type, quantization parameters and target bit rate; The recording task is started and stopped according to preset recording trigger conditions, including manual commands, time scheduling and specific event detection. The compressed video stream data and corresponding control command data are encapsulated to generate a recording file containing device identification and time information, and an index for retrieval is established.

[0006] In one exemplary embodiment, calculating the inter-frame pixel change rate of the current video frame relative to the reference frame includes: Iterate through every pixel at the same coordinate position in the current video frame and the reference frame; Calculate the absolute value of the component differences of the corresponding pixel in the RGB color space; If the absolute value exceeds the preset pixel tolerance threshold, the pixel is marked as a changed pixel, and the total number of changed pixels is accumulated. The inter-frame pixel change rate is determined based on the ratio of the total number of changed pixels to the total number of pixels in a single frame.

[0007] In an exemplary embodiment, selecting the compression mode and adjusting the encoding parameters based on the dynamic complexity includes: If the inter-frame pixel change rate is less than the preset static determination threshold, the current frame is determined to be a static frame, the target bit rate is set to the first bit rate range, and the high-frequency texture region in the current frame is identified, and the quantization parameter of the high-frequency texture region is set to a lossless encoding value. If the inter-frame pixel change rate is greater than a preset dynamic determination threshold, the current frame is determined to be a dynamic frame, and the target bit rate is set to the second bit rate interval; wherein, the lower limit of the second bit rate interval is greater than the upper limit of the first bit rate interval.

[0008] In one exemplary embodiment, identifying the high-frequency texture region in the current frame includes: Perform edge detection operations on the luminance components of the current frame to generate an edge gradient map; The edge gradient map is divided into several macroblocks of a preset size; Calculate the average gradient magnitude within each macroblock; if the average gradient magnitude exceeds a preset texture density threshold, then mark the corresponding macroblock region as the high-frequency texture region.

[0009] In an exemplary embodiment, controlling the start and stop of the recording task according to preset recording trigger conditions includes: Real-time analysis of the content of the control signals and the system status data of the target device; When the control signal is detected to contain a preset high-risk instruction string, or when the resource occupancy rate index in the system status data exceeds a preset warning value, a trigger recording signal is generated. Recording begins upon receiving the trigger recording signal and stops after a preset buffer time following the end of the trigger event.

[0010] In one exemplary embodiment, the step of encapsulating the compressed video stream data with the corresponding control command data to generate a recording file containing device identifier and time information includes: Create a media container file containing video tracks and subtitle tracks; Write the compressed video stream data into the video track; The control command data is converted into text format and written into the subtitle track according to its associated timestamp, so as to realize the synchronous overlay display of control commands and video images.

[0011] In one exemplary embodiment, the method further includes: The recorded file is parsed, and the video stream data and control command data are decoded simultaneously. The video frame is rendered on the playback interface, and the control command content corresponding to the current moment is rendered in the form of semi-transparent subtitles at the specified layer position of the frame. In response to a user's audit operation, audit marker data is inserted at a specific position on the timeline, and the audit marker data is associated with the index information.

[0012] According to another embodiment of the present invention, a system desktop recording system based on IPKVM is provided, comprising: The device association configuration module is used to establish and store the mapping relationship between IPKVM device ports and target devices, as well as network configuration parameters. The synchronous acquisition module is used to acquire video signals from the target device through the video interface and convert them into digital frames, capture peripheral control commands through the control interface, and generate time-stamped synchronous data packets based on the system clock. The adaptive compression module is used to calculate the inter-frame pixel change rate, thereby distinguishing between static and dynamic images, and dynamically assigning encoding modes and corresponding quantization parameters to them respectively. The recording control module is used to monitor trigger events and generate recording start / stop control signals according to preset logic; The data storage module is used to encapsulate the encoded audio and video data and control command stream into a standard format file and write it to a local or remote storage medium.

[0013] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0014] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] This invention establishes device mapping in an embedded IPKVM system and synchronously acquires video frames and peripheral control signals. It also introduces an adaptive compression algorithm based on the inter-frame pixel change rate, thereby achieving efficient, clear desktop recording with operational semantics. Therefore, it can solve the problem of blurry desktop text recording and improve the recording clarity. Attached Figure Description

[0016] Figure 1 This is a flowchart of a system desktop recording method based on IPKVM according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a system desktop recording system based on IPKVM according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the ROI QP Offset distribution according to an embodiment of the present invention; Figure 4 This is an adaptive bit rate linear control curve according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the edge gradient detection principle according to an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0018] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0019] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0020] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0021] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0022] Example 1 The solution provided in this application runs on a high-performance embedded IPKVM computing platform, which integrates a customized Linux 5.10 real-time kernel. At the hardware level, it is equipped with an ARM Cortex-A53 quad-core SoC and coupled with a dedicated FPGA video processing unit and a gigabit Ethernet controller. The IPKVM device is physically deployed at the top of the data center rack (ToR) and is physically connected to multiple downstream managed servers via KVM cables to provide out-of-band management services.

[0023] like Figures 1-2 As shown, the system desktop recording method based on IPKVM provided in this application specifically includes the following steps: S100. Device Association and Configuration Phase In this embodiment, the device association configuration module responds to the system initialization command by instantiating a non-volatile storage structure in the embedded file system of the IPKVM device to persistently store device topology information; wherein, the device association configuration module first defines a data type TargetDevice for structurally describing the attributes of the target device.

[0024] For example, the TargetDevice struct is configured to contain the following strongly typed fields: device_id: A fixed-length character array (chararray

[32] ) used to store the globally unique identifier of the target device (e.g., "Server_Rack01_Unit05"), which serves as the primary key for the system's internal index; kvm_port: An unsigned integer (uint8_t) used to map the input port index (e.g., port index) of the target device's physical access to the IPKVM backplane. ); resolution_w: An unsigned integer (uint16_t) used to define the number of horizontally valid pixels in the target device's video output (e.g., ...). ); resolution_h: An unsigned integer (uint16_t) used to define the number of vertically valid pixels in the target device's video output (e.g., ...). ); refresh_rate: An unsigned integer (uint8_t) used to define the vertical synchronization frequency of the target device's video signal (e.g., ...). Hz).

[0025] The following sub-steps are then used for further processing: S110. Mapping Table Construction and Persistence The device association configuration module running the Web service process receives the JSON-formatted configuration payload submitted by the administrator via the HTTPS interface. After parsing, it constructs a device mapping table device_map in the kernel-mode memory space. Here, the mapping table is instantiated in memory as an array of TargetDevice structures.

[0026] For example, in response to a new device's registration request, the module calls a memory allocation function to allocate heap space and sets the resolved IPv4 address "192.168.10.50", hostname "Database-Primary", and resolution obtained through automatic negotiation via the EDID protocol. Fill the index slots of the device_map Subsequently, the module calls file system APIs (such as fwrite) to serialize the mapping table structure in memory into a binary stream and persist it to the / etc / ipkvm / config / devices.dat path of the onboard eMMC storage to ensure that the configuration is not lost after the device restarts.

[0027] S120. Network Protocol Stack Initialization The device association configuration module further calls the netlink socket interface based on the mapping table to configure the network adapter parameters of IPKVM, thereby establishing a reliable data transmission channel.

[0028] For example, a system-level network configuration script is executed here to configure the IPKVM's eth0 physical interface to a static IPv4 address. Subnet mask And add routing rules to point the default gateway to the core switch. This network layer establishes TCP / IP connectivity between IPKVM and the remote network storage server (NAS), providing underlying link support for subsequent streaming of recorded files based on NFS or SMB protocols.

[0029] S121. Hardware Initialization and V4L2 Driver Configuration The device association and configuration phase includes the initialization of the underlying registers of the video acquisition subsystem. To ensure the frame-level stability of the video stream, the FPGA video bridge chip (such as ITE6604) integrated inside IPKVM must be configured with timing parameters via the I2C bus and expose the control interface to the user space via the V4L2 (Video for Linux 2) driver framework.

[0030] For example, during the system boot phase, the kernel loads the ite6604.ko driver module. The initialization daemon executes the following instructions: v4l2-ctl device= / dev / video0 set-fmt-video=width=1920,height=1080,pixelformat=RGB3 This allows for the explicit distribution of resolution and pixel format parameters to the driver layer.

[0031] Specifically, in response to the configuration instruction, the driver writes a mask of 0x01 to the FPGA's control register REG_Video_Detect (offset address 0x0A) via the I2C bus to activate the input signal detection logic. Once the FPGA hardware phase-locked loop (PLL) is locked and the EDID handshake is successful, the driver polls the status register REG_Status (offset address 0x0B) to read the timing status word of the input source. If the read status word is 0x3C (binary 00111100), the driver logic determines that the input signal has standard CEA-861-F timing (1080p60). After confirming the signal lock, the driver configures the DMA controller's descriptor table, pointing the source address to the FPGA's FIFO buffer base address 0xB0000000, the destination address to the kernel-reserved physical contiguous memory block (CMA) 0xC0000000, and sets the transfer length register to 1920*1080*3 bytes.

[0032] Through fine-tuning at the hardware register level, a high-speed data channel from the video input interface to system memory is established here, ensuring that the subsequent S200 stage can obtain each frame of original image data from the / dev / video0 character device with a zero-copy mechanism, thereby eliminating the CPU load and latency caused by user space memory copying.

[0033] S200. Synchronous Data Acquisition Phase In this embodiment, S200 is used to perform parallel acquisition and time-domain alignment of multimodal data, which involves a synchronous acquisition module with multi-threaded scheduling capabilities. Subsequently, two acquisition threads with real-time priority are started. These two threads are used to poll the video acquisition interface and the USB HID interrupt endpoint, respectively, and to achieve precise synchronization of the data stream based on the system's high-precision monotonic clock (CLOCK_MONOTONIC). Specifically, this includes the following sub-steps: S210. Video Stream Digitization and Buffering The synchronous acquisition module subscribes to the raw video stream output by the IPKVM video acquisition front-end (FPGA or dedicated ASIC) via the V4L2 interface.

[0034] For example, the video capture front end is configured to... The frame rate (fps) continuously outputs horizontal and vertical sync signals. For a resolution of [resolution value missing], [the following text is incomplete and requires further context: "fps" appears to be a fragment of a larger document and doesn't translate directly. It likely refers to a specific resolution or value, but without further information, a more coherent translation is impossible.] The input source, the acquisition module maps a block of capacity in kernel space. The DMA circular buffer (MB) is used to receive uncompressed pixel arrays in RGB888 format. Whenever the DMA controller triggers a "frame completion" hardware interrupt, the interrupt service routine (ISR) immediately reads the current system monotonic time. (Accurate to nanoseconds and converted to milliseconds), and append this as metadata to the video frame header; for example, if the system runtime counter at the time of the interruption is... If the time is less than 1 second, then the timestamp of that frame is marked as 1 second. ms.

[0035] S220. Control Command Capture and Parsing In parallel with video acquisition, the control flow thread of the synchronous acquisition module listens to the HID interface exposed by the IPKVM internal USB Gadget driver to capture input events from remote operation and maintenance terminals.

[0036] For example, when an operator triggers a key combination operation (such as "Ctrl+C") on the client, the USB controller receives the corresponding HID message; then the acquisition module parses the wScanCode and bModifier fields of the message and maps them to standardized instruction descriptors "Key_Down:Ctrl" and "Key_Down:C" by looking up a table.

[0037] Next, the module calls the clock_gettime system call to obtain the current monotonic time. Assume the key interrupt occurs during system operation. seconds, then ms; and so on.

[0038] S230. Synchronization data packet generation The synchronous acquisition module executes an alignment algorithm based on a sliding time window to aggregate asynchronously arriving video frames and control command streams into structured SyncPacket data units.

[0039] At this point, the algorithm predefines a time window threshold. This threshold is set to match the video frame interval, for example, timestamps. The video data is retrieved from the module's control command circular buffer, and all timestamps are extracted. Falling into the range Record the instructions; if the current video frame And there are in the buffer ("Ctrl") and For records containing "C", these two instruction objects are dequeued and linked to the SyncPacket structure of that video frame. The final generated synchronization data packet contains: a pointer to the image data buffer, image metadata (width / height / step), and an instruction counter. List of instruction objects and base timestamp .

[0040] S300. Adaptive Compression Coding Stage In this embodiment, this stage is used to adjust the encoding pipeline according to the dynamic characteristics of the visual content; specifically, the adaptive compression module receives the SyncPacket stream and dynamically reconfigures the quantization strategy and frame structure of the H.265 encoding core based on real-time calculated feature indicators to maximize information retention under limited bandwidth, specifically including the following sub-steps: S310. Calculation of inter-frame pixel change rate The adaptive compression module first loads the current frame. With the previous frame The SIMD (Single Instruction Multiple Data) acceleration unit executes a parallel pixel difference measurement algorithm; the algorithm initializes the atomic counter. The value is 0; then the image domain is iterated through. Where W is the width and H is the height. For each coordinate point, the SIMD instruction concurrently loads the RGB component vector. and .

[0041] Then calculate the L1 norm distance. :

[0042] in: These represent the coordinates of the current frame. The red, green, and blue component values ​​at the location (value range 0-255); These respectively indicate that the previous reference frame is in the same coordinate. The red, green, and blue component values ​​at the location.

[0043] This section introduces a noise threshold based on sensor characteristics. ,and ;in The value of is related to the signal-to-noise ratio (SNR) of the video acquisition front-end (A / D converter or HDMI receiver chip); typically, at 8-bit color depth, the quantization noise of analog signals fluctuates between 1 and 3 gray levels; considering the cumulative error of the RGB three channels, the theoretical noise floor is approximately To prevent misjudgment, a certain safety margin is set. For example, for digital signal input (HDMI / DVI), a certain safety margin is set. For analog signal input (VGA), the system acquires 10 frames of completely black images during the initialization phase to calculate the average noise floor. And dynamically set the threshold to ( (This is the standard deviation), and this value is usually 15-20. Comparison logic: If If the pixel changes significantly, it is determined that the atomic increments are used. After the traversal is complete, the inter-frame pixel change rate is calculated using normalization. :

[0044] For example, for The total number of pixels in the image. If the pixel statistics change ,but .

[0045] S320. Static Scene Coding Strategy The adaptive compression module will calculate the result Compared with the preset static judgment threshold For comparison, here .

[0046] In response to (as mentioned above) The module determines that it is currently in static display mode (such as a text editing scenario) and sends the following parameter set to the encoder control interface: a. Forced full I-frame mode: Setting the GOP length parameter KeyInt of the H.265 encoder to 1 forces the inter-frame prediction function to be disabled. This configuration ensures that each frame is an independently decoded IDR frame, eliminating the cumulative blurring effect caused by the P-frame reference chain.

[0047] b. Bitrate suppression: The target bit rate of the rate control module Clamped to low bandwidth range, set here. Kbps; Due to the extremely high redundancy of the image, this low bitrate is sufficient to maintain lossless static image quality.

[0048] c. Lossless encoding of the region of interest (ROI): The module calls an edge detection subroutine (such as Sobel filtering) to extract high-frequency texture masks.

[0049] Specifically, for each macroblock (CTU) covered by the mask, the module rewrites its quantization parameter offset (QPOffset), such as... Figure 3 As shown, for the coordinate region where the character texture is detected ,set up (corresponding to the lossless quantization step size), while maintaining the background region. This ROI map is injected into the encoder context via FFmpeg's av_frame_add_side_data interface.

[0050] Specifically, it includes the following sub-steps: Detailed Explanation of S321.H.265 Encoder Low-Level Configuration and ROI Parameter Injection To achieve region-level lossless optimization at the engineering level, it is necessary to delve into the low-level API of the H.265 coding library (such as x265); here, the H.265 standard uses a quadtree-structured coding tree unit (CTU) as the basic processing unit, typically with a size of [size missing]. .

[0051] In particular, the adaptive compression module adds metadata to each frame by filling the x265_picture structure; specifically, in order to achieve fine-grained QP control, the module utilizes the "external quantization offset table" interface provided by x265.

[0052] First, the module allocates an integer array `qmap` in the heap memory, the length of which is equal to the total number of CTUs contained in the image; for a 1080p image and a block size of 16x16, the array length is... .

[0053] For example, in the C++ implementation code, the module performs the following initialization sequence: x265_picture pic; x265_picture_init(&pic); pic.quantOffsets = (int*)malloc(sizeof(int) * total_blocks); like Figure 3 As shown, the generated texture mask is traversed, and for each macroblock index marked as a ROI... The module writes a negative QP offset value: pic.quantOffsets[idx_{roi}]=-20; at this time, if the baseline QP is set to 25, the effective QP of this ROI region will be pulled down to This allows for the allocation of a large number of bits to preserve edge details.

[0054] Furthermore, adaptive quantization (AQ) and external offset support must be explicitly enabled in the encoder initialization parameter x265_param. For example, set param.rc.aqMode=X265_AQ_VAR to enable variance-based AQ and set param.bEnableQpOffsets=1. It should be noted that this configuration instruction forces the encoder kernel to read and apply the externally passed quantOffsets table when performing rate-distortion optimization (RDO) decisions, thereby achieving a directional tilt of bitrate resources towards text regions.

[0055] S330. Dynamic Scene Coding Strategy The adaptive compression module will With dynamic judgment threshold Compare them.

[0056] For example, configuration Simultaneously responding to (For example, playing a full-screen video causes...) The module determines that it is currently in a high-dynamic mode and issues the following parameter set: a. Long GOP structure enabled: Restores the standard IBBP frame structure; at this time, gop_size=30, that is, insert an IDR frame every 30 frames, and fill the gaps with P frames and B frames; this configuration uses inter-frame prediction to significantly reduce temporal redundancy.

[0057] b. Rate boosting: Increasing the target bit rate Dynamically adjust to the high bandwidth range. For example, set... bps (3Mbps) to provide a sufficient bit budget to encode motion vectors and residual data.

[0058] c. Fast motion estimation: Enable fast motion estimation algorithms such as diamond search to reduce encoding latency.

[0059] S340. Transition Scene Handling In response to The module uses a linear interpolation algorithm to calculate the target bitrate. (Kbps) like Figure 4 As shown, this achieves a control curve that smoothly changes the bitrate with the complexity of the image.

[0060] S350. Detailed Explanation of High-Frequency Texture Region Recognition and Lossless Encoding To support the ROI generation logic, a gradient texture extraction algorithm is used here. This algorithm achieves efficient localization based on edge detection operators in image processing, and specifically includes the following sub-steps: S351. Brightness Extraction and Gradient Convolution like Figure 5 As shown, the adaptive compression module first calls the color space conversion function to convert the RGB frame to YUV420P format, and retains only the Y (luminance) component plane.

[0061] For each pixel in the Y plane The module applies the Sobel operator to perform convolution operations to extract horizontal gradients. and vertical gradient Here, the Sobel kernel is defined as:

[0062] Then the gradient magnitude is calculated. :

[0063] Specifically, for pixels at the edge of text, where the brightness changes drastically (e.g., from 255 to 0), the convolution result... This will produce large values ​​(such as) (), while in flat background areas the value is close to 0.

[0064] S352. Macroblock-level Energy Statistics and Binarization To match the block structure of the encoder, the module divides the gradient map into... The statistics window (corresponding to CTU); this choice is made here because of the macroblock size. The choice of CTU size depends on the resolution and encoding standard of the input video. The default maximum CTU size for the H.265 standard is [missing value]. Specifically, for 1080p (1920×1080) resolution, standard text (such as a 12pt Console font) appears to be approximately 16-20 pixels high on the screen. Macroblocks can cover complete character blocks with moderate computational overhead; however, for 4K (3840×2160) and higher resolutions, due to increased pixel density, text of the same physical size occupies more pixels. Therefore, to ensure the precision of ROI segmentation and prevent the omission of small-sized text, the system automatically adjusts the macroblock size to [specific value]. Or keep However, the density threshold should be adjusted; for this purpose, this embodiment preferentially uses a fixed density threshold. The dimensions are adapted to the x265 default configuration.

[0065] Then, the total gradient energy within the window is calculated. :

[0066] in, This indicates the horizontal and vertical pixel coordinates of the top-left corner of the current processing macroblock (CTU); This represents the horizontal and vertical traversal offsets within the macroblock, with a traversal range of... arrive (correspond macroblocks); This represents the gradient magnitude of the coordinate point calculated in the previous step.

[0067] Then, an experience-based texture threshold is set. : because The setting is not a fixed value, but is determined based on the minimum readable character density model; typically, If a macroblock contains text, at least 5%-10% of its pixels are high-gradient edge points. Assuming the average gradient magnitude (black and white contrast) of these edge points is 200, the theoretical minimum energy... To address situations where text colors fade (e.g., gray annotations) or fonts are thin, this system introduces an adaptive calibration mechanism. Specifically, for standard high-contrast scenes, an initial calibration is set... For 4K resolution, due to the significant increase in the number of edge pixels, the threshold is automatically adjusted. ( (This refers to the resolution factor, such as 2.0). If the average brightness across the entire screen is low (dark mode), it indicates that the text contrast may be reduced, and the system will automatically adjust accordingly. Reduce the contrast to 50% (e.g., 40,000) to ensure that low-contrast text can also be recognized as an ROI; for example, if a certain CTU... Then the block is marked as ROI (1), and the corresponding quantization parameters are... Set to 10 (lossless / near-lossless); otherwise mark as background (0), corresponding to Set to 35, this binarization mask serves as the input for the QP offset table, and so on.

[0068] S353. Spatial Adaptive Quantization Mapping Based on the above mask, the module constructs the final QP Map; that is, for the ROI region, it configures... This extremely low QP value ensures a very small quantization step size, preserving high-frequency details; for the background region, the configuration... This high QP value allows the quantizer to discard most of the high-frequency coefficients, achieving an extremely high compression ratio.

[0069] As a result, the system reduces the average bitrate of a typical operations and maintenance desktop to the level of 50KB / s while ensuring text readability.

[0070] S360 Linear Interpolation Rate Adjustment Mechanism The aforementioned linear adjustment logic essentially constitutes a proportional (P) controller, used to adjust the rate of change. In the linear adjustment region (i.e.) Smoothly adjust the bitrate during the process, where The lower limit of the linear adjustment range (corresponding to the aforementioned static judgment threshold) (e.g., 5%) The upper limit of the linear adjustment range (corresponding to the aforementioned dynamic judgment threshold) (e.g., 30%) The maximum allowed target bitrate (e.g., 3000 Kbps); The minimum allowed target bitrate (e.g., 800 Kbps); the error input here is... Control gain Output control quantity .

[0071] In order to suppress the factor The system uses a first-order IIR low-pass filter to address the sudden changes in bit rate caused by instantaneous jitter.

[0072] Correspondingly, the discrete-time difference equation is:

[0073] in: Indicates the first The smoothed target bit rate of the actual frame sent to the encoder; Indicates the previous frame ( The actual bitrate; This represents the instantaneous target bitrate calculated based on the current frame change rate (i.e., the above). ); The smoothing factor (range 0-1); where This determines the lag in bitrate adjustments; if If the bitrate is too small (approaching 0), the bitrate will follow too quickly, easily causing a breathing effect in the image; if An excessively large value (approaching 1) results in a sluggish bitrate response, potentially causing initial blurring in dynamic scenes. This embodiment, based on experimental testing, uses a smoothing factor here. This achieves the optimal balance between smoothness and response speed. If the bitrate was 1500Kbps in the previous step and the current calculated value is 2000Kbps, then the actual output will be... Kbps, thus effectively avoiding the risk of encoder buffer overflow or underflow, and so on.

[0074] S400. Recording Control and File Packaging Stage In this embodiment, the state machine-based recording task management and multi-stream encapsulation includes the following sub-steps: S410. Multimodal Triggered State Machine The recording control module maintains a finite state machine (FSM) that transitions between the IDLE, RECORDING, and PAUSED states based on input events. a. Scheduling trigger: Parse the Crontab-style schedule_table, if the current system time... This triggers EVT_TIME_MATCH.

[0075] b. Semantic Trigger: Real-time matching of regular expressions in the command stream, where the rule Regex: "sudo|rm-rf" is defined. When a captured character sequence matches this rule, EVT_DANGEROUS_CMD is triggered, and the metadata label Security_Audit is set.

[0076] c. Abnormal Trigger: Receive device alarms via SNMPTrap; if a CPU_LOAD_HIGH Trap is received, trigger EVT_SYSTEM_ALARM.

[0077] S420 MP4 package conforming to ISO / IEC 14496-12 standard. The data storage module initializes the MP4Muxer component and creates a file container conforming to the ISOBMFF standard. Specifically: a. Video track construction: Create a trak box of type vide; specifically, write the H.265 NALU stream output from S300 into the mdat box and update stts (timestamp table) and stsz (sample size table).

[0078] b. Command Track Construction: Create a trak box of type sbtl (subtitle); specifically, encapsulate the command text into a Tx3g or SRT format sample and write it into the corresponding time slot.

[0079] c. Sharding and Indexing: Enable automatic sharding strategy. When writing bytes... When GB is reached, close the current moov box and create a new file; at the same time, write the file path, start and end times, and event tags to the SQLite index database records.db.

[0080] S421.MP4 Container Box Structure and Custom Metadata Encapsulation To achieve efficient encapsulation and retrieval, the data storage module performs in-depth MP4 Box structure customization; specifically, the module first writes a standard ftyp Box to declare compatibility.

[0081] Specifically, for private device information, the module constructs a custom UUID Box, which is filled with a 16-byte UUID identifier and serialized JSON device metadata.

[0082] For H.265 streams, the module writes the VPS / SPS / PPS parameter set to the hvcC Box (located under the stsd table entry).

[0083] For control commands, the module creates a separate track and marks handler_type='meta' in the hdlr box. The command binary data is stored as a sample in mdat, which not only supports subtitle display but also binary parsing by the background auditing program.

[0084] In addition, the module implements a periodic fast-start or pre-fill update mechanism for moovBox, ensuring that even if the file is not closed properly (such as in the event of a power outage), the written mdat data can still be recovered by rebuilding the index.

[0085] S500. Playback and Audit Demonstration In this embodiment, the front-end player loads the MP4 file and parses out the video track and subtitle track. Subsequently, the video decoder (such as the WASM version of FFmpeg) renders the screen; at the same time, the subtitle renderer overlays the instruction text as a semi-transparent layer on top of the video.

[0086] When the auditor clicks on a marker on the timeline, the player queries the SQLite index to obtain the corresponding byte offset and directly executes an HTTPRange request to achieve millisecond-level image positioning.

[0087] The following examples illustrate this point.

[0088] The method for recording system desktop based on IPKVM provided by this invention includes the following steps: 1. Configuration of the association between the IPKVM device and the target device Device Information Registration: In the embedded control system of the IPKVM device, a mapping table of "target device - IPKVM port" is established to record the unique identifier of each target device (such as device ID, hostname, IP address), the corresponding IPKVM physical port number or logical channel number, and the basic parameters of the target device such as desktop resolution and refresh rate. Network parameter configuration: Configure the network parameters of the IPKVM device (such as static IP, subnet mask, gateway) to ensure network connectivity between IPKVM and the target device, remote operation and maintenance terminal, and storage server (local / cloud); at the same time, enable the "control command log module" of IPKVM to collect control signals such as keyboard key codes, mouse coordinates and click events.

[0089] 2. Synchronous acquisition of desktop video stream and control commands Video stream acquisition: IPKVM acquires the original desktop video stream (in RGB or YUV format) from the target device through video interfaces such as VGA, HDMI, and DP, and converts the original video stream into digital video frames that can be processed by the IPKVM embedded system through the built-in video acquisition chip (the frame rate is consistent with the refresh rate of the target device, such as 60fps or 30fps). Control command acquisition: IPKVM's control command log module captures keyboard commands (such as ASCII codes and shortcut key combinations) and mouse commands (such as left click, right click, scroll wheel, and coordinates) sent from the remote operation and maintenance terminal to the target device in real time, and adds a timestamp (accurate to milliseconds) to each command. Synchronization Alignment: Based on the acquisition timestamp of the video frame, control commands within the same time window (e.g., 10ms) are associated with the video frame to generate a synchronization data packet of "video frame-control command". The data packet contains: video frame data, control command set, target device identifier, and acquisition timestamp.

[0090] 3. Adaptive compression coding of video streams Compression algorithm selection: Considering the characteristics of desktop video streams—high proportion of static frames, numerous text / lines, and minimal dynamic changes—a hybrid compression strategy of "H.265 (HEVC) + static frame optimization" is adopted. When the video frame is a static image (such as desktop icons or text documents), the H.265 intraframe (I-frame) compression mode is enabled, and the "lossless compression submodule" is used for the text area to preserve the clarity of the text edges; When the video frame is a dynamic scene (such as window dragging or video playback), the H.265 inter frame (P frame / B frame) compression mode is enabled, which only encodes the inter-frame changing area to reduce the amount of data. Compression parameters are adaptively adjusted: The IPKVM embedded system analyzes the dynamic complexity of video frames (such as the inter-frame pixel change rate) in real time and automatically adjusts the compression bitrate: when the dynamic complexity is high (such as the change rate > 30%), the bitrate is adjusted to 2-4 Mbps; when the dynamic complexity is low (such as the change rate < 5%), the bitrate is adjusted to 500-1000 Kbps, ensuring a balance between the video clarity and storage efficiency after compression.

[0091] 4. Flexible control and triggering of recording tasks Recording mode settings: Supports three recording trigger modes, which users can configure through IPKVM's web management interface or remote API. Manual trigger mode: Operation and maintenance personnel send "start recording" and "stop recording" commands through remote operation and maintenance terminals. IPKVM starts / stops the recording task after receiving the command. Time-triggered mode: Users set a recording time period (e.g., 9:00-18:00 daily), and the IPKVM embedded system automatically starts and stops recording according to the built-in clock; Event-triggered mode: Set a trigger event (such as remote maintenance terminal logging into IPKVM, the target device executing a specific command (such as "sudo" or "regedit"), or the target device experiencing an abnormal state (such as CPU utilization > 90%)). When IPKVM detects the trigger event, it will automatically start recording and stop recording after a preset time (such as 30 seconds) after the event ends. Recording parameter configuration: Users can configure the storage path of recorded files (local SD card / hard drive, remote NAS, cloud storage), file segment size (e.g., one file per 1GB), and file format (encapsulated as MP4, including video stream and control command subtitle stream).

[0092] 5. Storage and retrieval of recorded data Data storage: IPKVM encapsulates the compressed video stream and control instruction subtitle stream (in SRT format, including timestamps and control instruction content) into an MP4 file, and embeds the target device identifier and recording start time (e.g., "Device_001_20241001_090000.mp4") in the file name, and saves it to a local or remote storage server according to the configured storage path; Index creation: IPKVM creates index information for each recording file, including the target device ID, recording time period, trigger mode, and key events (such as abnormal commands and login events), and uploads the index information to the index server, allowing users to search by "device ID", "time range", and "event type". Data backup: Supports automatic backup strategies for recorded files, such as dual backups of local storage and cloud storage, to avoid data loss.

[0093] 6. Playback and auditing of recorded files Synchronous playback: When users open a recorded file through the IPKVM web management interface or a dedicated client, they can select the "video + control command synchronous playback" mode. During playback, the video screen plays in chronological order, and the control commands are superimposed on the screen as subtitles (such as "2024-10-01 09:05:23 Keyboard: Ctrl+C" "2024-10-01 09:05:25 Mouse: Left click (120,340)"), realizing synchronous tracking of "operation-screen". Audit Markers: During playback, users can add audit marks to key time points (such as the time of failure or abnormal operation) and enter remarks. The mark information is stored in association with the recording file, which facilitates the generation of subsequent audit reports.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0095] This embodiment also provides a system desktop recording system based on IPKVM, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0096] Figure 2 This is a structural block diagram of a system desktop recording system based on IPKVM according to an embodiment of the present invention, as shown below. Figure 2 As shown, the system includes: 1. Hardware Architecture Layer The system is built on an ARM Cortex-A53 quad-core processor (SoC) and includes the following core hardware components: Video capture front-end: Includes an FPGA video bridging chip (such as ITE6604), configured to physically connect to the VGA or HDMI output port of the target server. This front-end integrates a hardware EDID emulator for communicating display parameters to the target server.

[0097] USB Device Controller (UDC): Configured in Gadget mode, it connects to the target server's USB port via a USB cable and emulates a standard USB keyboard and mouse composite device.

[0098] Hardware codec engine (VPU): A dedicated H.265 encoding hardware core integrated inside the SoC, supporting real-time encoding capabilities up to 4K@60fps.

[0099] Storage interface: Includes a high-speed eMMC controller (for storing system firmware and database) and an SDIO interface (for expanding recording storage).

[0100] 2. Software module layer The system runs on a trimmed-down version of the Linux 5.10 real-time operating system, and its user-space applications are divided into the following independent functional modules: 1. Device association configuration module, which is configured as a lightweight web service process (based on Lighttpd).

[0101] Functionality configuration: It is configured to receive user configuration requests via HTTP / RESTful API and maintain a persistent SQLite database config.db.

[0102] Network stack management: It directly manipulates the kernel routing table and firewall rules through the netlink interface to ensure network isolation in the management plane.

[0103] 2. Synchronous acquisition module, which is set as the core acquisition daemon (CaptureDaemon).

[0104] Video Stream Pipeline: It establishes a zero-copy DMA channel with the FPGA front end via the V4L2 API ( / dev / video0). Internally, it maintains a ring buffer to smooth acquisition jitter.

[0105] Command Stream Pipeline: It captures HID messages written to the USB Gadget endpoint in real time by listening to the / dev / hidg0 character device.

[0106] Clock synchronizer: It holds a monotonic clock handle and is responsible for stamping every frame of video and every HID message entering the system with a uniform nanosecond-level timestamp.

[0107] 3. Adaptive compression module, which is a customized wrapper around the FFmpeg library.

[0108] Analysis Engine: It contains a pixel difference calculation kernel optimized with SIMD instructions, configured to process raw RGB data in a pipeline manner, and outputs... Indicators and texture masks.

[0109] Encoding Controller: It holds the context handle of the H.265 encoder (x265). It is configured to dynamically adjust the gop_size, rc.bitrate, and picture.quantOffsets parameters in the x265_param structure at the frame level based on the output of the analysis engine.

[0110] 4. Recording Control Module This module is configured as a finite state machine (FSM) and performs the following operations: Trigger logic: It subscribes to system logs (Syslog) and SNMP Trap messages, and matches them against a pre-defined regular expression rule base in real time.

[0111] Session management: In response to trigger signals, it is responsible for coordinating the start and stop of the acquisition module and the storage module, ensuring the safe closure of file handles and the complete writing of buffers to disk.

[0112] 5. Data storage module This module is implemented as a multi-format multiplexer.

[0113] Container encapsulation: It integrates the libavformat library and is configured to interleave and encapsulate the encoded H.265 NALU stream and a custom binary instruction stream into an MP4 container.

[0114] IO scheduling: It implements an asynchronous IO write strategy, using large blocks of memory (such as 16MB) as a write cache to reduce wear and fragmentation of the underlying Flash memory.

[0115] Indexing service: It simultaneously submits metadata records to the records.db database while writing files, supporting fast SQL-based retrieval.

[0116] 3. Inter-module communication (IPC) The software modules communicate with each other through System V shared memory and Unix domain sockets; for large amounts of data, the original video frames are transferred between the acquisition and compression modules through a shared memory pool to avoid memory copying, and signaling such as recording start / stop and configuration changes are synchronized between modules through a socket broadcast mechanism.

[0117] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0118] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0119] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0120] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0121] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0124] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A system desktop recording method based on IPKVM, characterized in that, include: Establish the association configuration between the IPKVM device and the target device, and generate a device mapping table; The IPKVM device acquires the raw video stream from the target device and converts it into digital video frames. At the same time, it captures the control signals sent by the remote operation and maintenance terminal, and stamps the digital video frames and the control signals with millisecond-level timestamps based on a unified clock source. Based on the timestamps, it associates the video frames and control signals within the same time window to generate synchronization data packets. Adaptive compression coding is performed on the video frames in the synchronization data packet, the adaptive compression coding including calculating the inter-frame pixel change rate of the current video frame relative to the reference frame; The dynamic complexity of the current frame is determined based on the inter-frame pixel change rate, and the compression mode and encoding parameters are selected and adjusted according to the dynamic complexity; wherein, the encoding parameters include frame type, quantization parameters and target bit rate; The recording task is started and stopped according to preset recording trigger conditions, including manual commands, time scheduling and specific event detection. The compressed video stream data and corresponding control command data are encapsulated to generate a recording file containing device identification and time information, and an index for retrieval is established.

2. The method according to claim 1, characterized in that, The calculation of the inter-frame pixel change rate of the current video frame relative to the reference frame includes: Iterate through every pixel at the same coordinate position in the current video frame and the reference frame; Calculate the absolute value of the component differences of the corresponding pixel in the RGB color space; If the absolute value exceeds the preset pixel tolerance threshold, the pixel is marked as a changed pixel, and the total number of changed pixels is accumulated. The inter-frame pixel change rate is determined based on the ratio of the total number of changed pixels to the total number of pixels in a single frame.

3. The method according to claim 1, characterized in that, The step of selecting the compression mode and adjusting the encoding parameters based on the dynamic complexity includes: If the inter-frame pixel change rate is less than the preset static determination threshold, the current frame is determined to be a static frame, the target bit rate is set to the first bit rate range, and the high-frequency texture region in the current frame is identified, and the quantization parameter of the high-frequency texture region is set to a lossless encoding value. If the inter-frame pixel change rate is greater than a preset dynamic determination threshold, the current frame is determined to be a dynamic frame, and the target bit rate is set to the second bit rate interval; wherein, the lower limit of the second bit rate interval is greater than the upper limit of the first bit rate interval.

4. The method according to claim 3, characterized in that, The identification of high-frequency texture regions in the current frame includes: Perform edge detection operations on the luminance components of the current frame to generate an edge gradient map; The edge gradient map is divided into several macroblocks of a preset size; Calculate the average gradient magnitude within each macroblock; if the average gradient magnitude exceeds a preset texture density threshold, then mark the corresponding macroblock region as the high-frequency texture region.

5. The method according to claim 1, characterized in that, The step of controlling the start and stop of the recording task according to the preset recording trigger conditions includes: Real-time analysis of the content of the control signals and the system status data of the target device; When the control signal is detected to contain a preset high-risk instruction string, or when the resource occupancy rate index in the system status data exceeds a preset warning value, a trigger recording signal is generated. Recording begins upon receiving the trigger recording signal and stops after a preset buffer time following the end of the trigger event.

6. The method according to claim 1, characterized in that, The step of encapsulating the compressed video stream data with the corresponding control command data to generate a recording file containing device identifier and time information includes: Create a media container file containing video tracks and subtitle tracks; Write the compressed video stream data into the video track; The control command data is converted into text format and written into the subtitle track according to its associated timestamp, so as to realize the synchronous overlay display of control commands and video images.

7. The method according to claim 1, characterized in that, The method further includes: The recorded file is parsed, and the video stream data and control command data are decoded simultaneously. The video frame is rendered on the playback interface, and the control command content corresponding to the current moment is rendered in the form of semi-transparent subtitles at the specified layer position of the frame. In response to a user's audit operation, audit marker data is inserted at a specific position on the timeline, and the audit marker data is associated with the index information.

8. A desktop recording system based on IPKVM, characterized in that, include: The device association configuration module is used to establish and store the mapping relationship between IPKVM device ports and target devices, as well as network configuration parameters. The synchronous acquisition module is used to acquire video signals from the target device through the video interface and convert them into digital frames, capture peripheral control commands through the control interface, and generate synchronous data packets with timestamps based on the system clock. The adaptive compression module is used to calculate the inter-frame pixel change rate, thereby distinguishing between static and dynamic images, and dynamically assigning encoding modes and corresponding quantization parameters to them respectively. The recording control module is used to monitor trigger events and generate recording start / stop control signals according to preset logic; The data storage module is used to encapsulate the encoded audio and video data and control command stream into a standard format file and write it to a local or remote storage medium.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 7.

Images