Memory-based video playback method, system, device and readable storage medium

Abstract

The application discloses a memory-based video playback method, system and device and a readable storage medium, relates to the technical field of video playback, and comprises the following steps: preprocessing acquired image data to obtain an image frame sequence, wherein each frame of data in the image frame sequence is a scheduling record, the scheduling record contains image index information of a plurality of virtual video devices belonging to a same synchronization group at a same scheduling moment, and the frames are sequentially sorted in ascending order of time stamps; a video playback layer is controlled to read current frame data from the image frame sequence and write the current frame data into a shared memory, and send a file descriptor corresponding to the shared memory to a virtual video device; and a sensor receiving layer is controlled to acquire the file descriptor through the virtual video device and access and process the current frame data in the shared memory based on the file descriptor, so that zero-copy video playback is realized. The application realizes video playback through the shared memory, can effectively reduce cost and deployment complexity, and improves flexibility.

Description

Technical Field

[0001] This application relates to the field of video re-annotation technology, specifically to a memory-based video re-annotation method, system, device, and readable storage medium. Background Technology

[0002] In autonomous driving systems, cameras, as core sensors, are widely used in tasks such as lane detection, target recognition, and obstacle tracking. To verify the stability and robustness of the visual perception module, it is often necessary to re-inject the acquired or constructed image sequences into the system for functional testing, simulation verification, and scene reproduction. To achieve this, the industry commonly uses "video back-injection" technology, which involves re-injecting existing image frame data into the camera data path for the testing, verification, or debugging of the perception module, making the entire system "believe" that the images come from real sensors.

[0003] The current mainstream technical approach is to use a video re-injection card, which essentially simulates one or more cameras through hardware, enabling the receiving platform to collect image frame data received from the link. (See [link]). Figure 1 As shown, its typical workflow is as follows:

[0004] (1) Host driver: The video re-injection card needs to be installed on a host device, and the host device uses a specific driver to configure the re-injection card, load images and control re-injection;

[0005] (2) Image data loading: The host transmits the pre-prepared image frames to the frame buffer of the video injection card through communication interfaces such as USB and PCIe (Peripheral Component Interconnect Express, high-speed serial computer expansion bus standard) as the image source for the analog camera signal output;

[0006] (3) Signal output: The video injection card can output simulated camera video signals to the receiver through high-speed serial interfaces such as GMSL (Gigabit Multimedia Serial Link) to achieve an image data injection effect similar to that of a real camera;

[0007] (4) The receiver uses a deserializer to restore the received camera video signal to a MIPI-CSI (a serial communication interface) data stream and connects it to the camera interface module CIF of the system-on-a-chip (SoC). The CIF completes the channel access and converts the data into an image frame data stream that conforms to the image signal processor (ISP) input specification and sends it to the ISP to complete the image reception and processing process.

[0008] Therefore, the existing technology for video back-injection using video back-injection cards has the following problems: 1) High cost: High-performance video back-injection cards are not only expensive, but also need to be installed on a host that supports specific drivers, resulting in high system costs. When more cameras are needed, multiple video back-injection cards may be required, which can multiply the cost; 2) Complex deployment: It involves dedicated interface connections, power supply, driver installation, platform matching, etc., and debugging is complex and time-consuming; 3) Insufficient flexibility: When the team's hardware resources are limited, it is difficult to flexibly and quickly meet the usage needs of each member. Summary of the Invention

[0009] This application provides a memory-based video re-injection method, system, device, and readable storage medium, which can solve the technical problems of high cost, complex deployment, and insufficient flexibility caused by using video re-injection cards to implement video re-injection in the prior art.

[0010] In a first aspect, embodiments of this application provide a memory-based video back-annotation method, the method comprising:

[0011] The acquired image data is preprocessed to obtain an image frame sequence. Each frame in the image frame sequence is a scheduling record, which contains the image index information of multiple virtual video devices belonging to the same synchronization group at the same scheduling time. The frames are sorted in ascending order of timestamp.

[0012] The video re-injection layer is controlled to read the current frame data from the image frame sequence and write it into the shared memory, and to send the file descriptor corresponding to the shared memory to the virtual video device;

[0013] The control sensor receiving layer obtains a file descriptor through the virtual video device and accesses and processes the current frame data in the shared memory based on the file descriptor to achieve zero-copy video re-injection.

[0014] In conjunction with the first aspect, in one implementation, the format of each frame of data consists of a single-frame scheduling entry and an image index entry; the single-frame scheduling entry includes global timestamp information, synchronization group information and its corresponding video channel information, the synchronization group information is used to mark the synchronization group ID corresponding to the frame of data, and the video channel information is used to mark all video channel IDs contained in its corresponding synchronization group; the image index entry includes the video processing channel ID and image data path corresponding to each video channel ID.

[0015] In conjunction with the first aspect, in one embodiment, after the step of controlling the video back-injection layer to read current frame data from the image frame sequence and write it to shared memory, the method further includes:

[0016] The target prediction time is determined based on the system trigger time of the next frame, the target IO read time, and the preset safety margin;

[0017] When the real-time system time reaches the target prediction time, the video re-injection layer is controlled to read the next frame data from the image frame sequence and pre-write it to the shared memory;

[0018] The target IO read time is predicted based on the average IO read time, burst jitter time, maximum IO read time and system load correction factor, and the time window length corresponding to the maximum IO read time is less than the time window length corresponding to the average IO read time.

[0019] In conjunction with the first aspect, in one implementation, the calculation expression for the target prediction time is:

[0020]

[0021] In the formula, Indicates the target prediction time. Indicates the system trigger time for the next frame. Indicates the time taken for the target I / O read. This indicates a preset safety margin.

[0022] In conjunction with the first aspect, in one implementation, the calculation expression for the target IO read time is:

[0023]

[0024] In the formula, This indicates the average I / O read time. Indicates the maximum I / O read time. Indicates the time taken for sudden shaking. This represents the system load correction factor.

[0025] In conjunction with the first aspect, in one implementation, the burst jitter time is determined based on the I / O read time of the previous frame and the I / O read time of the frame before that, and the system load correction factor is determined based on the CPU utilization rate within the statistical period and the proportion of time the CPU is in the I / O waiting state within the statistical period; wherein, the calculation expression for the burst jitter time is:

[0026]

[0027] The calculation expression for the system load correction factor is as follows:

[0028]

[0029] In the formula, This indicates the time taken for the previous frame's I / O read. This indicates the time taken to read the I / O from the frame two frames ago. This indicates the CPU utilization rate within the statistical period. This indicates the percentage of time the CPU spends in an I / O waiting state within the statistical period. and All represent weighting coefficients. This indicates the upper limit of the load correction factor.

[0030] In conjunction with the first aspect, in one implementation, sending the file descriptor corresponding to the shared memory to the virtual video device includes:

[0031] If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is lower than the preset requirement, then when the real-time system time and the timestamp information corresponding to the frame data are the same, the video back injection layer is controlled to send the file descriptor corresponding to the shared memory to the virtual video device.

[0032] If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is higher than the preset requirement, then when the timer count value reaches the preset trigger time, the video back injection layer is controlled to send the file descriptor corresponding to the shared memory to the virtual video device.

[0033] Secondly, embodiments of this application provide a memory-based video re-injection system, the system including a data storage module, a video re-injection layer, at least one virtual video device, and a sensor receiving layer;

[0034] The data storage module is used to preprocess the acquired image data to obtain an image frame sequence and store it. Each frame in the image frame sequence is a scheduling record, which contains the image index information of multiple virtual video devices belonging to the same synchronization group at the same scheduling time, and the frames are sorted in ascending order of timestamp.

[0035] The video back-injection layer is used to read the current frame data from the image frame sequence stored on the data storage module and write it into the shared memory, and to send the file descriptor corresponding to the shared memory to the virtual video device;

[0036] The sensor receiving layer obtains a file descriptor through the virtual video device and accesses and processes the current frame data in the shared memory based on the file descriptor to achieve zero-copy video re-injection.

[0037] In conjunction with the second aspect, in one implementation, the format of each frame of data consists of a single-frame scheduling entry and an image index entry; the single-frame scheduling entry includes global timestamp information, synchronization group information and its corresponding video channel information, the synchronization group information is used to mark the synchronization group ID corresponding to the frame of data, and the video channel information is used to mark all video channel IDs contained in its corresponding synchronization group; the image index entry includes the video processing channel ID and image data path corresponding to each video channel ID.

[0038] In conjunction with the second aspect, in one implementation, the video back-annotation layer is also used for:

[0039] The target prediction time is determined based on the system trigger time of the next frame, the target IO read time, and the preset safety margin;

[0040] When the real-time system time reaches the target prediction time, the video re-injection layer is controlled to read the next frame data from the image frame sequence and pre-write it to the shared memory;

[0041] The target IO read time is predicted based on the average IO read time, burst jitter time, maximum IO read time and system load correction factor, and the time window length corresponding to the maximum IO read time is less than the time window length corresponding to the average IO read time.

[0042] In conjunction with the second aspect, in one implementation, the calculation expression for the target prediction time is:

[0043]

[0044] In the formula, Indicates the target prediction time. Indicates the system trigger time for the next frame. Indicates the time taken for the target I / O read. This indicates a preset safety margin.

[0045] Furthermore, in one embodiment, the calculation expression for the target IO read time is:

[0046]

[0047] In the formula, This indicates the average I / O read time. Indicates the maximum I / O read time. Indicates the time taken for sudden shaking. This represents the system load correction factor.

[0048] In conjunction with the second aspect, in one implementation, the burst jitter time is determined based on the I / O read time of the previous frame and the I / O read time of the frame before that, and the system load correction factor is determined based on the CPU utilization rate within the statistical period and the proportion of time the CPU is in the I / O waiting state within the statistical period; wherein, the calculation expression for the burst jitter time is:

[0049]

[0050] The calculation expression for the system load correction factor is as follows:

[0051]

[0052] In the formula, This indicates the time taken for the previous frame's I / O read. This indicates the time taken to read the I / O from the frame two frames ago. This indicates the CPU utilization rate within the statistical period. This indicates the percentage of time the CPU spends in an I / O waiting state within the statistical period. and All represent weighting coefficients. This indicates the upper limit of the load correction factor.

[0053] In conjunction with the second aspect, in one implementation, the video back-injection layer is specifically used for:

[0054] If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is lower than the preset requirement, then when the real-time system time is the same as the timestamp information corresponding to the frame data, the file descriptor corresponding to the shared memory is sent to the virtual video device.

[0055] If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is higher than the preset requirement, then when the timer count value reaches the preset trigger time, the file descriptor corresponding to the shared memory is sent to the virtual video device.

[0056] Thirdly, embodiments of this application provide a memory-based video re-injection device, which includes a processor, a memory, and a memory-based video re-injection program stored in the memory and executable by the processor. When the memory-based video re-injection program is executed by the processor, it implements the steps of the aforementioned memory-based video re-injection method.

[0057] Fourthly, embodiments of this application provide a computer-readable storage medium storing a memory-based video re-annotation program, wherein when the memory-based video re-annotation program is executed by a processor, it implements the steps of the aforementioned memory-based video re-annotation method.

[0058] The beneficial effects of the technical solutions provided in this application include:

[0059] By preprocessing the acquired image data, an image frame sequence is obtained. Each frame in the image frame sequence serves as a scheduling record, containing image index information from multiple virtual video devices belonging to the same synchronization group at the same scheduling time. Frames are ordered in ascending order of timestamps, providing a data foundation for the synchronization, parallel processing, and efficient processing of multiple virtual video devices. The control video back-injection layer reads the current frame data from the image frame sequence and writes it to shared memory. It also sends a file descriptor corresponding to the shared memory to the virtual video devices, enabling the sensor receiving layer to obtain the file descriptor through the virtual video devices and access and process the current frame data in the shared memory based on the file descriptor, thus achieving zero-copy video back-injection. Therefore, this application implements video back-injection of multiple virtual video devices through shared memory—a purely software-based approach—without requiring any dedicated hardware such as video back-injection cards and serial decoders. This not only effectively saves costs but also simplifies deployment and debugging, improves R&D efficiency, and eliminates limitations imposed by other hardware resources, making it more flexible and convenient to use. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of a typical video recharge card system in the prior art;

[0061] Figure 2 This is a flowchart illustrating an embodiment of the memory-based video back-injection method of this application;

[0062] Figure 3 This is a schematic diagram of the data storage format involved in the embodiments of this application;

[0063] Figure 4 This is a schematic diagram of the overall video re-injection architecture involved in the embodiments of this application;

[0064] Figure 5 This is a schematic diagram of the overall video re-injection process involved in the embodiments of this application;

[0065] Figure 6 This is a schematic diagram of the hardware structure of a memory-based video re-injection device involved in the embodiments of this application. Detailed Implementation

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0068] Firstly, embodiments of this application provide a memory-based video back-injection method.

[0069] In one embodiment, reference is made to Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the memory-based video back-annotation method of this application. Figure 2 As shown, the memory-based video back-annotation method includes:

[0070] Step S10: Preprocess the acquired image data to obtain an image frame sequence. Each frame in the image frame sequence is a scheduling record, which contains image index information of multiple virtual video devices belonging to the same synchronization group at the same scheduling time, and the frames are sorted in ascending order of timestamp.

[0071] In this exemplary embodiment, a virtual video device refers to a camera input interface created in software within the operating system. It simulates the video output behavior of a physical camera, and thus can also be understood as a virtual camera. The number of virtual video devices corresponds to the number of physical cameras to be simulated; for example, if there are three physical cameras to be simulated, then there are also three virtual video devices. It is worth noting that the core function of the virtual video device is to control and transfer shared memory between the video injection layer and the sensor receiving layer, and to simulate the buffer queue management and scheduling behavior of the real camera driver, ensuring that the acquisition, queuing, and consumption of injection frames at the sensor receiving layer are consistent with the input from a real camera.

[0072] It should be noted that the image data acquisition methods in this embodiment include the following two types: The first type is historical data collected from actual vehicles or video injection cards, which records the image frame data input by real cameras or video injection cards. These are usually uncompressed or unprocessed raw images used for high-fidelity playback. The second type is based on publicly available multimodal autonomous driving datasets such as nuScenes and KITTI. Data conversion tools are needed to convert the data in the dataset into a standardized storage format (including converting the dataset image format into the graphic format required by the system, such as converting JPG to RAW images).

[0073] Furthermore, it is understandable that in multi-camera systems, there are often situations where multiple cameras need to work synchronously, and different application scenarios may require different cameras to work synchronously, such as multi-view, multi-surround view, driver's perspective, etc. Based on this, this embodiment will group the cameras that need to work synchronously according to the application scenario, so as to efficiently read a large number of image frames from the data storage module (such as local disk or cloud storage module). For example, multiple cameras that need to work synchronously in the multi-view scenario (such as cameras 0 to k, where k is a positive integer) are divided into synchronization group 1 (i.e., sync group1), and multiple cameras that need to work synchronously in the multi-view scenario (such as cameras k+1 to n, where n is a positive integer) are divided into sync group2.

[0074] Based on this, all image data will be preprocessed to form a unified storage format. For details, see [link to documentation]. Figure 3As shown, for image data corresponding to real cameras or video replay cards, the data can be directly recorded to form an image frame sequence. This means that the image data is organized frame by frame according to the synchronization group method, so that the same frame contains multiple-channel synchronization of data. For example, for sync group1, the image data corresponding to cameras 0 to k with the same timestamp are stored together as one frame. Similarly, for sync group2, the image data corresponding to cameras k+1 to n with the same timestamp are stored together as one frame. That is, the frames corresponding to each synchronization group are independent of each other and do not affect each other. All frame data are sorted in ascending order of timestamp to obtain the image frame sequence. It can be seen that each frame data in the image frame sequence of this embodiment serves as a scheduling record. It contains the image index information of multiple virtual video devices belonging to the same synchronization group at the same scheduling time. That is, the image data in the same frame have the same timestamp, and the frames are sorted in ascending order of timestamp to provide a data basis for the subsequent synchronization and playback of multi-channel image frames.

[0075] For image data in open source datasets, it is necessary to first convert the data and then organize it into an image frame sequence. That is, first convert the image data into a standardized storage format, and then organize the image data with the standardized storage format frame by frame according to the synchronization group. It should be noted that the specific organization method and principle are the same as those of the first type of image data, and will not be repeated here.

[0076] Step S20: Control the video back-injection layer to read the current frame data from the image frame sequence and write it into the shared memory, and send the file descriptor corresponding to the shared memory to the virtual video device.

[0077] Exemplary, see Figure 4As shown, this embodiment constructs the DMA (Direct Memory Access) data path of the virtual video device through shared memory. The virtual video device implements the same processes as the real camera, such as cache allocation REQBUFS, cache enqueue QBUF, cache dequeue DQBUF, setting image format S_FMT, starting data stream on, stopping data stream off, and cache data ready. This ensures that the lifecycle of the injected frame in the driver layer is completely consistent with the real DMA buffer, and makes the ISP and autonomous driving algorithm links in the sensor receiving layer completely unaware that the data comes from the simulation source.

[0078] Based on this, the video back-injection layer allocates an independent shared memory block for each virtual video device and generates a corresponding file descriptor (fd). This file descriptor (fd) acts as a "memory reference token," enabling multiple processes to securely share the same physical memory area without copying actual data between processes. Therefore, during video back-injection, the video back-injection layer reads image frames sequentially from the image frame sequence on the data storage module and caches them in shared memory. In other words, the video back-injection layer reads the current frame data from unified format storage data and writes it into the shared memory of the corresponding virtual video device, while also transmitting the corresponding file descriptor (fd) to the virtual video device. This shared memory approach allows for the transfer of image frames, enabling zero-copy image data flow. For example, if the current frame data contains image data corresponding to virtual video devices 0 to k, then the current frame data is written into the shared memory corresponding to virtual video device 0, the shared memory corresponding to virtual video device 1, ..., and the shared memory corresponding to virtual video device k, respectively. In summary, each virtual video device corresponds to an independent shared memory and scheduling thread to support multi-channel parallel injection and synchronous control. Among these, a circular buffer is preferred for implementing shared memory.

[0079] It is understandable that Dmabuf (a shared DMA buffer handle that supports shared memory between the system and devices as well as system processes) is essentially the low-level interface of the camera processing channel. Therefore, this embodiment will use this mechanism to simulate the input of a real camera, that is, Dmabuf is preferably used as a unified shared memory handle, and the image data is made to flow with zero copy by providing the file descriptor fd of Dmabuf to the virtual video device.

[0080] Step S30: Control the sensor receiving layer to obtain the file descriptor through the virtual video device and access and process the current frame data in the shared memory based on the file descriptor to achieve zero-copy video re-injection.

[0081] As an example, it should be noted that the sensor receiving layer refers to the module in the autonomous driving platform responsible for acquiring image data from the camera's data acquisition module. That is, it receives frame data from the virtual video device and submits it to the subsequent link processing, making the video re-injection method completely consistent with the processing link of the real camera after the sensor receiving layer.

[0082] See Figure 4 As shown, in this embodiment, after obtaining the file descriptor (fd), the virtual video device transmits the file descriptor fd to the sensor receiving layer. This allows the ISP (Instrument Service Provider) in the sensor receiving layer to access the corresponding shared memory and execute the image processing pipeline via the file descriptor fd. Finally, the processing result is transmitted to the Autonomous Driving Application (ADAA) so that the ADCA can make behavioral control decisions based on the processing result. It is evident that this embodiment injects frames into the ISP via shared memory, ensuring that the ISP receives a real DMA input. This guarantees that the processing from the ISP to the algorithm is completely consistent with that of a real camera, achieving an equivalent replacement for the traditional CIF hardware input path. Based on this, this embodiment uses shared memory to achieve zero-copy data transfer between the data storage module, the virtual video device, and the sensor receiving layer, constructing a purely software-based multi-channel video back-injection system that supports multi-threaded parallel and synchronous control.

[0083] As can be seen, this embodiment provides a simulation system that can completely simulate the hardware behavior of a camera at the software level. It not only achieves efficient utilization of shared memory, but also reproduces real camera behavior in memory. Unlike traditional simple caching or video playback methods, it provides complete camera link behavior simulation capabilities, ensuring the synchronization, low latency and reproducibility of multi-camera injection. Moreover, the entire process is passed through file descriptors of shared memory, realizing zero-copy data exchange across processes, which significantly reduces CPU load and latency.

[0084] Furthermore, in one embodiment, the format of each frame of data consists of a single-frame scheduling entry and an image index entry; the single-frame scheduling entry includes global timestamp information, synchronization group information and its corresponding video channel information, the synchronization group information is used to mark the synchronization group ID corresponding to the frame of data, and the video channel information is used to mark all video channel IDs contained in its corresponding synchronization group; the image index entry includes the video processing channel ID and image data path corresponding to each video channel ID.

[0085] As an example, in this embodiment, raw video data from real cameras, video re-injection cards, or public datasets are pre-organized into a well-structured, uniformly formatted, and efficiently playable multi-channel image frame sequence file, and stored on disk or in the cloud in a format adapted to the target system. It is understood that existing common formats (such as JPEG sequences, ROS bag, nuScenes raw formats, etc.) cannot express the "timing scheduling information required for the re-injection link." Based on this, this embodiment proposes a lightweight, data-driven video re-injection scheduling format, which consists of two parts: "single-frame scheduling entries" and "image index entries," to improve multi-channel synchronization performance and support synchronized playback through timestamp sorting.

[0086] For details, see Figure 3 As shown, a single-frame scheduling entry includes: timestamp (global timestamp information), sync-group (synchronization group information), and camera list (video channel information). The timestamp describes the absolute time of the frame on a unified timeline, ensuring that data from different cameras can be aligned to the same timeline. This addresses how to reconstruct the actual shooting timeline on the CPU during re-injection and supports loop playback and acceleration / deceleration playback based on the camera frame rate calculated from the timestamp. It's understandable that multi-camera systems often have multiple synchronization groups, such as multi-view, multi-surround, and driver's view. Different synchronization groups have different ID values. Therefore, the sync-group describes the synchronization group ID corresponding to the frame data. For example, the synchronization group ID for a multi-view scene is set to 0, and the synchronization group ID for a multi-surround scene is set to 1, to simultaneously support re-injection behavior of "strong synchronization within the group and weak synchronization between groups."

[0087] The camera list is used to mark all video channel IDs contained in its corresponding synchronization group, supporting a hybrid approach of "one frame from multiple cameras" and "separate writing from multiple camera groups," thereby achieving efficient multi-channel synchronization. For example, in a multi-channel panoramic scene, cameras 0 to k need to work synchronously, and the video channel IDs corresponding to cameras 0 to k are 0 to k, then cameras 0 to k all belong to the synchronization group with ID value 0. Therefore, the camera list corresponding to the synchronization group with ID value 0 contains 0 (i.e., video channel ID 0, or camera channel 0) to k (i.e., video channel ID k, or camerachannel k). It can be understood that a video channel refers to the data from a specific camera, and video channel ID k represents the data collected by the k-th camera.

[0088] Image index entries are responsible for mapping single-frame scheduling entries to actual image resources. They include the channel number, frame path, and corresponding relationships for each channel, and are organized by timestamp, making scheduling entirely data-driven. Specifically, image index entries contain the video processing channel ID (i.e., cameraapipeline ID) and image data path for each video channel ID. The video processing channel ID indicates the data link that a certain data path needs to take, and the image data path indicates the path where the image data corresponding to a certain video channel is stored. For example, camera channel 0 corresponds to cameraapipeline ID 0, meaning that the data from video channel ID 0 needs to go through the video processing channel with cameraapipeline ID 0. The image data path indicates the location information where the data acquired by different video channels is stored.

[0089] As can be seen, the purpose of designing the above format in this embodiment is not to store images, but to express the relationship between camera behavior and scheduling. That is, the combination of timestamp, sync-group and camera list can describe complex multi-camera synchronization modes, while the image index entries enable the back-injection system to consistently reproduce camera behavior across platforms. This format does not depend on specific image encoding formats or file naming rules, and can uniformly organize multi-source data from real vehicles, simulations, public datasets and other sources to achieve highly reproducible video back-injection scheduling.

[0090] Further, in one embodiment, sending the file descriptor corresponding to the shared memory to the virtual video device includes:

[0091] If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is lower than the preset requirement, then when the real-time system time and the timestamp information corresponding to the frame data are the same, the video back injection layer is controlled to send the file descriptor corresponding to the shared memory to the virtual video device.

[0092] If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is higher than the preset requirement, then when the timer count value reaches the preset trigger time, the video back injection layer is controlled to send the file descriptor corresponding to the shared memory to the virtual video device.

[0093] As an example, this embodiment provides two multi-channel synchronous triggering mechanisms to achieve synchronous control during multi-camera video re-injection: timestamp-trigger and soft-trigger. The timestamp-trigger unlocks and schedules frames according to the global timestamp order, based on the timestamp field pre-recorded in the data storage module and driven by system time to achieve triggered injection of each frame from multiple cameras. This is suitable for normal playback, fast playback, slow playback, and other scenarios. The soft-trigger refers to sending a synchronization signal to multiple channels in the same sync-group when the real-time system time reaches a preset trigger point, simultaneously injecting the corresponding frame into the virtual video device. Specifically, the system clock periodically sends soft-synchronization trigger signals to multiple video re-injection channels at specified times to notify the video re-injection layer to immediately play the current frame. This allows the video re-injection layer to simultaneously send file descriptors to multiple virtual video devices within the same sync group, completing the synchronous injection of all channels at a given time point. Since the soft-trigger timeline is entirely dependent on the trigger event, it can achieve stronger synchronization.

[0094] It is worth noting that this embodiment can preferably use different triggering methods for different synchronization groups based on the level of synchronization requirements. For example, soft-trigger synchronization can be used for channels within groups with high synchronization requirements (i.e., higher than the preset requirements, the specific setting of which can be determined according to actual needs and is not limited here), while timestamp-trigger synchronization can be used for groups with low synchronization requirements (i.e., lower than the preset requirements, such as low real-time groups or non-critical channels). Specifically, a target field can be set to characterize whether a synchronization group belongs to a high-synchronization-requirement group or a low-synchronization-requirement group. For example, if the target field of a synchronization group is set to 1, it indicates that the synchronization group belongs to a high-synchronization-requirement group; similarly, if the target field of a synchronization group is set to 0, it indicates that the synchronization group belongs to a low-synchronization-requirement group.

[0095] Based on this, if a synchronization group is detected to belong to a low synchronization group, then all virtual video devices within that synchronization group have synchronization requirements lower than the preset requirements. Therefore, when the real-time system time and the timestamp information corresponding to the frame data are the same, the video back-injection layer is controlled to send the file descriptor corresponding to the shared memory to the virtual video devices within that synchronization group. Conversely, if a synchronization group is detected to belong to a high synchronization group, then all virtual video devices within that synchronization group have synchronization requirements higher than the preset requirements. Therefore, when the timing counter value corresponding to the system clock timing reaches the preset trigger time, the video back-injection layer is controlled to send the file descriptor corresponding to the shared memory to the virtual video devices within that synchronization group.

[0096] As can be seen, this embodiment can achieve near-zero-time-difference synchronous back injection of cameras within a group through the above mechanism, while ensuring cross-group flexibility and system stability, and supporting functions such as fast playback, frame skipping, and speed-up playback, making it suitable for autonomous driving simulation and algorithm regression testing scenarios.

[0097] In addition, this embodiment can also select only timestamp-trigger or soft-trigger to achieve synchronization control. That is, it can control the video back-injection layer to send the file descriptor corresponding to the shared memory to the virtual video device by simply checking whether the real-time system time and the timestamp information corresponding to the frame data are the same, or it can control the video back-injection layer to send the file descriptor corresponding to the shared memory to the virtual video device by simply checking whether the timer count value reaches the preset trigger time, without having to distinguish between synchronization groups with different synchronization requirements.

[0098] Furthermore, in one embodiment, after the step of controlling the video back-injection layer to read the current frame data from the image frame sequence and write it to shared memory, the method further includes:

[0099] The target prediction time is determined based on the system trigger time of the next frame, the target IO read time, and the preset safety margin;

[0100] When the real-time system time reaches the target prediction time, the video re-injection layer is controlled to read the next frame data from the image frame sequence and pre-write it to the shared memory;

[0101] The target IO read time is predicted based on the average IO read time, burst jitter time, maximum IO read time and system load correction factor, and the time window length corresponding to the maximum IO read time is less than the time window length corresponding to the average IO read time.

[0102] Exemplary, it is worth noting that, in order to cope with the impact of dynamic disk jitter and system load changes, this embodiment proposes a predictive frame preloading method to accurately predict "when to read the data storage module" before each frame is actually required to be output. Among them, the average IO read time (i.e., the stable average read time) refers to the average IO read time of the past N_avg (N_avg is a positive integer) frames. The single-frame IO read time refers to the time from the start of reading all channel images belonging to this frame to the completion of reading all channel images belonging to this frame. The burst jitter time refers to the recent estimated burst jitter time, which can be determined according to the I read time estimates of the previous frame and the previous two frames. The maximum IO read time refers to the short-term extreme slow protection value, that is, the maximum IO read time within the past N_peak frames. It is worth noting that N_peak < N_avg (e.g., N_avg = 150, N_peak = 20). The system load correction factor is used to actively amplify the predicted value when the system operating load is large to start loading frame data earlier. It can be obtained by the linear combination of the CPU usage rate and the proportion of I / O wait time. Based on this, this embodiment can predict the target IO read time according to the average IO read time, burst jitter time, maximum IO read time, and system load correction factor. It should be understood that the average IO read time, maximum IO read time, and burst jitter time respectively represent the estimated time within the medium-term, short-term, and the most recent two time scales.

[0103] Then, based on the target IO read time, the system trigger time of the next frame (which represents the time when the next frame of data needs to be back-injected), and a preset safety margin, the target prediction time is determined, so that when the real-time system time reaches this target prediction time, the video back-injection layer reads the next frame of data from the image frame sequence and pre-writes it into the shared memory. That is, after the prediction time arrives, the "preloading thread pool" starts to read the disk, and the image frame data is directly written into the corresponding shared memory, making the frame enter the "ready" state to wait for the timestamp-trigger or soft-trigger to dequeue. It should be noted that the preset safety margin is used to offset the thread scheduling delay and kernel scheduling uncertainty, and its specific value can be adaptively adjusted according to the device performance. For example, it can be preferably set to 1 - 5 ms. It is worth noting that in this embodiment, the scheduling granularity for preloading prediction is the synchronization group, that is, each channel is not considered separately, but the overall time of all paths within the group is used for decision-making to calculate a more advanced preloading time to complete the loading under high load. It can be seen that the predictive frame preloading method in this embodiment has an adaptive ability, can significantly improve the multi-channel synchronization stability, eliminate the frame misalignment caused by disk jitter, and meet the millisecond-level synchronization requirements in the autonomous driving simulation scenario.

[0104] In summary, this embodiment provides an efficient memory buffer structure and image frame preloading mechanism: during the injection process, a multi-threaded asynchronous preloading mechanism is adopted to preload several frames into the buffer area before video playback, so as to realize parallel access of data storage module and shared memory, and improve the real-time performance and continuity of frame injection.

[0105] Furthermore, in one embodiment, the calculation expression for the target prediction time is:

[0106]

[0107] In the formula, Indicates the target prediction time. Indicates the system trigger time for the next frame. Indicates the time taken for the target I / O read. This indicates a preset safety margin.

[0108] As an example, in this embodiment, for the target prediction time The calculation can determine the system trigger time for the next frame. Target I / O read time and preset safety margin Substitute into the formula It is obtained through calculation. It should be noted that... The specific value is known and can be determined during the image data acquisition process.

[0109] Furthermore, in one embodiment, the calculation expression for the target IO read time is:

[0110]

[0111] In the formula, This indicates the average I / O read time. Indicates the maximum I / O read time. Indicates the time taken for sudden shaking. This represents the system load correction factor.

[0112] As an example, in this embodiment, the target IO read time The calculation can estimate the time spent over a medium-term timescale (i.e., the average I / O read time). Estimated latency over a short timescale (i.e., maximum I / O read latency) ) and the estimated timeouts for the two most recent timescales (i.e., the timeout for sudden jitter) The maximum value of the three and the system load correction factor By multiplying the results, the target I / O read time can be calculated. That is, the time taken to read the target I / O. The calculation formula is: .

[0113] Further, in one embodiment, the burst jitter time is determined based on the IO read time of the previous frame and the IO read time of the frame before that, and the system load correction factor is determined based on the CPU utilization rate within the statistical period and the proportion of time the CPU is in the IO waiting state within the statistical period; wherein, the calculation expression for the burst jitter time is:

[0114]

[0115] The calculation expression for the system load correction factor is as follows:

[0116]

[0117] In the formula, This indicates the time taken for the previous frame's I / O read. This indicates the time taken to read the I / O from the frame two frames ago. This indicates the CPU utilization rate within the statistical period. This indicates the percentage of time the CPU spends in an I / O waiting state within the statistical period. and All represent weighting coefficients. This indicates the upper limit of the load correction factor.

[0118] As an example, in this embodiment, the time taken for sudden jitter is... Based on the IO read time of the previous frame (i.e., the time spent on the most recent IO read) and the time spent on the IO read of the frame before that. (That is, the time taken for the second-to-last I / O read) is determined, that is... System load correction factor Based on CPU utilization within the statistical period The percentage of time the CPU spends waiting for I / O during the statistical period Determined, that is ,in, and These are all weighting coefficients, and their specific values ​​can be adjusted based on experience. This represents the upper limit of the load correction factor. Its specific value can be determined according to actual needs. For example, it can be preferably set to a value between 1.5 and 2.0.

[0119] In summary, in this embodiment, the video back-injection layer, virtual video device, and sensor receiving layer exchange frame data through shared memory, avoiding multiple memory copies and achieving zero-copy frame transmission. This approach can be widely applied to scenarios such as scene reproduction, extreme scenario evaluation, algorithm verification, and hardware-in-the-loop simulation in autonomous driving systems. The following, in conjunction with... Figure 5The data interaction among the three is described using one of the virtual video devices as an example.

[0120] It should be clarified that the interaction between the video back-injection layer and the virtual video device follows a typical queue model. Based on this, for the interaction of the first frame of data, the sensor receiving layer submits a blank buffer (empty frame) to the virtual video device, i.e., submitting the empty frame to the virtual video device's writable queue (i.e., Device can write) through a buffer enqueue. After detecting the empty frame, the virtual video device wakes up and transmits the empty frame to the video back-injection layer. After detecting the empty frame through polling, the video back-injection layer reads the current frame data corresponding to the virtual video device from the data storage module and writes it to the shared memory corresponding to the virtual video device. It also writes the file descriptor corresponding to the shared memory into the empty frame, dequeuing the frame. The next frame data is then retrieved from the data storage module and filled into the shared memory (i.e., performing the steps of Dequeue and read next file to buffer), entering the waiting queue, and continuously waiting for a trigger moment to enter the queue buffer (i.e., Queue onebuffer), and then re-enqueuing it to the virtual video device's readable queue (i.e., Device can write). (read); The virtual video device returns the filled frame to the sensor receiving layer, that is, wakes up the sensor receiving layer and writes the filled frame into the sensor receiving layer's receiving queue (i.e., Ringbuffer), thereby forming a complete frame stream closed loop in the system.

[0121] For the second frame of data and subsequent interactions, the sensor receiving layer submits an empty frame to the virtual video device. The virtual video device transmits the empty frame to the video re-injection layer. The video re-injection layer reads the file descriptor of the shared memory corresponding to the virtual video device from the shared memory and writes it into the empty frame, dequeuing the frame. It then retrieves the next frame of data from the data storage module and fills it into the shared memory, placing it in a waiting queue to be re-enqueued into the readable queue of the virtual video device at the trigger moment. The virtual video device returns the filled frame to the sensor receiving layer, thus forming a complete frame stream loop in the system. Therefore, this embodiment, without a physical camera or video re-injection card, uses a virtual video device to write image data from the data storage module into shared memory and injects the shared memory into the system's video acquisition link to simulate the output behavior of a real camera, achieving zero-copy video frame transmission without the need for an external acquisition DMA link, USB, or cross-SoC communication.

[0122] In summary, this embodiment completely migrates the video back-injection process to the software level. It achieves frame injection without physical transmission through a shared memory mechanism and a virtual video device interface, ensuring that the injected frames behave identically to real camera input at the sensor receiving layer, achieving the same link effect as a real camera. This allows for direct use in algorithm verification and simulation. Furthermore, it features specialized designs for multi-channel synchronization, timing control, memory reuse, and asynchronous preloading to achieve high-precision synchronization and real-time performance in multi-camera back-injection. A unified storage format for the multi-channel video frame structure is designed, with each frame containing a timestamp and multi-camera image data. This supports standardized storage and time-sequential playback of various image data sources (such as the open-source dataset nuscenes, real-vehicle capture, and video back-injection cards), enabling time-sequence management and synchronization control for convenient scene reproduction. By simulating real camera behavior through a virtual video device and employing a circular buffer and read / write queue mechanism, asynchronous preloading and timestamp / soft synchronization triggered injection of video frames are achieved. This ensures multi-camera frame-level consistency, improves the efficiency of frame reading and memory injection in the data storage module, and guarantees continuous real-time playback, thereby ensuring multi-channel synchronization and real-time performance.

[0123] Secondly, embodiments of this application also provide a memory-based video re-injection system.

[0124] In one embodiment, the memory-based video re-injection system includes: a data storage module, a video re-injection layer, at least one virtual video device, and a sensor receiving layer;

[0125] The data storage module is used to preprocess the acquired image data to obtain an image frame sequence and store it. Each frame in the image frame sequence is a scheduling record, which contains the image index information of multiple virtual video devices belonging to the same synchronization group at the same scheduling time, and the frames are sorted in ascending order of timestamp.

[0126] The video back-injection layer is used to read the current frame data from the image frame sequence stored on the data storage module and write it into the shared memory, and to send the file descriptor corresponding to the shared memory to the virtual video device;

[0127] The sensor receiving layer obtains a file descriptor through the virtual video device and accesses and processes the current frame data in the shared memory based on the file descriptor to achieve zero-copy video re-injection.

[0128] Furthermore, in one embodiment, the format of each frame of data consists of a single-frame scheduling entry and an image index entry; the single-frame scheduling entry includes global timestamp information, synchronization group information and its corresponding video channel information, the synchronization group information is used to mark the synchronization group ID corresponding to the frame of data, and the video channel information is used to mark all video channel IDs contained in its corresponding synchronization group; the image index entry includes the video processing channel ID and image data path corresponding to each video channel ID.

[0129] Furthermore, in one embodiment, the video back-injection layer is also used for:

[0130] The target prediction time is determined based on the system trigger time of the next frame, the target IO read time, and the preset safety margin;

[0131] When the real-time system time reaches the target prediction time, the video re-injection layer is controlled to read the next frame data from the image frame sequence and pre-write it to the shared memory;

[0132] The target IO read time is predicted based on the average IO read time, burst jitter time, maximum IO read time and system load correction factor, and the time window length corresponding to the maximum IO read time is less than the time window length corresponding to the average IO read time.

[0133] Furthermore, in one embodiment, the calculation expression for the target prediction time is:

[0134]

[0135] In the formula, Indicates the target prediction time. Indicates the system trigger time for the next frame. Indicates the time taken to read the target I / O. This indicates a preset safety margin.

[0136] Furthermore, in one embodiment, the calculation expression for the target IO read time is:

[0137]

[0138] In the formula, This indicates the average I / O read time. Indicates the maximum I / O read time. Indicates the time taken for sudden jitter. This represents the system load correction factor.

[0139] Further, in one embodiment, the burst jitter time is determined based on the IO read time of the previous frame and the IO read time of the frame before that, and the system load correction factor is determined based on the CPU utilization rate within the statistical period and the proportion of time the CPU is in the IO waiting state within the statistical period; wherein, the calculation expression for the burst jitter time is:

[0140]

[0141] The calculation expression for the system load correction factor is as follows:

[0142]

[0143] In the formula, This indicates the time taken for the previous frame's I / O read. This indicates the time taken to read the I / O from the frame two frames ago. This indicates the CPU utilization rate within the statistical period. This indicates the percentage of time the CPU spends in an I / O waiting state within the statistical period. and All represent weighting coefficients. This indicates the upper limit of the load correction factor.

[0144] Furthermore, in one embodiment, the video back-injection layer is specifically used for:

[0145] If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is lower than the preset requirement, then when the real-time system time is the same as the timestamp information corresponding to the frame data, the file descriptor corresponding to the shared memory is sent to the virtual video device.

[0146] If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is higher than the preset requirement, then when the timer count value reaches the preset trigger time, the file descriptor corresponding to the shared memory is sent to the virtual video device.

[0147] The functions of each part in the memory-based video back-injection system correspond to the steps in the memory-based video back-injection method embodiment, and their functions and implementation processes will not be described in detail here.

[0148] Thirdly, embodiments of this application provide a memory-based video re-injection device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0149] Reference Figure 6 , Figure 6This is a schematic diagram of the hardware structure of a memory-based video re-injection device involved in an embodiment of this application. In this embodiment, the memory-based video re-injection device may include a processor, a memory, a communication interface, and a communication bus.

[0150] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0151] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the memory-based video re-injection device, as well as interfaces used for interconnecting the memory-based video re-injection device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0152] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0153] The processor can be a general-purpose processor, which can call a memory-based video re-annotation program stored in memory and execute the memory-based video re-annotation method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the memory-based video re-annotation program is called can be referred to in the various embodiments of the memory-based video re-annotation method of this application, and will not be repeated here.

[0154] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0155] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0156] This application has a memory-based video re-annotation program stored on a readable storage medium, wherein when the memory-based video re-annotation program is executed by a processor, it implements the steps of the memory-based video re-annotation method described above.

[0157] The method implemented when the memory-based video back-injection program is executed can be referred to in various embodiments of the memory-based video back-injection method of this application, and will not be repeated here.

[0158] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0159] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0160] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0161] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0162] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0164] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A memory-based video back-annotation method, characterized in that, The method includes: The acquired image data is preprocessed to obtain an image frame sequence. Each frame in the image frame sequence is a scheduling record, which contains the image index information of multiple virtual video devices belonging to the same synchronization group at the same scheduling time. The frames are sorted in ascending order of timestamp. The video re-injection layer is controlled to read the current frame data from the image frame sequence and write it into the shared memory, and to send the file descriptor corresponding to the shared memory to the virtual video device; The control sensor receiving layer obtains a file descriptor through the virtual video device and accesses and processes the current frame data in the shared memory based on the file descriptor to achieve zero-copy video re-injection; Sending the file descriptor corresponding to the shared memory to the virtual video device includes: If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is lower than the preset requirement, then when the real-time system time and the timestamp information corresponding to the frame data are the same, the video back injection layer is controlled to send the file descriptor corresponding to the shared memory to the virtual video device. If it is detected that the synchronization requirement of the synchronization group to which the virtual video device belongs is higher than the preset requirement, then when the timer count value reaches the preset trigger time, the video back injection layer is controlled to send the file descriptor corresponding to the shared memory to the virtual video device.

2. The memory-based video back-annotation method as described in claim 1, characterized in that: Each frame of data is formatted as a single-frame scheduling entry and an image index entry. The single-frame scheduling entry contains global timestamp information, synchronization group information and its corresponding video channel information. The synchronization group information is used to mark the synchronization group ID corresponding to the frame of data, and the video channel information is used to mark all video channel IDs contained in its corresponding synchronization group. The image index entry contains the video processing channel ID and image data path corresponding to each video channel ID.

3. The memory-based video back-annotation method as described in claim 1, characterized in that, After the step of controlling the video back-injection layer to read the current frame data from the image frame sequence and write it to shared memory, the method further includes: The target prediction time is determined based on the system trigger time of the next frame, the target IO read time, and the preset safety margin; When the real-time system time reaches the target prediction time, the video re-injection layer is controlled to read the next frame data from the image frame sequence and pre-write it to the shared memory; The target IO read time is predicted based on the average IO read time, burst jitter time, maximum IO read time and system load correction factor, and the time window length corresponding to the maximum IO read time is less than the time window length corresponding to the average IO read time.

4. The memory-based video back-annotation method as described in claim 3, characterized in that, The calculation expression for the target prediction time is: In the formula, Indicates the target prediction time. Indicates the system trigger time for the next frame. Indicates the time taken to read the target I / O. This indicates a preset safety margin.

5. The memory-based video back-annotation method as described in claim 3 or 4, characterized in that, The expression for calculating the target I / O read time is: In the formula, This indicates the average I / O read time. Indicates the maximum I / O read time. Indicates the time taken for sudden jitter. This represents the system load correction factor.

6. The memory-based video back-annotation method as described in claim 5, characterized in that, The burst jitter time is determined based on the I / O read time of the previous frame and the I / O read time of the frame before that. The system load correction factor is determined based on the CPU utilization rate within the statistical period and the percentage of time the CPU spends in the I / O waiting state within the statistical period. The calculation expression for the burst jitter time is: The calculation expression for the system load correction factor is as follows: In the formula, This indicates the time taken for the previous frame's I / O read. This indicates the time taken to read the I / O from the frame two frames ago. This indicates the CPU utilization rate within the statistical period. This indicates the percentage of time the CPU spends in an I / O waiting state within the statistical period. and All represent weighting coefficients. This indicates the upper limit of the load correction factor.

7. A memory-based video re-injection system, characterized in that: The system includes a data storage module, a video re-injection layer, at least one virtual video device, and a sensor receiving layer. The data storage module is used to preprocess the acquired image data to obtain an image frame sequence and store it. Each frame in the image frame sequence is a scheduling record, which contains the image index information of multiple virtual video devices belonging to the same synchronization group at the same scheduling time, and the frames are sorted in ascending order of timestamp. The video back-injection layer is used to read the current frame data from the image frame sequence stored on the data storage module and write it into the shared memory, and to send the file descriptor corresponding to the shared memory to the virtual video device; The sensor receiving layer obtains a file descriptor through the virtual video device and accesses and processes the current frame data in the shared memory based on the file descriptor to achieve zero-copy video re-injection; Specifically, the video back-injection layer is used to send a file descriptor corresponding to the shared memory to the virtual video device when the synchronization requirement of the synchronization group to which the virtual video device belongs is lower than the preset requirement, and when the real-time system time and the timestamp information corresponding to the frame data are the same; if the synchronization requirement of the synchronization group to which the virtual video device belongs is higher than the preset requirement, the video back-injection layer is used to send a file descriptor corresponding to the shared memory to the virtual video device when the timer count value reaches the preset trigger time.

8. A memory-based video re-injection device, characterized in that, The memory-based video re-injection device includes a processor, a memory, and a memory-based video re-injection program stored in the memory and executable by the processor, wherein when the memory-based video re-injection program is executed by the processor, it implements the steps of the memory-based video re-injection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a memory-based video re-injection program, wherein when the memory-based video re-injection program is executed by a processor, it implements the steps of the memory-based video re-injection method as described in any one of claims 1 to 6.

Images