Image conversion and stream pushing method and device for enhancing real-time performance

Abstract

The invention discloses an image conversion and flow pushing method and device for enhancing real-time performance, and the method comprises the steps: employing a multi-thread architecture, collecting images through a producer thread, storing the images in a double-end queue, carrying out the batch extraction of image frames to a temporary container through a conveying thread, and awakening a thread pool for parallel processing; in image conversion optimization, integer fixed-point calculation is designed to replace traditional floating-point calculation, a YUV conversion formula coefficient is rounded after being scaled by 256 times, and calculation time consumption is reduced; a YUV data pointer is directly mapped to an AVFrame structural body of the FFmpeg through a coexistence internal sharing mechanism, so that the deep copy overhead of a memory is eliminated; the temporary container capacity is dynamically adjusted based on an error accumulator, production and consumption rates are matched in real time, and queue overflow or thread starvation is avoided. According to the technical scheme of combining multi-thread parallel processing and image conversion optimization, real-time efficient flow pushing is achieved, and the method is particularly suitable for resource-limited scenes such as unmanned aerial vehicles and embedded equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and video stream transmission, and specifically to a method and device for image conversion and streaming with enhanced real-time performance, which is particularly suitable for low-latency video stream transmission of edge computing devices such as drones in high-pixel image processing scenarios. Background Art

[0002] With the in-depth application of drones in agricultural inspections, disaster monitoring, logistics and transportation, etc., the acquisition, conversion and streaming technology of real-time video streams have become a key support. However, when processing high-resolution images, existing solutions generally face technical bottlenecks such as insufficient real-time performance and low resource utilization. The current mainstream implementation methods mainly rely on the following two technical solutions, but both have significant limitations: 1. Solution 1: Local storage and then transmission ( Figure 1 shown): Taking DJI M30 as an example, the drone video stream is captured in OpenCV's Mat format to form Mat video frames, which are stored frame by frame as local MP4 video files using the VideoWrite tool and then uploaded to a remote server via the HTTP / FTP protocol for download and playback by the client.

[0003] The interface of this solution is simple and convenient, and does not require too many real-time encoding and decoding operations. Because it is saved in advance, there will be no data loss caused by inconsistent streaming and decoding speeds. During video playback, the smoothness of local playback mainly depends on whether the video file has been completely transmitted to the local device. Specifically, playback will only begin when all data of the video file is completely transmitted to the local device. This means that during playback, network conditions will not affect local playback, because the content being played has been stored locally and does not need to rely on real-time network transmission. However, this solution has the following core defects: a) I / O blocking causes performance collapse: During the video transmission process, the transmission of video frames through the network is an I / O intensive operation in itself. When these video frames arrive locally, they need to be further written to the disk, which involves another I / O intensive operation. Since these operations take a lot of time, the entire process is difficult to ensure real-time performance. Saving videos requires writing to the disk frame by frame, causing the program to frequently block I / O operations, and the CPU cannot efficiently perform subsequent tasks. The overall processing rate is limited by the dual performance bottlenecks of network transmission and disk throughput.

[0004] b) End-to-end delay is uncontrollable: Videos must be fully stored before they can be transmitted, resulting in a link delay of up to several seconds from acquisition to playback, which cannot meet real-time monitoring needs (such as emergency situation identification during drone inspections).

[0005] 2. Solution 2: Real-time streaming based on the producer-consumer model ( Figure 2 , Figure 3 shown) To improve real-time performance, the existing technology uses the FFmpeg tool chain combined with the RTMP protocol to obtain the Mat frame and convert it into the YUV420 format, fill the AVFrame structure and encode it, and then push the stream to the streaming server (such as SRS / Nginx-RTMP-Module / Nginx-HTTP-FLV-Module) through the RTMP protocol. The client pulls the stream from the streaming server through protocols such as HTTP-FLV, WebRTC or HLS (such as Figure 2 This solution optimizes the processing flow through the producer-consumer model (dual-thread collaboration): Thread A (producer) maintains a double-ended queue of size N, inserts data from the tail of the queue, and data exceeding N will overflow from the head of the queue. Thread B (consumer) takes the head data from the queue exclusively, converts Mat to YUV420 format, and then fills the data in the AVFrame structure. Finally, it is pushed after encoding, and the cycle repeats. The processing logic is as follows: Figure 3 As shown, here N=5 is taken as an example.

[0006] Under ideal conditions, the speed at which data is added to the queue and the speed at which data is retrieved should basically match or the former should be slightly greater than the latter (if the former is less than the latter, consumer starvation will occur, that is, the video frames have been consumed and the consumer cannot obtain the latest video frames, resulting in stuck and discontinuous streaming video), and the queue length should be kept in a smaller range to ensure real-time and streaming video continuity.

[0007] Under this design, when the pixel volume of a single Mat data is small, the speed of data retrieval and processing will be greater than the speed of data filling the queue. At this time, an appropriate amount of delay waiting can be added to consumer thread B to ensure that the data processing speed matches the speed of filling the queue, so that there will be no new data arriving in thread B, and the streaming video will not be briefly interrupted due to vain waiting.

[0008] However, this solution has the following core flaws: a) Queue overflow and frame skipping in large pixel scenes: Figure 3As shown in the figure, when the pixel of the image to be transmitted is large and the image quality is not to be sacrificed, the speed at which thread B takes out and processes the data from the queue will be much slower than the speed at which thread A fills in the data, while thread A continues to fill in data to ensure the real-time nature of the video frames, resulting in a lot of data in the queue that overflows before being processed, such as data 1. At this time, during the video push process, the interval between the two frames is too large (the overflowed data is not processed, and the subsequent processing is the data with an interval of several frames), which causes freezes and frame skipping when the pushed video is played, reducing the user viewing experience.

[0009] b) Resource waste of single-threaded serial processing: Figure 3 As shown in the process, the consumer thread (thread B) only relies on a single core to perform serial operations and cannot utilize the parallel computing capabilities of a multi-core CPU (for example, an 8-core CPU only occupies 1 core), resulting in idle hardware resources and efficiency bottlenecks.

[0010] c) The contradiction between high-precision conversion and the computing power of embedded devices: The existing OpenCV YUV conversion interface uses floating-point operations (such as coefficients 0.299, 0.587, etc.). Although it ensures image quality, its computational complexity causes the single-frame conversion time to account for more than 60% (such as 50ms / frame), which aggravates processing delays and energy consumption pressures in edge devices such as drones.

[0011] d) Redundant memory copies increase latency: e.g. Figure 2 In the "Fill AVFrame structure" step, the existing solution copies the YUV420 data to the AVFrame structure through deep copy (memcpy), which increases the time consumption of single-frame memory operations (for example, from 1ms to 5ms), further weakening the real-time performance.

[0012] In summary, if Figures 1 to 3 As shown in the figure, the existing solutions have significant defects in real-time performance, resource utilization and algorithm efficiency. Especially in high-resolution image processing scenarios, queue overflow, single-thread bottleneck and memory redundancy problems seriously restrict the real-time transmission capability of drone video streams. How to systematically solve the above technical pain points through multi-threaded scheduling optimization, lightweight algorithm design and memory management innovation, and provide an efficient solution for real-time video stream processing of edge devices, is a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0013] The present invention aims to solve the problems of insufficient real-time performance and low resource utilization in the prior art of high-pixel image streaming. Through multi-threaded parallel processing and image conversion optimization, the processing speed is significantly improved and the computing load is reduced to ensure the continuity of the video stream. It is suitable for embedded and edge computing scenarios.

[0014] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present application discloses a method for image conversion and streaming with enhanced real-time performance, comprising the following steps: Acquire original image data from an image source through a producer thread, and store the original image data in a double-ended queue in order; Extracting multiple image frames from the double-ended queue in sequence through a conveying thread model, assigning a unique serial number to each image frame according to the order in the queue, and storing them in a temporary container; Awaken multiple processing threads in the thread pool and perform the following processing on the image frames in the temporary container in parallel: a) Convert the image frame from Mat format to YUV420 format, and use fixed-point calculation to optimize the conversion process; b) Map the converted YUV420 data directly to the data field of FFmpeg's AVFrame structure through the shared memory mechanism; The processed image frames are stored in a global sorting container according to the sequence numbers, wherein the global sorting container automatically arranges the frames in ascending order according to the sequence numbers, controls the atomicity of the multi-threaded insertion operation through a global mutex lock, and counts the number of completed frames through an atomic counter; When the count of the atomic counter reaches a preset threshold, the streaming thread obtains data from the global sorting container in sequence and streams it to the remote server; According to the difference between the current processing speed recorded by the error accumulator and the target speed, the capacity of the temporary container and the size of the thread pool are dynamically adjusted to match the data extraction speed with the processing speed.

[0015] In a preferred embodiment, the optimization process of the fixed-point calculation includes: Multiply the floating-point coefficient in the YUV conversion formula by 256 and round it to an integer. After the conversion is complete, divide the result by 256 to restore the accuracy. The specific conversion formula is: Y = (77R + 150G + 29B) / 256 U=(-38R-74G+112B) / 256 V=(157R-132G-26B) / 256 Among them, R, G, B are the three components of the Mat format image, and Y, U, V are the three components of the YUV420 format.

[0016] In a preferred embodiment, the wake-up strategy of the thread pool is: when the number of image frames in the temporary container reaches a preset batch threshold, threads equal to the batch number are woken up for parallel processing.

[0017] In a preferred embodiment, the process of dynamically adjusting the capacity of the temporary container and the size of the thread pool includes: judging according to the difference between the current processing speed recorded by the error accumulator and the target speed: When the difference exceeds the set threshold range, the number of image frames extracted by the temporary container in the next batch is adjusted, and the thread pool size is adjusted synchronously so that the number of threads matches the capacity of the temporary container; When the difference is within the set threshold range, the current temporary container capacity and thread pool size are maintained.

[0018] In a preferred embodiment, the step of converting the image frame from Mat format to YUV420 format further includes: Call the GPU parallel accelerator for fixed-point calculations and implement pixel-level parallel processing through CUDA kernel functions; Dynamically enable or disable GPU acceleration based on system power or computing load.

[0019] In a second aspect, the present application discloses a device for image conversion and streaming with enhanced real-time performance, comprising: A data acquisition module, configured to obtain raw image data from an image source through a producer thread, and store the raw image data in a double-ended queue in sequence; A batch extraction module is configured to extract multiple image frames from the double-ended queue in sequence through a conveying thread model, assign a unique serial number to each image frame according to the order in the queue, and then store them in a temporary container; The parallel processing module is configured to wake up multiple processing threads in the thread pool and process the image frames in the temporary container in parallel, including: The format conversion unit is used to convert the image frame from Mat format to YUV420 format, and the fixed-point calculation is used to optimize the conversion process; The memory mapping unit is used to map the converted YUV420 data directly to the data field of the FFmpeg AVFrame structure through the shared memory mechanism; A data sorting module is configured to store the processed image frames into a global sorting container according to the sequence number, wherein the global sorting container automatically arranges the frames in ascending order according to the sequence number, controls the atomicity of the multi-threaded insertion operation through a global mutex lock, and counts the number of completed frames through an atomic counter; The dynamic streaming module is configured to obtain data from the global sorting container in sequence and stream it to the remote server when the atomic counter reaches a preset threshold, and dynamically adjust the capacity of the temporary container and the thread pool size according to the difference between the current processing speed recorded by the error accumulator and the target speed.

[0020] In a preferred embodiment, the format conversion unit includes a fixed-point calculation module configured to: Multiply the floating-point coefficient in the YUV conversion formula by 256 and round it to an integer. After the conversion is complete, divide the result by 256 to restore the accuracy. The specific conversion formula is: Y = (77R + 150G + 29B) / 256 U=(-38R-74G+112B) / 256 V=(157R-132G-26B) / 256 Among them, R, G, B are the three components of the Mat format image, and Y, U, V are the three components of the YUV420 format.

[0021] In a preferred embodiment, the thread pool wake-up strategy of the parallel processing module is configured as follows: When the number of image frames in the temporary container reaches a preset batch threshold, threads equal to the batch number are awakened for parallel processing.

[0022] In a preferred embodiment, the dynamic streaming module includes an error accumulator configured as follows: The difference between the current processing speed and the target speed is recorded, and adjustment is triggered based on whether the difference exceeds a set threshold range. When the difference exceeds the set threshold range, the number of image frames extracted by the temporary container in the next batch is adjusted proportionally, and the thread pool size is adjusted synchronously.

[0023] In a preferred embodiment, the format conversion unit further includes a GPU acceleration module configured as follows: Call the GPU parallel accelerator for fixed-point calculations and implement pixel-level parallel processing through CUDA kernel functions; Dynamically enable or disable GPU acceleration based on system power or computing load.

[0024] In a preferred embodiment, the global sorting container is a thread-safe ordered map configured as follows: The data is stored in a key-value pair structure, where the key is the sequence number of the image frame and the value is the corresponding processing result; Automatically sort in ascending order by serial number; The atomicity of multi-threaded insert operations is controlled by mutex locks.

[0025] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention provides a method and device for image conversion and streaming with enhanced real-time performance, which solves the problems of insufficient real-time performance and low resource utilization in high-resolution video stream processing such as drones. The technical means include: Multi-threaded parallel processing architecture: Using the thread pool model, consumer threads are expanded into multi-threaded parallel tasks. By dynamically allocating image frames to multiple threads, multi-core CPU resources are fully utilized, significantly improving processing throughput.

[0026] Lightweight fixed-point calculation optimization: The YUV conversion formula of traditional floating-point calculation (such as Y=0.299R+0.587G+0.114B) is replaced by integer fixed-point calculation (such as Y=(77R+150G+29B) / 256), which reduces the time consumption of single-frame conversion while ensuring visual quality.

[0027] Shared memory mechanism: By directly passing the YUV data pointer to the AVFrame structure, deep memory copy is eliminated, reducing memory bandwidth usage and processing delay.

[0028] Dynamic load balancing: Based on the feedback mechanism of the error accumulator, the capacity of temporary containers is dynamically adjusted to balance the production and consumption rates and avoid queue overflow and thread starvation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a flow chart of storing MP4 video files locally and transmitting them using HTTP protocol in the prior art.

[0031] Figure 2 This is a flow chart of the prior art of using the FFmpeg streaming tool to stream to a remote server via the RTMP protocol, and the client pulling the stream for display via HTTP-FLV / WebRTC / HLS.

[0032] Figure 3 It is a logical diagram of double-ended queue data processing based on producer and consumer design ideas in the prior art.

[0033] Figure 4 It is a schematic diagram of the system architecture for multi-threaded parallel processing of image frames and streaming proposed by the present invention.

[0034] Figure 5 It is a schematic diagram of the present invention for directly mapping the converted YUV420 data to the data field of the AVFrame structure through a shared memory mechanism. DETAILED DESCRIPTION

[0035] In order to make the above and other features and advantages of the present invention more clear, the present invention is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] In view of the defects in the prior art, the present invention optimizes the real-time streaming of large images from the following two dimensions: First, to address the problem of single-threaded serial processing taking too long, dynamic thread pool technology is used to parallelize the processing process. By pre-caching a certain number of image frames (such as 5 frames), the multi-threaded parallel processing mechanism is triggered when the cache reaches the threshold, so that the processing time of multiple frames of images is compressed to close to the processing time of a single frame, and the CPU multi-core resources are fully utilized. Although multi-threaded synchronization will introduce additional overhead, it is negligible compared to serial frame-by-frame processing.

[0038] In the second dimension, to address the problem of high latency in single-frame processing, we can optimize processes such as memory copying and complex calculations during the processing. For example, we can use shared memory mechanisms and fixed-point calculation optimization to reduce computing load and improve real-time requirements.

[0039] In view of the above considerations, the present invention proposes a method and device for image conversion and streaming with enhanced real-time performance, and solves the above problems through the following technical solutions: Multi-threaded parallel processing: Design a dynamic thread pool to distribute image frames to multiple threads for parallel processing, and use a global sorting container to ensure the order of processing results.

[0040] Fixed-point calculation optimization: Convert floating-point coefficients into integers for calculation, and combine GPU acceleration and shared memory mechanism to significantly improve conversion efficiency.

[0041] Dynamic load balancing: Based on the error accumulator and adaptive adjustment strategy, it matches the processing speed with the data generation speed in real time, and optimizes resource utilization by dynamically adjusting the thread pool size.

[0042] Embodiment 1: This embodiment provides a method for image conversion and streaming with enhanced real-time performance, which solves the problems of high latency and high resource consumption in high-resolution image streaming through multi-threaded parallel processing and image conversion optimization.

[0043] Among them, multi-threaded parallel processing can be solved by adopting a multi-threaded model. The main idea is to design a thread pool. By processing image frames in parallel, the speed of retrieving data can be increased, thereby increasing the continuity and real-time performance of the image. The system mainly includes a thread pool, a transmission thread model, a streaming thread model, a data transmission container a, a global sorting container map, an atomic counter (to ensure the mutual exclusivity of shared variables) and an error accumulator. The principle is as follows Figure 4 shown.

[0044] The specific implementation steps are as follows: Step S1: obtaining original image data from an image source through a producer thread, and storing the original image data in a double-ended queue in sequence.

[0045] The producer thread continuously obtains raw image data from the image source (such as a drone camera) in the OpenCV Mat format.

[0046] Image data is stored in a double-ended queue in the order of acquisition. The double-ended queue adopts a thread-safe design and supports concurrent operations of producer and consumer threads.

[0047] Step S2: extract multiple image frames from the double-ended queue in sequence through the conveying thread model, and then store them in a temporary container (container a) after assigning unique serial numbers according to the order of the image frames in the queue. At this time, there are multiple image frames with serial numbers in container a.

[0048] The transport thread model sequentially extracts multiple image frames from the head of the double-ended queue (for example, extracts 5 frames each time), and assigns a globally unique increasing sequence number to each frame (for example, starting from 0 and increasing frame by frame).

[0049] The extracted image frames are stored in container a, which supports dynamic capacity adjustment (the initial capacity is N, such as N=5).

[0050] Step S3: When container a reaches the threshold capacity, multiple processing threads in the thread pool are awakened, and the image frames in container a are assigned to different threads for processing, so as to process the image frames in parallel. Specifically, the processing steps include: 1. Thread pool management and task allocation When the number of image frames in container a reaches a preset threshold (such as 5 frames), processing threads in the thread pool equal to the number of image frames are awakened (for example, 5 frames correspond to 5 threads).

[0051] The thread pool size is dynamically adjusted according to the container capacity: if the container capacity is expanded, the number of threads is increased; if the container capacity is reduced, the number of threads is reduced.

[0052] 2. Processing tasks Each thread allocates a frame of data from container a for processing. The processing task includes the following two sub-steps: a) Convert the image frame from Mat format to YUV420 format, and use fixed-point calculation to optimize the conversion process.

[0053] In the existing OpenCV library, the provided interface can achieve high-precision conversion, but the power consumption caused by excessive precision will also increase sharply, and the speed of single calculation will also decrease. Video streams are generally collected by embedded devices and processed in real time on edge devices. Most of these devices have limited computing power and have high requirements for endurance, especially in the scenario of drones, where high-performance computing devices cannot be deployed and the battery resources carried are also very limited. In addition, the real-time performance of drones after streaming is the most important, and the requirements for image quality are not particularly high.

[0054] In the traditional way, the conversion principle of converting the image frame from the Mat format to the YUV420 format is as follows, where R, G, and B are the three components of the Mat format image, and Y, U, and V are the three components of the YUV420 format.

[0055] Y = 0.299 * R + 0.587 * G + 0.114 * B U = -0.147 * R - 0.289 * G + 0.436 * B V = 0.615 * R - 0.515 * G - 0.100 * B The existing API interface in OpenCV uses the above formula to complete the conversion, which achieves high-precision conversion. This API does not have a reserved precision parameter to choose from, so it will consume more time.

[0056] In view of the above situation, this embodiment uses fixed-point calculation to optimize the calculation speed. Fixed-point calculation is a way to replace floating-point calculation. By enlarging the decimal part to an integer, it can achieve faster calculation speed and lower power consumption. For example, to express a number with a precision of 4 decimal places, you can convert the floating-point number into an integer by multiplying it by a constant (such as 10000), and then divide the result by this constant (such as 10000) after calculation to restore the original precision.

[0057] For the above formula, in order to avoid loss of precision, this embodiment multiplies the floating point number by 256 and then rounds the result to an integer. The coefficients are: Ycoef 1 = 0.299 * 256 = 76.544 ≈ 77 Ycoef 2 = 0.587 * 256 = 150.272 ≈ 150 Ycoef 3 = 0.114 * 256 = 29.184 ≈ 29 The formula for converting Mat format to YUV420P format is reconstructed as: Y = (77R + 150G + 29B) / 256 U=(-38R-74G+112B) / 256 V=(157R-132G-26B) / 256 By using fixed-point calculations, computing efficiency can be significantly improved, especially in embedded systems or where efficient image processing is required. Although there is a loss of precision, this loss is acceptable in many scenarios, especially when the conversion accuracy is not required to be very high.

[0058] In a preferred embodiment, a GPU parallel accelerator can also be configured to call the underlying GPU resources for accelerated computing. When it is detected that the processing delay is greater than the threshold (that is, when the real-time requirement needs to be improved), GPU acceleration is automatically started, and when the real-time requirement is met, the accelerated computing is turned off and switched back to CPU mode to reduce power consumption.

[0059] b) Directly map YUV420 data to the data field of FFmpeg's AVFrame structure through the shared memory mechanism to avoid deep copy (memcpy) operations.

[0060] The previous approach was to use memcpy to perform a deep copy of memory to avoid pointer management and complex variable lifecycle issues, but this would introduce a large memory overhead. However, the data is read-only and not write-able, so there is no need to repeatedly create and destroy it.

[0061] In this embodiment, the data address of YUV420P is directly transferred to the data element of AVFrame through the shared memory mechanism, thereby realizing the reuse of Mat frame data. Figure 5 As shown in the figure. By improving this method, the speed of streaming can be greatly improved to ensure real-time performance.

[0062] 3. Processing result storage After the processing is completed, the result and the serial number are formed into a key-value pair and inserted into the global sort container (map) through a global mutex lock, so that they are automatically sorted in ascending order according to the serial number to ensure that the subsequent streaming is carried out in the order in which the images are generated.

[0063] Step S4: store the processed image frames into a global sorting container according to the sequence number, and count the number of completed frames through an atomic counter.

[0064] The global sort container stores all processed image frames, the key is the sequence number, and the value is the processing result (AVFrame pointer).

[0065] The atomic counter counts the number of completed frames, and the counter is incremented after each inserted frame.

[0066] Step S5: When the count of the atomic counter reaches a preset threshold, the streaming thread obtains data from the global sorting container in sequence and streams it to the remote server.

[0067] The streaming thread continuously monitors the value of the atomic counter. When the count reaches the preset threshold (such as 5 frames), it performs the following operations: Get AVFrame data from the global sort container (map) in ascending order (such as serial numbers 0 to 4).

[0068] Push data to the remote server through FFmpeg's streaming interface (such as av_interleaved_write_frame), supporting protocols such as RTMP and HTTP-FLV.

[0069] After the push is completed, the global sort container (map) is cleared and the atomic counter is reset to 0.

[0070] Step S6: The error accumulator dynamically adjusts the capacity of container a and the size of the thread pool according to the speed of this processing, thereby ensuring that the speed of storing and retrieving from the queue matches.

[0071] The error accumulator records the difference between the processing speed (such as the number of frames per second) and the target speed in real time (Δ = target speed - actual speed).

[0072] If the Δ value exceeds the set threshold range (e.g. |Δ|>2 frames / second), trigger: 1. Adjust the capacity of the current container a: For example, the capacity of the current container a is adjusted proportionally (such as expanding the capacity by Δ×k, where k is an empirical coefficient).

[0073] If the current capacity is 5 frames, Δ=4 frames / second, k=0.5, then the new capacity is: 5+0.5×4=7 frames.

[0074] 2. Synchronously adjust the thread pool size: Dynamically increase or decrease the number of threads based on the new capacity to ensure that all image frames in the container can be processed in parallel.

[0075] For example, if the new capacity is 10 frames, the thread pool activates 10 threads to ensure that all frames can be processed in parallel.

[0076] If the Δ value is within the set threshold range (e.g. |Δ|≤2 frames / second), then: Maintain the current container capacity and thread pool size to avoid system overhead caused by frequent adjustments.

[0077] Embodiment 2: Based on the same design concept, this embodiment also provides a device for image conversion and streaming with enhanced real-time performance, which specifically includes the following modules: A data acquisition module, configured to obtain raw image data from an image source through a producer thread, and store the raw image data in a double-ended queue in sequence; A batch extraction module is configured to extract multiple image frames from the double-ended queue in sequence through a conveying thread model, assign a unique serial number to each image frame according to the order in the queue, and then store them in a temporary container; The parallel processing module is configured to wake up multiple processing threads in the thread pool and process the image frames in the temporary container in parallel, including: The format conversion unit is used to convert the image frame from Mat format to YUV420 format, and the fixed-point calculation is used to optimize the conversion process; The memory mapping unit is used to map the converted YUV420 data directly to the data field of the FFmpeg AVFrame structure through the shared memory mechanism; A data sorting module is configured to store the processed image frames into a global sorting container according to the sequence number, wherein the global sorting container automatically arranges the frames in ascending order according to the sequence number, controls the atomicity of the multi-threaded insertion operation through a global mutex lock, and counts the number of completed frames through an atomic counter; The dynamic streaming module is configured to obtain data from the global sorting container in sequence and stream it to the remote server when the atomic counter reaches a preset threshold, and dynamically adjust the capacity of the temporary container and the thread pool size according to the difference between the current processing speed recorded by the error accumulator and the target speed.

[0078] In a preferred embodiment, the format conversion unit includes a fixed-point calculation module configured to: Multiply the floating-point coefficient in the YUV conversion formula by 256 and round it to an integer. After the conversion is complete, divide the result by 256 to restore the accuracy. The specific conversion formula is: Y = (77R + 150G + 29B) / 256 U=(-38R-74G+112B) / 256 V=(157R-132G-26B) / 256 Among them, R, G, B are the three components of the Mat format image, and Y, U, V are the three components of the YUV420 format.

[0079] In a preferred embodiment, the dynamic streaming module includes an error accumulator configured as follows: The difference between the current processing speed and the target speed is recorded, and adjustment is triggered based on whether the difference exceeds the set threshold range. When the difference exceeds the set threshold range, the number of image frames extracted by the next batch of temporary containers is adjusted proportionally, and the thread pool size is adjusted synchronously.

[0080] In a preferred embodiment, the thread pool wake-up strategy of the parallel processing module is configured as follows: When the number of image frames in the temporary container reaches a preset batch threshold, threads equal to the batch number are awakened for parallel processing.

[0081] In a preferred embodiment, the format conversion unit further includes a GPU acceleration module configured as follows: Call the GPU parallel accelerator for fixed-point calculations and implement pixel-level parallel processing through CUDA kernel functions; Dynamically enable or disable GPU acceleration based on system power or computing load.

[0082] In a preferred embodiment, the global sorting container is a thread-safe ordered map configured as follows: The data is stored in a key-value pair structure, where the key is the sequence number of the image frame and the value is the corresponding processing result; Automatically sort in ascending order by serial number; The atomicity of multi-threaded insert operations is controlled by mutex locks.

[0083] It can be understood that the modules recorded in the device for enhancing real-time image conversion and streaming are similar to Figure 4 Therefore, the steps, features and beneficial effects described above for the method for image conversion and streaming based on enhancing real-time performance are also applicable to the device for image conversion and streaming based on enhancing real-time performance and the modules contained therein, and will not be repeated here.

[0084] In summary, the present invention achieves efficient conversion and low-latency streaming of high-resolution video streams through the combination of multi-threaded synchronization, fixed-point computing optimization, shallow copy memory reuse and dynamic adjustment strategies, providing a reliable technical solution for real-time video transmission scenarios such as drones and edge computing devices.

[0085] The specific embodiments of the present invention are described in detail above, but they are only examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions made to the present invention are also within the scope of the present invention. Therefore, the equalization changes and modifications made without departing from the spirit and scope of the present invention should be included in the scope of the present invention.

Claims

1. A method for image conversion and streaming with enhanced real-time performance, characterized in that: The steps include: Acquire original image data from an image source through a producer thread, and store the original image data in a double-ended queue in order; Extracting multiple image frames from the double-ended queue in sequence through a conveying thread model, assigning a unique serial number to each image frame according to the order in the queue, and storing them in a temporary container; Awaken multiple processing threads in the thread pool and perform the following processing on the image frames in the temporary container in parallel: a) Convert the image frame from Mat format to YUV420 format, and use fixed-point calculation to optimize the conversion process; b) Map the converted YUV420 data directly to the data field of FFmpeg's AVFrame structure through the shared memory mechanism; The processed image frames are stored in a global sorting container according to the sequence numbers, wherein the global sorting container automatically arranges the frames in ascending order according to the sequence numbers, controls the atomicity of the multi-threaded insertion operation through a global mutex lock, and counts the number of completed frames through an atomic counter; When the count of the atomic counter reaches a preset threshold, the streaming thread obtains data from the global sorting container in sequence and streams it to the remote server; According to the difference between the current processing speed recorded by the error accumulator and the target speed, the capacity of the temporary container and the size of the thread pool are dynamically adjusted to match the data extraction speed with the processing speed.

2. The method for image conversion and streaming with enhanced real-time performance according to claim 1, characterized in that: The optimization process of the fixed-point calculation includes: Multiply the floating-point coefficient in the YUV conversion formula by 256 and round it to an integer. After the conversion is complete, divide the result by 256 to restore the accuracy. The specific conversion formula is: Y = (77R + 150G + 29B) / 256 U=(-38R-74G+112B) / 256 V=(157R-132G-26B) / 256 Among them, R, G, B are the three components of the Mat format image, and Y, U, V are the three components of the YUV420 format.

3. The method for image conversion and streaming with enhanced real-time performance according to claim 1, characterized in that: The wake-up strategy of the thread pool is: when the number of image frames in the temporary container reaches a preset batch threshold, threads equal to the batch number are woken up for parallel processing.

4. The method for image conversion and streaming with enhanced real-time performance according to claim 1, characterized in that: The process of dynamically adjusting the capacity of the temporary container and the size of the thread pool includes: judging according to the difference between the current processing speed recorded by the error accumulator and the target speed: When the difference exceeds the set threshold range, the number of image frames extracted by the temporary container in the next batch is adjusted, and the thread pool size is adjusted synchronously so that the number of threads matches the capacity of the temporary container; When the difference is within the set threshold range, the current temporary container capacity and thread pool size are maintained.

5. The method for image conversion and streaming with enhanced real-time performance according to claim 1, characterized in that: The step of converting the image frame from Mat format to YUV420 format also includes: Call the GPU parallel accelerator for fixed-point calculations and implement pixel-level parallel processing through CUDA kernel functions; Dynamically enable or disable GPU acceleration based on system power or computing load.

6. A device for image conversion and streaming with enhanced real-time performance, characterized in that: include: A data acquisition module, configured to obtain raw image data from an image source through a producer thread, and store the raw image data in a double-ended queue in sequence; A batch extraction module is configured to extract multiple image frames from the double-ended queue in sequence through a conveying thread model, assign a unique serial number to each image frame according to the order in the queue, and then store them in a temporary container; The parallel processing module is configured to wake up multiple processing threads in the thread pool and process the image frames in the temporary container in parallel, including: The format conversion unit is used to convert the image frame from Mat format to YUV420 format, and the fixed-point calculation is used to optimize the conversion process; The memory mapping unit is used to map the converted YUV420 data directly to the data field of the FFmpeg AVFrame structure through the shared memory mechanism; A data sorting module is configured to store the processed image frames into a global sorting container according to the sequence number, wherein the global sorting container automatically arranges the frames in ascending order according to the sequence number, controls the atomicity of the multi-threaded insertion operation through a global mutex lock, and counts the number of completed frames through an atomic counter; The dynamic streaming module is configured to obtain data from the global sorting container in sequence and stream it to the remote server when the atomic counter reaches a preset threshold, and dynamically adjust the capacity of the temporary container and the thread pool size according to the difference between the current processing speed recorded by the error accumulator and the target speed.

7. The device for image conversion and streaming with enhanced real-time performance according to claim 6, characterized in that: The format conversion unit includes a fixed-point calculation module configured as follows: Multiply the floating-point coefficient in the YUV conversion formula by 256 and round it to an integer. After the conversion is complete, divide the result by 256 to restore the accuracy. The specific conversion formula is: Y = (77R + 150G + 29B) / 256 U=(-38R-74G+112B) / 256 V=(157R-132G-26B) / 256 Among them, R, G, B are the three components of the Mat format image, and Y, U, V are the three components of the YUV420 format.

8. The device for image conversion and streaming with enhanced real-time performance according to claim 6, characterized in that: The thread pool wake-up strategy of the parallel processing module is configured as follows: When the number of image frames in the temporary container reaches a preset batch threshold, threads equal to the batch number are awakened for parallel processing.

9. The device for image conversion and streaming with enhanced real-time performance according to claim 6, characterized in that: The dynamic streaming module includes an error accumulator configured as follows: The difference between the current processing speed and the target speed is recorded, and adjustment is triggered based on whether the difference exceeds a set threshold range. When the difference exceeds the set threshold range, the number of image frames extracted by the temporary container in the next batch is adjusted proportionally, and the thread pool size is adjusted synchronously.

10. The device for image conversion and streaming with enhanced real-time performance according to claim 6, characterized in that: The format conversion unit also includes a GPU acceleration module configured as follows: Call the GPU parallel accelerator for fixed-point calculations and implement pixel-level parallel processing through CUDA kernel functions; Dynamically enable or disable GPU acceleration based on system power or computing load.

11. The device for image conversion and streaming with enhanced real-time performance according to claim 6, characterized in that: The global sorting container is a thread-safe ordered mapping table, configured as follows: The data is stored in a key-value pair structure, where the key is the sequence number of the image frame and the value is the corresponding processing result; Automatically sort in ascending order by serial number; The atomicity of multi-threaded insert operations is controlled by mutex locks.

Images