Multi-thread video processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention provides a multi-thread video processing method and device, electronic equipment and a computer readable storage medium. Determining a matched target scaling algorithm from a plurality of preset scaling algorithms according to the equipment information of the playing equipment; according to a preset segmentation resolution, segmenting original data frames included in the obtained video decoding data to obtain a plurality of segmentation blocks corresponding to each original data frame; according to the number of the segmentation blocks, a target scaling algorithm and a preset thread number in a preset thread pool, generating a plurality of execution data packets corresponding to the original data frame; and the execution data packets are distributed to threads in a preset thread pool for respective execution, and a scaled target data frame is obtained. According to the method, the multi-thread scaling processing efficiency can be improved, and frame loss and picture jamming caused by processing a large amount of data are avoided through processing of the matched target scaling algorithm.

Description

Technical Field

[0001] This application relates to the field of video processing technology, specifically to a multi-threaded video processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Video scaling is a crucial part of video playback, and efficient video scaling provides a better user experience. In related technologies, hardware video acceleration is often achieved through the VAAPI (Video Acceleration API) video decoding protocol, with a dedicated hardware VPU (Vision Processing Unit) for processing video data. In some scenarios where a hardware VPU is not deployed, the CPU is typically used to perform video processing tasks, including video scaling. Summary of the Invention

[0003] The present invention aims to provide a multi-threaded video processing method, apparatus, electronic device, and computer-readable storage medium, which at least solves the problems in the prior art.

[0004] To solve the above-mentioned technical problems, the present invention is implemented as follows: In a first aspect, embodiments of the present invention provide a multi-threaded video processing method, the method comprising: Based on the device information of the playback device, a matching target scaling algorithm is determined from multiple preset scaling algorithms; According to the preset segmentation resolution, each original data frame in the video decoding data is segmented to obtain multiple segmentation blocks corresponding to each original data frame; Based on the number of segmented blocks corresponding to each original data frame, the target scaling algorithm, and the number of threads in the preset thread pool, multiple execution data packets corresponding to the original data frames are generated respectively. The multiple execution data packets are distributed to preset threads in the preset thread pool for execution, resulting in scaled target data frames.

[0005] Secondly, embodiments of the present invention also provide a multi-threaded video processing apparatus, the apparatus comprising: The acquisition module is used to acquire video decoding data, which includes multiple raw data frames; The algorithm selection module is used to determine the target scaling algorithm from multiple preset scaling algorithms based on the device information of the playback device. The segmentation module is used to segment each original data frame in the video decoding data according to a preset segmentation resolution, so as to obtain multiple segmentation blocks corresponding to each original data frame; The generation module is used to generate multiple execution data packets corresponding to the original data frame based on the number of segmented blocks corresponding to each original data frame, the target scaling algorithm, and the number of threads in the preset thread pool. The execution module is used to distribute the multiple execution data packets to preset threads in the preset thread pool for execution, thereby obtaining scaled target data frames.

[0006] Thirdly, embodiments of the present invention also provide an electronic device, which includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0007] Fourthly, embodiments of the present invention also provide a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0008] In summary, in the multi-threaded video processing method of this embodiment, a matching target scaling algorithm is determined from multiple preset scaling algorithms based on the device information of the playback device. Each original data frame in the video decoding data is segmented based on a preset segmentation resolution to obtain multiple segmentation blocks corresponding to each original data frame. Based on a preset number of threads in a preset thread pool, the multiple segmentation blocks are grouped, and the grouping of segmentation blocks is combined with the target scaling algorithm to form an execution data packet. Multiple execution data packets are distributed to preset threads in the preset thread pool for execution, resulting in a scaled target data frame. The target scaling algorithm is selected based on the processor utilization of the playback device. The complexity of the target scaling algorithm is positively correlated with the processor utilization of the playback device. Using a matching scaling algorithm reduces processor utilization, improves the processing efficiency of the playback device's processor, and utilizes multi-threaded concurrent processing to improve image data processing efficiency. Scaling the original data frame using a matching target scaling algorithm avoids the loss of frames and screen stuttering caused by processing large amounts of data with an incompatible algorithm. Therefore, multi-threaded image software processing can replace the hardware scaling unit function to achieve frame-by-frame scaling of the raw data frames of video decoding data. Attached Figure Description

[0009] Figure 1 This is a flowchart of the steps of a multi-threaded video processing method provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the overall steps of a multi-threaded video processing method provided in an embodiment of the present invention. Figure 3This is a block diagram of a multi-threaded video processing device provided in an embodiment of the present invention; Figure 4 This is a block diagram of another multi-threaded video processing device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0010] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0011] Multithreaded image processing improves efficiency by leveraging the parallel computing power of multi-core processors. The basic idea is to divide the original video image into multiple sub-blocks, each with independent data. Each thread processes one sub-block, and the results of all sub-blocks are then combined to form the final image. However, when playing 4K high-bitrate videos, especially those encoded with H.264 / HEVC, some video playback devices lack a hardware VPU and rely solely on the CPU for software decoding, resulting in CPU utilization consistently exceeding 90% or even 100%.

[0012] Video playback devices are mainly categorized into: operational playback devices, embedded playback devices, ecosystem-supported playback devices, and command-driven playback devices. Operational playback devices are independent, operable multimedia playback devices, typically equipped with a complete operating system and user interface. Embedded playback devices are dedicated playback systems integrated into other devices, application-centric, with customizable hardware and software to meet specific application needs. Ecosystem-supported playback devices rely on a specific ecosystem and support multi-device collaboration. The VPU (Vision Processing Unit) is primarily responsible for core tasks such as video encoding / decoding and real-time stream processing. The VPP (Video Post Processing) unit typically refers to the image processing module within the VPU, responsible for image enhancement, special effects addition, etc.

[0013] Multi-threaded graphics scaling processing is added to the VPP module in the VPU driver, enabling concurrent multi-threaded video scaling. This is particularly useful on VPU hardware platforms lacking hardware scaling support, as software-based graphics scaling can enhance video processing capabilities.

[0014] Figure 1 This is a flowchart of the steps of a multi-threaded video processing method provided in an embodiment of the present invention, as follows: Figure 1 As shown, the method may include: Step 101: Based on the device information of the playback device, determine the matching target scaling algorithm from multiple preset scaling algorithms.

[0015] It should be noted that the hardware decoding unit is a dedicated module specifically designed to accelerate video decoding tasks. It decompresses the input compressed video stream and outputs video decoded data. This video decoded data includes multiple raw data frames and originates from the output data of the hardware decoding unit. The video decoded data contains raw data frames and metadata. Raw data frames are the decoded video frame data, such as pixel data (YUV, RGB, etc.) in an AVFrame (Audio / video Frame, a container for audio or video frames). Metadata describes additional information about the decoded data frames in the video decoded data, such as timestamps, pixel formats, and resolutions. Video decoding is the process of restoring encoded video data, such as H.264 and H.265, back to the original data. The format of video decoded data involves two encoding standards: H.264 and H.265. The video decoding format usually corresponds to the video encoding format; for example, H.264 encoded video requires a decoder that supports H.264 for decoding. YUV is a color encoding method used for displaying both black-and-white and color video. It separates luminance (Y) and chrominance (UV) information, reducing the data volume while maintaining visual quality by lowering the chrominance sampling rate, such as YUV420. Encoding is the process of converting YUV data into a specific video encoding format, while decoding is the process of converting encoded video data back into YUV format. Decoding encoded video data into YUV data allows for scaling, enabling the scaling of YUV data from one resolution to another.

[0016] In this embodiment, the device information of the playback device may include: playback device platform information, playback device processor load information, and the display types supported by the playback device. Specifically, if the hardware VPU lacks a video scaling unit, a target scaling algorithm can be matched from multiple preset scaling algorithms included in the scaling algorithm library through algorithm features or lookup table structure, based on the playback device's processor load information and the display types supported by the playback device. The video scaling algorithms in the FFmpeg (Fast Forward Moving Picture Experts Group) codec library may include: nearest neighbor interpolation algorithm, bilinear interpolation algorithm, bicubic interpolation algorithm, etc. Key parameters of the scaling algorithm (e.g., interpolation type, block size, quality level, etc.) can be quantified into indexable numerical features. Different algorithm levels are set for the multiple scaling algorithms based on the key parameters of the scaling algorithm. For example, using the interpolation type as the key parameter, the bicubic interpolation algorithm is set to 3 (high level), the bilinear interpolation algorithm to 2 (medium level), and the nearest neighbor interpolation algorithm to 1 (low level). The target scaling algorithm of the appropriate algorithm level is selected considering the processor utilization of the playback device. For example, under high processor load, an algorithm with high video scaling efficiency and moderate image display distortion is selected. Under low processor load, an algorithm with moderate video scaling efficiency and low image display distortion is selected.

[0017] For example, different ranges of processor utilization of the playback device can be associated with the algorithm levels of different scaling algorithms. Based on the GetSystemTimes function, the overall CPU (Central Processing Unit) utilization is calculated by calculating the difference between the system idle time, kernel time, and user time between two samples, combined with the formula (kernel time + user time - idle time) / (kernel time + user time). For multi-core CPUs, the average utilization of each core needs to be calculated. For example, the distortion rate can be used to set the effect level of each preset scaling algorithm. A scaling algorithm with a distortion rate greater than 0.3 can be set as a low-level algorithm; a scaling algorithm with a distortion rate greater than 0.1 and less than or equal to 0.3 can be set as a medium-level algorithm; and a scaling algorithm with a distortion rate less than 0.1 can be set as a high-level algorithm. Since the algorithm level is set based on the distortion rate of the display effect, there may be multiple preset scaling algorithms set to the same level. Therefore, we can further consider the characteristic parameters of the scaling algorithm, such as complexity or computational load, and set the calling priority within the same level for multiple scaling algorithms of the same level. For example, scaling algorithms with a display distortion rate of less than 0.1 include bicubic interpolation algorithm and Lanczos interpolation algorithm. Since the computational complexity of the Lanczos interpolation algorithm is greater than that of the bicubic interpolation algorithm, under the same conditions, the bicubic interpolation algorithm with lower computational complexity should be selected first to ensure CPU processing efficiency and improve response speed.

[0018] Since CPU utilization is inversely proportional to image processing capability, a lookup table can be used to generate a mapping between CPU utilization ranges and different preset scaling algorithms, and this mapping can be stored in an in-memory data structure, file, or database. For example, CPU utilization greater than 66% corresponds to a low-level algorithm, CPU utilization greater than 33% and less than 66% corresponds to a medium-level algorithm, and CPU utilization less than 33% corresponds to a high-level algorithm. This mapping is then used to generate a lookup table and stored in a database linked list.

[0019] In this embodiment, the current CPU utilization is obtained through system calls such as `get_cpu_usage()`, and then the matching algorithm level is queried from a lookup table. When the current CPU utilization is greater than 66%, a low-level algorithm is selected; when the current CPU utilization is greater than 33% but less than 66%, a medium-level algorithm is selected; and when the current CPU utilization is less than 33%, a high-level algorithm is selected. Since multiple preset scaling algorithms may exist at the same level based on display effect settings, the characteristic parameters of the scaling algorithms, such as complexity or computational cost, can be considered to select a more suitable target scaling algorithm from multiple scaling algorithms at the same level. Furthermore, when playing on multiple displays, a suitable scaling algorithm can be selected from the scaling algorithm library as the target scaling algorithm for each display window based on the window size of each display and the resolution of the original data frames in the video decoding data.

[0020] Therefore, since the scaling algorithms mentioned above have different scaling efficiencies and display effects, the corresponding scaling algorithm is selected based on the current utilization rate of the processor on the playback device to ensure smooth playback of various videos and improve the efficiency of frame-by-frame processing.

[0021] Step 102: According to the preset segmentation resolution, each original data frame in the video decoding data is segmented to obtain multiple segmentation blocks corresponding to each original data frame.

[0022] It's important to note that resolution is a metric for measuring the amount of information stored in an image. In image segmentation, the resolution of the original data frame directly affects the model's ability to recognize image details. Segmentation resolution in image segmentation processing refers to the total number of pixels in each segmentation block, reflecting the image's sharpness and level of detail. Higher segmentation resolution results in fewer segmentation blocks in the original data frame, leading to a sharper target data frame after scaling. Specifically, segmentation resolution is determined by the product of the number of pixels in the horizontal and vertical directions; for example, 1920×1080 represents 1920 pixels horizontally and 1080 pixels vertically.

[0023] In this embodiment, the preset segmentation resolution can be set according to the playback device platform information and the pixel alignment method of the selected algorithm. The preset segmentation resolution can be set based on the maximum number of pixels that a single thread of the CPU on the playback device can process in one screen refresh cycle when running the scaling algorithm. The preset segmentation resolution is 800×600. When the resolution of the original data frame is 800×600, no segmentation is required. When the resolution of the original data frame is 1920×1080, segmentation is performed using a fixed-size segmentation method and an edge block processing method. Horizontally, every 800 pixels results in two complete blocks and one remaining block (320 pixels). Vertically, every 600 pixels results in one complete block and one remaining block (480 pixels). The two remaining blocks are then processed separately, and padding can be used to obtain two complete blocks, resulting in a total of six segmented blocks. Each 1920×1080 original data frame can be divided into six 800×600 segmented blocks, and these segmented blocks can be re-encoded into independent video files for storage. By setting appropriate encoding parameters and segmentation strategies, the original data frame can be divided into six 800×600 sub-video files, thus ensuring the continuity of the video. Each segment can be indicated using a custom data descriptor variable, which can store feature parameters in the form of a NumPy array or an OpenCV Mat. The descriptor object is initialized first, and then the data descriptor is extracted using the detectAndCompute method for easy access by the computer.

[0024] For example, if the original data frame of the video decoding data has a resolution of 1920×1080, and the maximum number of pixels that a single thread of the CPU on the playback device can process in one screen refresh cycle while running the video scaling algorithm is 480,000, then the preset segmentation resolution can be set to 800×600 based on the aspect ratio of the original data frame. Then, FFmpeg's filter system can be used to segment the 1920×1080 original data frame into a 3×2 800×600 grid layout to obtain 6 segment blocks. The segment blocks use the base address of the original frame data, the starting coordinates of the segmentation, and the three parameters of the segment block (length, width, and height) stored in a temporary storage area in memory. These blocks can also exist in the codec, filter chain, or memory management module to temporarily store data frames, ensuring the smoothness and correctness of video processing.

[0025] Step 103: Based on the number of segmented blocks corresponding to each original data frame, the target scaling algorithm, and the number of threads in the preset thread pool, generate multiple execution data packets corresponding to the original data frames.

[0026] It's important to note that the default thread pool is pre-created and maintains a set of threads used to execute submitted tasks. Threads in the pool are not destroyed after completing a task but instead await task allocation. The thread pool limits the maximum number of threads to prevent excessive thread contention. Each segment can be indicated using OpenCV feature descriptors (HOG, SURF, and ORB) or custom data descriptor variables for easy access by the computer.

[0027] In this embodiment, the `getPoolSize()` method can return the total number of threads in the preset thread pool, including idle threads and threads currently executing tasks. The maximum number of threads in the thread pool is typically set to the number of CPU cores plus one. The number of threads in the preset thread pool is obtained, and the number of blocks corresponding to each original data frame is evenly divided according to the number of threads by reading the number of blocks in the temporary cache, and then packaged into task data packets. Each task data packet is an independent task package, encapsulating independent execution units such as operation parameters and callback functions, and typically includes task logic and required data. It can be processed in parallel by a multi-threaded system, and the task data packet includes combinations of multiple segments that can be processed independently.

[0028] For example, with 8 threads and 40 segments, the segments can be evenly distributed, combining 5 segments together to generate 8 task data packets. For 9 threads and 50 segments, a dynamic distribution method can be used, allocating 5 to 6 segments to each thread. Specifically, 4 threads are allocated 5 segments each, and 5 threads are allocated 6 segments each, thus covering all 50 segments. The segments allocated to each thread are then combined and processed. Multiple segment data can be combined using a structure array. Each segment can be defined as a structure containing a base address, starting coordinates, and width and height parameters. These structures are stored in contiguous memory, and their members are accessed and modified via array indexes. All segments are then processed using a loop.

[0029] Then, the generated task data packets are combined with the target scaling algorithm to obtain schedulable execution data packets. Task encapsulation is performed using structures or classes, encapsulating the operations of the task data packets and the target scaling algorithm within a unified structure to form independent execution data packets. Each execution data packet contains complete input parameters, algorithm configuration, and output buffers, enabling direct execution by threads without additional data preparation. Furthermore, schedulable execution descriptors are generated to provide execution for processor threads. The task data packets are bound to the target scaling algorithm via function pointers or virtual functions, supporting dynamic selection of various interpolation methods. Typically, each schedulable execution data packet can be indicated using a custom schedulable execution descriptor for easy computer invocation. The execution data packets provide standardized execution interfaces for the scheduling system, including a task submission interface that distributes tasks via thread-safe queues, a progress monitoring interface that provides task execution status feedback, and a result collection interface that uses thread-safe containers to store processing results.

[0030] Therefore, multiple segments and algorithms can be combined and processed according to the number of threads in the thread pool to obtain schedulable execution data packets. The thread pool adds the execution data packets to a queue, and the threads in the thread pool execute them on demand, realizing orderly task scheduling. Existing threads can be reused to reduce the overhead of thread creation and destruction.

[0031] Step 104: Distribute the multiple execution data packets to preset threads in the preset thread pool for execution, and obtain scaled target data frames.

[0032] It's important to note that different business types use different thread pools. A thread number refers to the logical number of a thread in the thread pool, typically assigned by the thread pool manager, used internally to identify threads for task scheduling or status monitoring. Image scaling processes recalculate and lay out pixel data to change the size of an image without significantly degrading quality. Scaling can be divided into shrinking and enlarging. During shrinking, some pixels are ignored; while during enlargement, additional pixels need to be created.

[0033] In this embodiment, execution data packets can be assigned to preset threads in a preset thread pool based on pre-bound thread numbers to perform scaling processing according to the requirements of the target scaling algorithm. For example, thread numbers could be thread-1, thread-2, and thread-3. Thread data binding and thread pool task encapsulation can be implemented using Java's ThreadLocal mechanism or Python's threading.local mechanism, effectively binding thread numbers to execution data packets. By creating an independent data structure for each thread to associate the execution data packet with the executing thread, each worker thread can maintain its own task count, execution time, and other statistical information. For example, by creating independent data structures, three execution data packets, data.1 and thread-1, data.2 and thread-2, and data.3 and thread-3, are bound respectively. Mutexes and condition variables are used to ensure data safety in a multi-threaded environment. Execution data packets are passed through task parameters, thread-local storage is used to save the execution data packets bound to the thread, and callback functions are used to process task results. Multiple execution data packets corresponding to each original data frame are first submitted to the execution task queue. Then, the thread scheduler built into the playback software obtains idle available threads from the thread pool and execution descriptors from the execution queue.

[0034] Efficient task scheduling is achieved by allocating tasks to pre-specified threads on demand to perform scaling algorithm calculations. This process can be repeated until all threads in the thread pool are exhausted; if execution descriptors have not been fully dispatched, the process blocks and waits for available threads to dispatch. An appropriate task queue capacity can be set based on the system load of the playback device, avoiding the storage of large amounts of data or scarce resources in ThreadLocal.

[0035] For example, preset threads in a preset thread pool can be executed individually to obtain scaled target data frames. The execution results of each subtask, such as data.11, data.22, and data.33, are obtained through the thread pool or task framework (e.g., Fork, Join), representing the scaled target data corresponding to data.1, data.2, and data.3. Then, the execution results of each subtask are integrated into the final data result (e.g., scaling the original 1920×1080 data frame to an 800×600 video frame) according to logical rules (e.g., splicing) through region similarity merging, thus restoring the target data frame. The target data is a video frame with a native resolution that conforms to the window size of the display device, ensuring clear presentation of video content, adapting to hardware performance, and improving the user experience.

[0036] The execution descriptors corresponding to the segmented block groups can be distributed to threads in the thread pool for processing. The execution descriptors indicate the schedulable execution data packets of the segmented block groups in the original data frame. The execution results can be aggregated into a scaled target data frame through region similarity merging. First, merging can be performed based on the image features of the regions occupied by adjacent segmented blocks. For example, statistical features such as the grayscale mean and standard deviation are calculated for the processed features of each segmented block. Then, a preset merging threshold is used; merging is performed when the feature difference between the regions occupied by adjacent segmented blocks is less than the preset threshold, for example, merging upwards from the smallest segmented block until no adjacent segmented blocks satisfy the condition. The efficiency of image scaling processing can be improved by splitting the parallel processing task into multiple subtasks for concurrent execution.

[0037] In summary, in the multi-threaded image processing method of this embodiment, the target scaling algorithm is selected according to the device parameters of the playback device. The calculation of the segmented blocks of the original data frame is performed by the target scaling algorithm, which can improve the processing efficiency of the playback device processor. The scaling is performed by available threads in the thread pool created in the playback device, which can improve the efficiency of multi-threaded image scaling processing. The processing of the original data frame by the target scaling algorithm can avoid frame loss caused by processing a large amount of data, thereby effectively avoiding video stuttering on the display. Furthermore, the scaling processing of video decoding data can be achieved by replacing the hardware scaling unit function with software processing, thereby saving hardware costs.

[0038] Figure 2 This is a flowchart illustrating the steps of another multi-threaded video processing method provided in an embodiment of the present invention. Figure 2 As shown, this multi-threaded video processing method may include: Step 201: Obtain the current processor usage rate of the playback device.

[0039] It should be noted that processor utilization reflects the proportion of time actually spent executing tasks. CPU utilization refers to the percentage of time the CPU spends executing tasks. For example, if a single-core CPU occupies 0.5 seconds out of every second it runs, the utilization rate is 50%. High utilization usually indicates that the CPU is busy, which can lead to slower program response.

[0040] For example, the number of processor cores of a playback device can be obtained through the system API and file parsing methods. The number of available cores can be directly obtained using the `Runtime.availableProcessors()` API. In the Android system, CPU utilization can be obtained by reading the ` / proc / stat` file, which dynamically records information about all CPU activity. The method to calculate the current CPU utilization of the playback device is to calculate the changes in various CPU activity information over short intervals. The maximum utilization rate for a single-core CPU is 100%, and the maximum utilization rate for a multi-core CPU is the number of cores multiplied by 100%. For example, the maximum utilization rate for a 4-core CPU is 400%.

[0041] Current utilization rate refers to the CPU utilization rate at the current point in time. The utilization rate can be uneven or exceed 100%.

[0042] Step 202: Based on the current processor utilization rate and the preset correspondence, select the scaling algorithm corresponding to the current processor utilization rate as the target scaling algorithm. The preset correspondence includes the correspondence between the processor utilization rate and the preset scaling algorithm.

[0043] It should be noted that the preset mapping relationship includes the mapping relationship between processor utilization and preset scaling algorithm, or the mapping relationship between processor load rate and preset scaling algorithm, and is stored in the local file system or database in JSON or XML format.

[0044] In this embodiment, a scaling algorithm of the corresponding level can be selected from a preset correspondence based on the current processor utilization rate as the adapted video scaling algorithm, and this adapted video scaling algorithm can be used as the target scaling algorithm. The preset correspondence refers to the one-to-one correspondence between processor utilization rate and preset different scaling algorithms.

[0045] For example, video decoding data is acquired, which includes multiple raw decoded data frames. When using the FFmpeg codec library, the video scaling algorithms include: nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, and Lanczos interpolation. When scaling the FFmpeg-corresponding video decoding data, a suitable preset scaling algorithm can be selected based on actual needs. The display effect after image processing can be based on multiple preset scaling algorithms, such as the corresponding relationships between PSNR (Peak Signal to Noise Ratio), SSIM (Structural Similarity Index), and key parameter values. Preset scaling algorithms with PSNR less than 30dB (significant image quality loss) are set to low-level algorithms; those with PSNR greater than 30dB but less than 40dB (minor image quality loss) are set to medium-level algorithms; and those with PSNR greater than 40dB (minimum image quality loss) are set to high-level algorithms. Since CPU utilization is inversely proportional to image processing capability without other hardware acceleration, this approach is recommended. In a single-core CPU scenario, more complex preset scaling algorithms result in higher computational load and better scaling effects, but also slower processing speeds. For example, the nearest neighbor interpolation algorithm has the lowest computational complexity, with CPU utilization below 20% during runtime; the bilinear interpolation algorithm has relatively low complexity, with CPU utilization between 20% and 30% during runtime; the bicubic interpolation algorithm has higher complexity, with CPU utilization reaching 30% to 50% during runtime; and the Lanczos interpolation algorithm has the highest computational complexity, with CPU utilization reaching 70% during runtime. Therefore, to avoid frame drops and stuttering caused by delays in image scaling by the playback device's processor, a CPU utilization of less than 20% can be associated with the Lanczos interpolation algorithm; a CPU utilization of 20% or more but less than 40% can be associated with the bicubic interpolation scaling algorithm; a CPU utilization of 40% or more but less than 70% can be associated with the bilinear interpolation scaling algorithm; and a CPU utilization of 70% or more can be associated with the nearest neighbor interpolation scaling algorithm. In this embodiment, a table mapping CPU utilization ranges to different levels of preset scaling algorithms is generated using a lookup table structure and stored in a memory data structure, file, or database. When calling from an in-memory data structure, if using a dictionary or hash table for storage, the current CPU usage is matched by traversing key-value pairs; when calling from a file (JSON format), the JSON file is read and parsed into a Python object for querying; when calling from a database, the corresponding algorithm is retrieved from the database table using SQL query statements.

[0046] Step 203: According to the preset segmentation resolution, each of the original data frames in the video decoding data is segmented to obtain multiple segmentation blocks corresponding to each of the original data frames.

[0047] The video decoding data also includes metadata, which includes the resolution of the original data frames.

[0048] This step can be referred to as step 102 above, and will not be repeated here.

[0049] Optionally, step 203 may also include: Sub-step 2031: Based on the resolution of the original data frame and the target scaling algorithm, determine the preset segmentation resolution, which defines the pixel specifications of the segmentation block.

[0050] Sub-step 2032: According to the preset segmentation resolution, each original data frame in the video decoding data is segmented to obtain multiple segmentation blocks corresponding to each original data frame.

[0051] It should be noted that segmentation resolution is an indicator in image segmentation that describes pixel-level classification accuracy, reflecting the system's ability to identify image details. A higher segmentation resolution value indicates that the smallest distinguishable feature unit is smaller, resulting in a more refined segmentation result.

[0052] In this embodiment, for sub-steps 2031 and 20432, the pixel specifications of the segmentation blocks are defined by a preset segmentation resolution. That is, the total number of pixels in the segmentation block is expressed as horizontal pixels × vertical pixels (e.g., 800 × 600). The width and height of the original data frame, for example 1920 and 1080, are obtained from the metadata of the video decoded data using command-line instructions from the ffprobe component of FFmpeg, and combined to form an original data frame with a resolution of 1920 × 1080. Without image scaling or distortion, the segmentation calculation method is as follows: Area ratio calculation: Original data frame resolution of 1920 × 1080: area 2,073,600 pixels; Preset segmentation resolution of 800 × 600: area 480,000 pixels. Number of segments: Horizontal: 1920 ÷ 800 = 2.4 (3 segments if calculated as integers), Vertical: 1080 ÷ 600 = 1.8 (2 segments if calculated as integers), Total segments: 3 × 2 = 6 segments. Resolution ratio calculation: 1920:800 = 2.4, 1080:600 = 1.8, neither is divisible, so it can be obtained by rounding. Number of segments = 3 × 2 = 6 (6 segments if rounded). If the results of the two segmentation calculation methods are consistent, black borders can be added to the rounded areas while maintaining the 800×600 ratio. According to the target scaling algorithm, each segment is scaled to the resolution supported by the monitor window size to achieve the best display effect. However, when segmenting the original data frame according to the preset segmentation resolution, it may be necessary to adjust the pixel size of the segmented blocks or use a non-integer number of blocks if the alignment method of the target scaling algorithm (e.g., integer scaling) is forced to be used for stitching, depending on the target scaling algorithm.

[0053] Alignment methods in image scaling algorithms include: origin coordinate alignment, corner coordinate alignment, and image center alignment. Origin coordinate alignment directly scales the coordinates proportionally, ensuring alignment only at the top-left corner (0,0), and is commonly used in nearest-neighbor interpolation or bilinear interpolation algorithms. Corner coordinate alignment ensures corner alignment between the source and target images during scaling, although the center point may shift; this is commonly used in convolutional neural networks. Image center alignment scales based on the geometric center of the image, ensuring center point alignment, although edges may slightly shift; this is commonly used in bilinear or bicubic interpolation algorithms. In short, different alignment methods affect interpolation accuracy and visual effects.

[0054] Sub-step 2031 may also include: Sub-step a: Within one screen refresh cycle of the playback device, obtain the maximum number of pixels processed by the target scaling algorithm on a single thread.

[0055] Sub-step b: Based on the pixel alignment method of the target scaling algorithm, select a resolution with a total number of pixels equal to the maximum number of pixels as the preset segmentation resolution.

[0056] It's important to note that the screen refresh rate is the number of times a monitor refreshes its image per second, usually measured in Hertz (Hz). For example, 60 Hz means 60 refreshes per second. The higher the refresh rate, the smoother the image. Monitor size typically refers to the length of the screen's diagonal, while resolution refers to the number of pixels arranged horizontally and vertically on the screen. The relationship between the two is mainly reflected in pixel density (PPI), which is the number of pixels per inch.

[0057] In this embodiment, for sub-steps a and b, the maximum number of pixels refers to the maximum number of pixels that a single thread of the CPU can process within one screen refresh cycle of 1 / 60th of a second, or 16ms, when running the video scaling algorithm. Since different algorithm levels are set for the video scaling algorithms in the FFmpeg codec library, different maximum number of pixels can be obtained based on the single-threaded image data processing capabilities of different algorithm levels, and the preset segmentation resolution can be determined based on the respective aspect ratios determined by each video scaling algorithm.

[0058] Specifically, based on the alignment methods described above in the target scaling algorithm, such as origin coordinate alignment, corner coordinate alignment, and image center alignment, a resolution with a total number of pixels equal to the maximum number of pixels can be selected as the preset segmentation resolution. Since the aspect ratio of the resolution is determined by the supported aspect ratio of the display device (e.g., 16:9, 4:3), for example, 1920×1080 is a 16:9 aspect ratio, while 800×600 is a 4:3 aspect ratio. If the resolution does not match the supported aspect ratio of the corresponding device, the display area may not be filled or the content may be compressed. Therefore, when multiple segmentation resolutions exist when the total number of pixels equals the maximum number of pixels, it is also necessary to consider the supported size and compatibility of the display device to select a segmentation resolution with a suitable aspect ratio to ensure matching with the native aspect ratio of the display device or output platform.

[0059] When the aspect ratio determined by the alignment method of the target scaling algorithm and the size supported by the display device are inconsistent, the `force_original_aspect_ratio` parameter can be used to ensure that the video is not distorted, and black borders can be added to adapt to the original aspect ratio of the output platform, ensuring that the proportion of the video image content is correct; alternatively, the edges of the original data frames can be cropped to completely fill the display area of ​​the display device; or dynamic parameters can be used to let FFFFmpeg automatically calculate the optimal size: `scale=800:-1` - fixed width, height automatically calculated; `scale=-1:600` - fixed height, width automatically calculated. The resolution information of the original video can be obtained first using `ffprobe`, and then the optimal preset segmentation resolution can be calculated based on the size of the target display device. When the total number of pixels corresponding to the resolution of each original data frame is less than the maximum number of pixels, it can be guaranteed that the CPU processes the video according to the optimal preset segmentation resolution by loading the target scaling algorithm, and no frames are dropped during video playback. Therefore, the number of pixels calculated at the preset segmentation resolution is less than or equal to the maximum number of pixels to ensure that the number of pixels of the image resolution displayed on the screen within the screen refresh cycle is less than or equal to the maximum number of pixels.

[0060] In one embodiment, a suitable preset segmentation resolution can be selected based on the specific application scenario and requirements. In scenarios with high real-time requirements, such as video processing, nearest neighbor or bilinear interpolation with higher scaling efficiency can be prioritized, and smaller segmentation blocks can be selected to improve processing speed. In scenarios with high quality requirements, such as professional image editing, a larger preset segmentation resolution can be set to obtain larger segmentation blocks to ensure image quality. In environments with limited system resources, such as mobile devices, the segmentation resolution can be dynamically adjusted.

[0061] Optionally, step 203 may also include: When the preset segmentation resolution is less than the resolution of the original data frame, the data is segmented according to the preset segmentation resolution to obtain multiple segmentation blocks corresponding to each original data frame. When the preset segmentation resolution is greater than or equal to the resolution of the original data frame, the segmentation process of the original data frame is stopped.

[0062] It's important to note that video resolution refers to the pixel size of the entire video frame, reflecting overall clarity; raw data frame resolution refers to the pixel size of a single frame, usually consistent with the video resolution. Resolution determines the fineness of detail in the image, i.e., clarity. The higher the video frame resolution, the more pixel information it contains, and the clearer the image.

[0063] In this embodiment, the preset segmentation resolution is a pre-set number of pixels used to segment the video image, such as 800×600. This preset segmentation resolution can be set according to the playback device platform information and the alignment requirements of the selected video scaling algorithm. The resolution of the original data frame is a parameter used to measure the amount of data within an image, usually expressed as ppi. Assuming the preset segmentation resolution is 800×600, if the resolution of the original data frame is 800×400, no segmentation is needed; when the resolution of the original data frame is 1920×1080, each original data frame is segmented at 800×600 to obtain multiple segmentation blocks for each original data frame.

[0064] In one embodiment, it can be determined whether the original data frames need to be segmented by comparing the total number of pixels calculated from the resolution. If the total number of pixels determined by the resolution of the original video data frames is less than or equal to the maximum number of pixels, segmentation is not required. If the total number of pixels determined by the resolution of the original video data frames is greater than the maximum number of pixels, segmentation is required.

[0065] Step 204: Based on the number of segmented blocks corresponding to each original data frame, the target scaling algorithm, and the number of threads in the preset thread pool, multiple execution data packets corresponding to the original data frames are generated respectively.

[0066] This step can be referred to as step 103 above, and will not be repeated here.

[0067] Optionally, step 204 may also include: Sub-step 2041: Obtain the maximum number of threads supported by the processor of the playback device, and create a preset thread pool with the maximum number of threads.

[0068] Sub-step 2042: Based on the maximum number of threads, the segmentation blocks corresponding to each original data frame are evenly divided into groups to obtain segmentation block groups.

[0069] Sub-step 2043: Each of the segmented blocks is grouped and combined with the target scaling algorithm to obtain the execution data packet.

[0070] It should be noted that video scaling is a CPU-intensive task. Scaling operations involve mathematical calculations on a large amount of pixel data, requiring continuous CPU computing resources while having low I / O dependence. For example, when processing 4K video using encoding standards such as H.265, CPU utilization will be significantly improved.

[0071] In this embodiment, the maximum number of threads supported by the processor is obtained based on the platform information of the video playback device, and a preset thread pool with the maximum number of threads is created. For example, if the processor supports a maximum of 12 threads, a thread pool of 12 threads can be created when scaling the video decoding data. Then, the segmented blocks corresponding to each original data frame are evenly divided into groups according to the maximum number of threads. Each group is then combined with the target scaling algorithm to obtain an execution data package equal to the maximum number of threads. The above combination method can be either package combination or task encapsulation. Task encapsulation can bundle data and methods for operating on the data together to form an independent class unit. Encapsulation declares style data packages, grouping information, and thread information as private, providing access interfaces only through public member functions.

[0072] For example, the core parameter configuration is as follows: `corePoolSize`: the number of core threads (e.g., 16), always alive; `maximumPoolSize`: the maximum number of threads (e.g., 64), scalable when tasks surge. `keepAliveTime`: the idle timeout for non-core threads (e.g., 10 seconds). `workQueue`: the task queue (e.g., `ArrayBlockingQueue`, capacity 64). The processor can use Hyper-Threading technology to implement two logical threads per physical core; therefore, a 32-core processor supports a maximum of 64 threads. The processor for acquiring the playback device supports a maximum of 64 threads. Each original data frame corresponds to 128 segmented blocks, so the segmented block grouping is 128 ÷ 64 = 2. Then, each of the 64 segmented block groups is combined with the target scaling algorithm to obtain 64 execution data packets. In this embodiment, the two segmented blocks allocated to each thread are combined using a structure array to combine the 64 segmented block groupings. Task encapsulation is performed using structures or classes, encapsulating the 64 segmented block groupings and the target scaling algorithm operations within a unified structure to form independent execution data packets. Each execution packet contains complete input parameters, algorithm configuration, and output buffers, enabling direct execution by a thread without additional data preparation. The thread scheduler then places 64 execution packets into an execution queue, which can be processed by the core thread (the first 16 packets are executed directly by the core thread), or by a queue buffer (subsequent tasks (17 to 64) are stored in the queue to be processed when the core thread is idle). Dynamic scaling occurs when the queue is full and the number of tasks exceeds 64, and the thread pool creates non-core threads (up to 64) to execute the tasks. A rejection policy is triggered if the number of threads reaches 64 and the queue is full (e.g., throwing an exception or discarding the task). Since the thread pool has a fixed size, e.g., 64 packets, execution packets exceeding 64 need to wait in the queue, thus enabling concurrent processing by the thread scheduler.

[0073] Optionally, each segment has a corresponding data descriptor, which contains the address of the original data frame. Step 204 may further include: Sub-step 2044: Obtain the maximum number of threads supported by the processor of the playback device, and create a preset thread pool with the maximum number of threads.

[0074] Sub-step 2045: Based on the maximum number of threads, the data descriptors corresponding to the segmented blocks of each original data frame are evenly divided into groups to obtain data descriptor groups.

[0075] Sub-step 2046: Each data descriptor group is combined with the target scaling algorithm to obtain an execution descriptor corresponding to each data descriptor group. The execution descriptor is used to indicate the schedulable execution data packets of the segmented block groups of the original data frame.

[0076] It should be noted that data descriptors are variables or indicators used to quantify the characteristics of data, and are typically used to simplify complex data analysis and comparisons.

[0077] In this embodiment, each segment can be described using data descriptors to simplify the grouping and encapsulation process. For example, segmented blocks A, B, C, and D have descriptors block1, block2, block3, and block4, respectively. The original data frame data within each segment is stored as files in a distributed file system. Each original data frame corresponds to an independent file or file block, while the corresponding data descriptors and the original data frame data of the segment are stored in the same or different nodes. A fast association is achieved using an index file or a distributed key-value table, such as Apache HBase. When accessing the original data frame data within each segment is needed, the playback device's system can first find the data descriptor through the index file, and then locate the associated original data frame data based on the metadata (such as frame number, timestamp, and resolution) in the data descriptor. Alternatively, structured data descriptors can be stored in relational data blocks for easy querying and association.

[0078] Therefore, a data descriptor can be generated for each segment corresponding to each original data frame to replace the segment for grouping and encapsulation, which can improve data processing efficiency and flexibility. Storing the data descriptor separately from the array data of the segment improves data access efficiency and supports the management of complex data structures.

[0079] Step 205: Distribute the multiple execution data packets to the preset threads in the preset thread pool for execution, and obtain the scaled target data frame.

[0080] In this embodiment, the execution results can be aggregated into a scaled target data frame by combining regions based on similarity. First, merging can be performed based on the image features of the regions occupied by adjacent segmented blocks. Statistical features such as the grayscale mean and standard deviation are calculated for the processed features of each segmented block. A pre-set segment merging threshold can be used; when the feature difference between the regions occupied by adjacent segmented blocks is less than the threshold, a recursive merging check is performed. Multiple scaled segmented blocks corresponding to each original data frame are then aggregated upwards until all segmented blocks are aggregated into an image that achieves the target display resolution. The step of distributing execution data packets to preset threads in a preset thread pool can be referred to step 104 above, and will not be repeated here.

[0081] Optionally, the method may further include: Step 206: Obtain the number of displays supported by the playback device.

[0082] Step 207: When the playback device plays video from multiple displays, select the scaling algorithm with the smallest computational load from multiple preset scaling algorithms as the target scaling algorithm; the computational load of the scaling algorithm is inversely proportional to the distortion rate of the video decoding data.

[0083] It should be noted that multi-monitor video playback refers to the use of technology to simultaneously or split-screen the content of one or more video signal sources across multiple monitors. Because monitors with different resolutions need to be automatically configured to ensure compatible video resolutions and optimal display quality, this process is crucial.

[0084] In this embodiment, the number of displays supported by the playback device and the number of currently connected and available displays are obtained. When the playback device outputs to multiple displays, different scaling processes can be performed on the multiple segments corresponding to the original data frames of the video decoding data, depending on the display method, such as separate display on multiple screens or simultaneous display on multiple screens. Different target scaling algorithms need to be combined for the segments output to different displays. During multi-display video playback, a large amount of decoded image data needs to be processed simultaneously, leading to excessive CPU load and potentially affecting the smoothness of video decoding and scaling processes. A scaling algorithm with low computational complexity, such as the nearest neighbor algorithm, can be selected as the target scaling algorithm.

[0085] In summary, a target scaling algorithm can be selected based on the device information of the playback device. Based on the target scaling algorithm and the preset segmentation resolution, the original data frames included in the video decoding data are segmented to obtain multiple segmented blocks for each original data frame. By grouping multiple segmented blocks and combining them with the target scaling algorithm to form an execution data packet, scaling processing is performed through the available threads of the playback device to simplify software scaling processing. Calculations are performed on the segmented blocks of the original data frames according to the target scaling algorithm, improving the processor's processing efficiency during video playback. This ensures that the target scaling algorithm is adapted to the processing of the original data frames to avoid frame loss due to excessive data processing, thereby preventing video stuttering on the display.

[0086] Figure 3 This is a block diagram of a multi-threaded video processing device provided in an embodiment of the present invention, such as... Figure 3 As shown, the device includes: The algorithm selection module 301 is used to determine the target scaling algorithm from multiple preset scaling algorithms based on the device information of the playback device.

[0087] The segmentation module 302 is used to segment each original data frame in the video decoding data according to a preset segmentation resolution, so as to obtain multiple segmentation blocks corresponding to each original data frame.

[0088] The generation module 303 is used to generate multiple execution data packets corresponding to the original data frame based on the number of segmented blocks corresponding to each original data frame, the target scaling algorithm, and the number of threads in the preset thread pool.

[0089] The execution module 304 is used to allocate the plurality of execution data packets to preset threads in the preset thread pool for execution, thereby obtaining scaled target data frames.

[0090] Optional, the algorithm selection module includes: The first acquisition submodule is used to acquire the current processor utilization rate of the playback device.

[0091] The first selection submodule is used to select the scaling algorithm corresponding to the current processor utilization rate as the target scaling algorithm based on the current processor utilization rate and the preset correspondence. The preset correspondence includes the correspondence between the processor utilization rate and the preset scaling algorithm.

[0092] Optionally, the algorithm selection module may also include: The third acquisition submodule is used to obtain the number of displays supported by the playback device.

[0093] The third selection submodule is used to select the scaling algorithm with the smaller computational load from multiple preset scaling algorithms as the target scaling algorithm when the playback device is playing video from multiple displays; the computational load of the scaling algorithm is inversely proportional to the distortion rate of the video decoding data.

[0094] Optionally, the segmentation module also includes: The determination submodule is used to determine the preset segmentation resolution based on the resolution of the original data frame and the target scaling algorithm, wherein the preset segmentation resolution defines the pixel specifications of the segmentation block.

[0095] The first segmentation submodule is used to segment each original data frame in the video decoding data according to the preset segmentation resolution, so as to obtain multiple segmentation blocks corresponding to each original data frame.

[0096] Optionally, the determination submodule also includes: The second acquisition submodule is used to acquire the maximum number of pixels processed by the target scaling algorithm on a single thread within one screen refresh cycle of the playback device. The second selection submodule is used to select a resolution with a total number of pixels equal to the maximum number of pixels as the preset segmentation resolution based on the pixel alignment method of the target scaling algorithm.

[0097] Optionally, the segmentation module also includes: The first segmentation submodule is used to segment the original data frame according to the preset segmentation resolution when the preset segmentation resolution is less than the resolution of the original data frame, so as to obtain multiple segmentation blocks for each original data frame.

[0098] The second segmentation submodule is used to stop segmenting the original data frame when the preset segmentation resolution is greater than or equal to the resolution of the original data frame.

[0099] Optionally, the generation module may also include: The first thread submodule is used to obtain the maximum number of threads supported by the processor of the playback device and create a preset thread pool with the maximum number of threads.

[0100] The first grouping submodule is used to divide the segmented blocks corresponding to each original data frame into equal groups based on the maximum number of threads, thereby obtaining segmented block groups.

[0101] The first combination submodule is used to combine each segmented block group with the target scaling algorithm to obtain the execution data packet.

[0102] Optionally, the generation module may also include: The second thread submodule is used to obtain the maximum number of threads supported by the processor of the playback device as the preset number of threads, and to create a preset thread pool.

[0103] The second grouping submodule is used to divide the data descriptors corresponding to the corresponding segment blocks into multiple data descriptor groups based on the preset number of threads and the number of segment blocks corresponding to each original data frame.

[0104] The second combination submodule is used to combine each of the data descriptor groups with the target scaling algorithm to obtain an execution descriptor corresponding to each of the data descriptor groups. The execution descriptor is used to indicate the schedulable execution data packets of the segmented block groups of the original data frame.

[0105] The multi-threaded video processing device provided in this embodiment of the invention can select a target scaling algorithm according to the device information of the playback device, and perform segmentation on the original data frames included in the video decoding data based on the target scaling algorithm and a preset segmentation resolution to obtain multiple segmented blocks for each original data frame. By grouping the multiple segmented blocks and combining them with the target scaling algorithm to form an execution data packet, scaling processing is performed by the available thread of the playback device to simplify software scaling processing. The calculation is performed on the segmented blocks of the original data frame according to the target scaling algorithm, which improves the processing efficiency of the processor in video playback, ensures the adaptation of the target scaling algorithm to the processing of the original data frame to avoid frame loss due to excessive data processing, and thus avoids video stuttering on the display.

[0106] Figure 4This is a block diagram of another multi-threaded video processing device provided in this embodiment of the invention. The video input module 401 is used to acquire video decoding data generated by the hardware VPU, i.e., image data, or image data generated by other decoding tools, which generally does not match the size of the display window. The scaling algorithm module 402 is used to select a suitable scaling algorithm from a scaling algorithm library based on the device information of the playback device and the video decoding data, wherein different scaling algorithms have different performance characteristics and display characteristics. The data segmentation module 403 is used to segment the image within each original data frame according to a preset segmentation resolution to obtain multiple segmentation blocks corresponding to each original data frame, wherein the preset segmentation resolution defines the pixel specifications of the segmentation blocks. Each segmentation block can be defined using a data descriptor, which includes current resolution information, target scaling algorithm information, the source address of the data to be processed, and the target address. The data segmentation module can also package and combine the segmented data descriptors according to the maximum number of threads to generate task data packets. It can also be used to combine and encapsulate multiple task data packets with the target scaling algorithm to generate an executable descriptor. The thread pool module 404 provides thread services for the playback device. Its main function is to identify the playback device platform information and create the maximum number of concurrent threads that the current processor can support. The task management module 405 receives executable descriptors and submits them to the execution queue. It waits until all executable descriptors have been executed in a loop to obtain the target image data for image scaling. The task management module can also bind executable descriptors to preset threads and clear process data during image scaling, such as raw data frames, segmented blocks, data descriptors, task data packets, and executable descriptors. The video output module 406 outputs the scaled target image data to a display device, converting the scaled video signal into a visible image or projection for display. The video output module can be, for example, a monitor, projector, television, or video wall.

[0107] In this embodiment, each raw data frame of the video decoding data can be segmented to obtain multiple segments, and a suitable target scaling algorithm can be selected, along with the maximum number of concurrent threads supported by the playback device's processor. The data descriptors corresponding to the segments are grouped according to the maximum number of threads and combined with the target scaling algorithm to form execution descriptors. Execution descriptors can be distributed to pre-bound threads for processing, repeating this operation until the thread pool is exhausted. After all execution descriptors have been distributed and executed, the target image data is obtained and output to the display, while all intermediate data is cleared. Therefore, multi-threaded image scaling processing can improve the processor's processing efficiency during video playback, ensure the target scaling algorithm's adaptation to the raw data frame processing, avoid frame drops due to excessive data processing, and thus prevent video stuttering on the display.

[0108] Optionally, embodiments of the present invention also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described multi-threaded image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0109] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0110] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present invention.

[0111] The electronic device 1300 includes, but is not limited to, components such as: radio frequency unit 1301, network module 1302, audio output unit 1303, input unit 1304, sensor 1305, display unit 1306, user input unit 1307, interface unit 1308, memory 1309, and processor 1310.

[0112] Those skilled in the art will understand that the electronic device 1300 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1310 through a power management playback device, thereby enabling functions such as managing charging, discharging, and power consumption through the power management playback device. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0113] It should be understood that, in this embodiment of the invention, the input unit 1304 may include a graphics processing unit (GPU) 13041 and a microphone 13042. The GPU 13041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1306 may include a display panel 13061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 1307 includes a touch panel 13071 and at least one of other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0114] The memory 1309 can be used to store software programs and various data. The memory 1309 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store application programs or instructions required to operate the playback device or at least one function (such as sound playback function, image playback function, etc.). Furthermore, the memory 1309 may include volatile memory or non-volatile memory, or both.

[0115] Processor 1310 may include one or more processing units; optionally, processor 1310 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operation of playback devices, user interfaces, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 1310.

[0116] Embodiments of the present invention also provide a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described multi-threaded video processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0117] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A multi-threaded video processing method, characterized in that, include: Based on the device information of the playback device, a matching target scaling algorithm is determined from multiple preset scaling algorithms; According to the preset segmentation resolution, each original data frame in the video decoding data is segmented to obtain multiple segmentation blocks corresponding to each original data frame; Based on the number of segmented blocks corresponding to each original data frame, the target scaling algorithm, and the preset number of threads in the preset thread pool, multiple execution data packets corresponding to the original data frames are generated respectively. The multiple execution data packets are distributed to preset threads in the preset thread pool for execution, resulting in scaled target data frames.

2. The method according to claim 1, characterized in that, The step of determining the matching target scaling algorithm from multiple preset scaling algorithms includes: Obtain the current processor utilization rate of the playback device; Based on the current processor utilization rate and a preset correspondence, a scaling algorithm corresponding to the current processor utilization rate is selected as the target scaling algorithm; the preset correspondence includes the correspondence between the processor utilization rate and the preset scaling algorithm.

3. The method according to claim 1, characterized in that, The step of segmenting each of the original data frames in the video decoding data includes: Based on the resolution of the original data frame and the target scaling algorithm, the preset segmentation resolution is determined; the preset segmentation resolution is used to define the pixel specifications of the segmented blocks. According to the preset segmentation resolution, each original data frame in the video decoding data is segmented to obtain the segmentation block corresponding to each original data frame.

4. The method according to claim 3, characterized in that, Determining the preset segmentation resolution based on the resolution of the original data frame and the target scaling algorithm includes: Within the refresh cycle of the playback device, obtain the maximum number of pixels processed by the target scaling algorithm on a single thread; Based on the alignment method of the target scaling algorithm, a resolution with a total number of pixels equal to the maximum number of pixels is selected as the preset segmentation resolution.

5. The method according to claim 1, characterized in that, The step of segmenting each of the original data frames in the video decoding data includes: When the preset segmentation resolution is less than the resolution of the original data frame, the original data frame is segmented according to the preset segmentation resolution to obtain multiple segmentation blocks corresponding to each original data frame; otherwise, the segmentation process of the original data frame is stopped.

6. The method according to claim 1, characterized in that, The step of generating multiple execution data packets corresponding to the original data frame includes: Obtain the maximum number of threads supported by the processor on the playback device as the preset number of threads, and create the preset thread pool; Based on the preset number of threads and the number of segments corresponding to each original data frame, the corresponding segments are evenly divided into groups to obtain multiple segments groupings; Each of the segmented blocks is grouped and combined with the target scaling algorithm to obtain the execution data packet.

7. The method according to claim 1 or 6, characterized in that, Each segment has a corresponding data descriptor, which contains the address of the original data frame; The step of generating multiple execution data packets corresponding to the original data frame includes: Obtain the maximum number of threads supported by the processor on the playback device as the preset number of threads, and create the preset thread pool; Based on the preset number of threads and the number of segments corresponding to each original data frame, the corresponding data descriptors of the corresponding segments are evenly divided into groups to obtain multiple data descriptor groups; Each data descriptor group is combined with the target scaling algorithm to obtain an execution descriptor corresponding to each data descriptor group; the execution descriptor is used to indicate the schedulable execution data packets in the segmented block groups of the original data frame.

8. The method according to claim 1, characterized in that, The target scaling algorithm for determining the matching target includes: When the playback device plays video from multiple displays, it selects a preset scaling algorithm with low computational complexity from multiple preset scaling algorithms as the target scaling algorithm; the computational complexity of the scaling algorithm is inversely proportional to the distortion rate of the video decoding data.

9. A multi-threaded video processing device, characterized in that, include: The algorithm selection module is used to determine the target scaling algorithm from multiple preset scaling algorithms based on the device information of the playback device. The segmentation module is used to segment each original data frame in the video decoding data according to a preset segmentation resolution, so as to obtain multiple segmentation blocks corresponding to each original data frame; The generation module is used to generate multiple execution data packets corresponding to the original data frame based on the number of segmented blocks corresponding to each original data frame, the target scaling algorithm, and the number of threads in the preset thread pool. The execution data packets include at least a portion of the segmented blocks of the original data frame and the target scaling algorithm. An execution module is used to allocate the plurality of execution data packets to preset threads in the preset thread pool, and to obtain scaled target data frames through scaling processing.

10. An electronic device, characterized in that, include: Memory; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 8.

Images