Large-screen splicing method and system

Abstract

The application relates to the technical field of image processing, and discloses a large-screen splicing method and system, which comprises the following steps: accessing multiple video streams, performing hardware parallel decoding, obtaining original video frames and storing the original video frames in a kernel shared buffer area identified by a file descriptor; batch reading each file descriptor, establishing address mapping and encapsulating into an accessible buffer object; obtaining a first estimated delay processed in a two-dimensional graphics acceleration mode and a second estimated delay processed in a general heterogeneous computing mode, comparing the two estimated delays and selecting the processing mode with the lower estimated delay, performing scaling, cropping or format conversion on the original video frames to obtain processed video frames; in response to changes in splicing layout parameters, the number of video channels or the resolution of any channel, releasing and rebuilding a buffer pool; performing layer mixing and synthesis on each processed video frame to obtain a synthesis frame and outputting the synthesis frame to a spliced large screen. The application can realize adaptive selection of processing modes and dynamic matching of buffer resources.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a large-screen splicing method and system. Background Technology

[0002] Large-screen splicing systems are typically used for the centralized display of multiple video feeds. The process involves multiple stages, including video input, video decoding, image scaling, cropping, format conversion, layer compositing, and display output. As the number of input video feeds and video resolutions increase, traditional software splicing methods relying on a central processing unit (CPU) require serial or semi-serial processing of multiple video frames. This can easily lead to increased processing latency with the scale of the input, making it difficult to consistently meet the processing requirements for simultaneous display of multiple high-resolution videos.

[0003] While some existing large-screen splicing solutions utilize graphics processors (GPUs) for image processing, multiple data transfers may still occur between video decoding output, the CPU buffer, the GPU buffer, and the display buffer. In scenarios involving concurrent multi-channel video splicing, the amount of video frame data increases with the number of channels and resolution. These multiple data transfers consume memory bandwidth and processing resources, making data transmission outside of image processing a factor affecting overall processing efficiency.

[0004] While some existing embedded video processing solutions can perform localized processing using hardware video decoders or single image acceleration units, they typically lack a continuous processing chain around large-screen splicing applications, encompassing hardware decoding, shared buffer transfer, image acceleration processing, layer blending and compositing, and finally, display output. For scenarios involving changes in splicing layout, the number of video channels, or channel resolution, existing solutions lack buffer pool reconstruction processing to match the large-screen splicing layout, which can easily lead to issues such as buffer size mismatch with the current display area, prolonged occupation of invalid handles, or untimely resource reclamation.

[0005] Furthermore, embedded platforms typically possess both 2D graphics acceleration and general-purpose heterogeneous computing capabilities. Different processing methods have varying degrees of adaptability to different color sampling formats, resolutions, and image processing operation types. If a single processing method is consistently used, it may be impossible to simultaneously achieve format compatibility and processing efficiency for certain video formats or combinations of processing operations.

[0006] Therefore, this application provides a large-screen splicing method to solve the above-mentioned technical problems. Summary of the Invention

[0007] The purpose of this invention is to provide a large-screen splicing method and system to solve the technical problems in the prior art of large-screen splicing processing, such as multiple data transfers of multiple video frames, fragmented embedded hardware processing links, limited processing options, and difficulty in matching buffer resources with changes in splicing services.

[0008] To address the aforementioned technical problems, this invention provides a large-screen splicing method, comprising:

[0009] The system receives multiple video streams to be spliced ​​and splicing layout parameters, binds each video stream to its corresponding processing channel, performs hardware parallel decoding, obtains the original video frames, and stores them in the kernel shared buffer identified by the file descriptor. Batch read each file descriptor, establish an address mapping for the kernel shared buffer, and based on the address mapping, encapsulate it into a buffer object that can be accessed by image scaling, cropping or compositing operations; For each original video frame, the color sampling format, resolution, and the type of image processing operation to be performed are read. From the pre-stored corresponding records indexed by the format, resolution, and operation type, the first estimated delay for processing in a two-dimensional graphics acceleration mode and the second estimated delay for processing in a general heterogeneous computing mode are obtained respectively. The first estimated delay and the second estimated delay are compared, and the processing mode with the lower estimated delay is selected. The original video frame is then scaled, cropped, or converted in format to obtain the processed video frame. When the splicing layout parameters, the number of video channels, or the resolution of any channel change, the buffer handles in the current buffer pool are released, the number of buffers and the buffer size are recalculated according to the changed splicing layout parameters, the number of video channels, and the channel resolution, and the buffer pool is reallocated. According to the splicing layout parameters, the processed video frames of each channel are layered and composited to obtain a composite frame, which is then output to the splicing screen.

[0010] In some specific embodiments, the process involves accessing multiple video streams to be spliced ​​and splicing layout parameters, binding each video stream to its corresponding processing channel, performing hardware parallel decoding to obtain the original video frames, and storing them in a kernel shared buffer identified by a file descriptor. Further steps include: Read the input interface type, channel identifier, and display area from the splicing layout parameters for each video stream; Establish a channel binding relationship based on the channel identifier and the display area, and write each video stream into the processing channel corresponding to the channel binding relationship; The video streams in each processing channel are decoded in hardware in parallel, and the decoded raw video frames are written to the kernel shared buffer. Generate a file descriptor for each kernel shared buffer and associate the file descriptor with the channel identifier, frame size, and color sampling format.

[0011] In some specific embodiments, each file descriptor is read in batches, an address mapping is established for the kernel shared buffer, and based on the address mapping, it is encapsulated into a buffer object that can be accessed by image scaling, cropping, or compositing operations, further including: Organize multiple file descriptors within the same processing cycle into an import list according to the channel identifier; Verify the access status and frame size of the kernel shared buffer corresponding to each file descriptor in the import list; For file descriptors that pass verification, establish an address mapping between the kernel shared buffer and the image processing address space, and retain the mapping handle; Write the mapping handle, channel identifier, frame size, and color sampling format into the buffer descriptor, and combine multiple buffer descriptors into a set of buffer objects.

[0012] In some specific embodiments, a first estimated latency for processing in a two-dimensional graphics acceleration manner and a second estimated latency for processing in a general heterogeneous computing manner are obtained from a pre-stored corresponding record indexed by the format, resolution, and operation type, respectively. The first estimated latency and the second estimated latency are compared, and the processing method with the lower estimated latency is selected. This further includes: Combine color sampling format, resolution, and image processing operation type into query conditions; Match the records that are identical to the query conditions in the pre-stored corresponding records, and read the first estimated delay and the second estimated delay; When no matching record is found, the first estimated delay and the second estimated delay are calculated by using the record with the same color sampling format and the same image processing operation type that is adjacent to the resolution, according to the ratio of the number of pixels of the resolution to the resolution of the adjacent record. Compare the first estimated delay and the second estimated delay, select the processing method corresponding to the lower one as the current processing method, and add the buffer object, clipping region, target size and output format to the pending operation queue.

[0013] In some specific embodiments, a processing method with lower estimated latency is selected to perform scaling, cropping, or format conversion on the original video frame to obtain a processed video frame, further including: When the selected processing method is 2D graphics acceleration, the buffer objects, clipping regions, target sizes and output formats in the processing queue will be clipped, scaled or formatted according to the 2D graphics acceleration processing method. When the selected processing method is general heterogeneous computing, the buffer objects in the queue of operations to be processed are mapped to parallel computing inputs, and parallel computing is performed according to the clipping region, target size and output format to complete clipping, scaling or format conversion; The processed video frames are written to the processed video frame queue.

[0014] In some specific embodiments, in response to changes in the stitching layout parameters, the number of video channels, or the resolution of any channel, the buffer handles in the current buffer pool are released, the number of buffers and the buffer size are recalculated according to the changed stitching layout parameters, the number of video channels, and the channel resolution, and the buffer pool is reallocated, further including: Monitor the splicing layout parameter version, channel binding relationship, and frame size of each channel; When any monitoring result is inconsistent with the current buffer pool record, the buffer handles in the current buffer pool that are not associated with the output composite frame will be released, and the buffer handles that are still referenced by the current processing cycle will be marked as pending release. Based on the changed stitching layout parameters, the number of video channels and the channel resolution, calculate the number and size of the input buffer, the post-processing buffer and the compositing buffer; Allocate a new buffer pool.

[0015] In some specific embodiments, the processed video frames are layer-blended and synthesized according to the splicing layout parameters to obtain a composite frame, further including: The splicing layout parameters are analyzed into the display area, display size, and image hierarchy of each processed video frame on the splicing screen. Based on the display area and display size, the corresponding processed video frames are position-mapped and size-matched to obtain the layer to be composited. Multiple layers to be composited are blended in order of image hierarchy to obtain a composite frame; When any processed video frame is missing, the processed video frame or preset fill frame from the previous processing cycle of the missing path is mapped to the corresponding display area and then participates in the transparency blending.

[0016] In some specific embodiments, outputting the synthesized frame to the video wall further includes: Write the composite frame to the display buffer associated with the splicing large screen display interface; After the display buffer returns a read complete status, the display buffer reference corresponding to the composite frame is released; Release the file descriptor references, address mapping handles, and buffer objects corresponding to the original and processed video frames within this processing cycle according to the channel identifier; Return to the batch read file descriptors for the next processing cycle until a termination command is received.

[0017] In some specific embodiments, based on the changed stitching layout parameters, the number of video channels, and the channel resolution, the number and size of the input buffer, the post-processing buffer, and the compositing buffer are calculated, further including: The changed splicing layout parameters are parsed into the width, height, and hierarchy of each display area; The input buffer size is calculated based on the channel resolution corresponding to each video channel, and the processed buffer size is calculated based on the width and height of the display area corresponding to the video channel. Calculate the size of the composite buffer based on the output size of the spliced ​​large screen display; The number of input buffers and post-processing buffers is determined according to the number of video channels, and the number of compositing buffers is determined according to the output buffer requirements for layer blending and compositing. The calculation results are then written to a new buffer pool.

[0018] Based on the same concept, the present invention also provides a large-screen splicing system, comprising: The video access decoding module is configured to access multiple video streams to be spliced ​​and splicing layout parameters, bind each video stream to the corresponding processing channel, perform hardware parallel decoding, obtain the original video frames, and store them in the kernel shared buffer identified by the file descriptor; The buffer import module is configured to read each file descriptor in batches, establish an address mapping for the kernel shared buffer, and encapsulate it into a buffer object that can be accessed by image scaling, cropping or compositing operations based on the address mapping. The accelerated selection processing module is configured to read the color sampling format, resolution, and image processing operation type for each original video frame, obtain the first estimated delay for processing in a two-dimensional graphics acceleration mode and the second estimated delay for processing in a general heterogeneous computing mode from the pre-stored corresponding records indexed by the format, resolution, and operation type, compare the first estimated delay and the second estimated delay, select the processing mode with the lower estimated delay, and perform scaling, cropping, or format conversion on the original video frame to obtain the processed video frame. The buffer pool reconstruction module is configured to release the buffer handles in the current buffer pool in response to changes in the splicing layout parameters, the number of video channels, or the resolution of any channel, and recalculate the number and size of buffers according to the changed splicing layout parameters, the number of video channels, and the channel resolution, and then reallocate the buffer pool. The composite output module is configured to perform layer blending and compositing on each processed video frame according to the splicing layout parameters to obtain a composite frame, which is then output to the splicing screen.

[0019] Compared with existing technologies, its advantages are as follows: This invention discloses a large-screen splicing method and system, which decodes multiple video streams in parallel through hardware to obtain original video frames, and stores the original video frames in a kernel shared buffer identified by a file descriptor. Each video frame can be recorded with a unified frame description after decoding and output, thus reducing the intermediate steps of organizing and copying multiple video streams separately on the central processing unit side.

[0020] The system reads file descriptors in batches, establishes address mappings for the kernel shared buffer, and encapsulates these address mappings into buffer objects accessible to image processing operations. The raw video frames output by the hardware parallel decoding can then enter the image processing operations through the address mapping relationship, reducing the need to repeatedly move video frame data between the decoding buffer and the image processing buffer.

[0021] For each original video frame, the color sampling format, resolution, and image processing operation type to be performed are read. The first estimated latency for processing in a two-dimensional graphics acceleration mode and the second estimated latency for processing in a general heterogeneous computing mode are obtained from the pre-stored corresponding records indexed by the format, resolution, and operation type, respectively, and compared. Different video formats, different resolutions, and different combinations of image processing operations can be assigned to the processing mode with lower estimated latency, avoiding the mismatch of processing links in some scenarios caused by the fixed use of a single processing mode.

[0022] When no matching record is found, records with the same color sampling format and the same image processing operation type that are adjacent to the resolution are used. Two estimated delays are calculated according to the ratio of the number of pixels of the resolution to that of the adjacent record. Even when the pre-stored corresponding records do not completely cover all input resolutions, an executable processing method selection basis can still be formed, so that the processing method selection can cover more actual input conditions.

[0023] In response to changes in splicing layout parameters, number of video channels, or resolution of any channel, the buffer handles in the current buffer pool are released, and the number and size of buffers are recalculated according to the changed splicing layout parameters, number of video channels, and channel resolution. The buffer pool can be rebuilt as the large screen splicing service status changes, reducing the problems of buffer size mismatch with the current display area and the continuous occupation of invalid buffer handles.

[0024] When releasing buffer handles, buffer handles that are not associated with the output composite frames are released, and buffer handles that are still referenced by the current processing cycle are marked as pending release. Buffer pool reconstruction can take into account both resource reclamation and data access in the current processing cycle, and avoid premature release of video frame resources that are being imported, processed or displayed.

[0025] The display area, display size, and layer order are analyzed according to the splicing layout parameters. The corresponding processed video frames are then mapped and matched in size according to the display area and display size before transparency blending. After multi-channel processing, the video frames can form a composite frame according to the target area in the splicing screen, so that a continuous data flow is formed between scaling, cropping, format conversion, and layer blending.

[0026] When any processed video frame is missing, the processed video frame or preset fill frame from the previous processing cycle of the missing channel is mapped to the corresponding display area and then participates in transparency blending. When a video stream fails to decode, import, or image processing, the processed video frames from other channels can still participate in the generation of composite frames, reducing the impact of single-channel anomalies on the entire composite output process. Attached Figure Description

[0027] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating some specific embodiments of the large-screen splicing method of the present invention; Figure 2 This is a structural schematic diagram of a large-screen splicing system according to some specific embodiments of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms "a," "the," and "the" as used in the embodiments of this application are also intended to include the plural forms, unless the context clearly indicates otherwise, and "multiple" generally includes at least two.

[0030] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0031] It should be understood that although the terms first, second, third, etc., may be used in the embodiments of this application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, first may also be referred to as second without departing from the scope of the embodiments of this application, and similarly, second may also be referred to as first.

[0032] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0033] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0034] It should be noted that any symbols and / or numbers present in the specification that are not marked in the accompanying drawings are not reference numerals.

[0035] Reference Figure 1 A method for splicing large screens, including: S101 receives the multiple video streams to be spliced ​​and the splicing layout parameters, binds each video stream to the corresponding processing channel, performs hardware parallel decoding, obtains the original video frames, and stores them in the kernel shared buffer identified by the file descriptor; S102, Batch read each file descriptor, establish address mapping for the kernel shared buffer, and based on the address mapping, encapsulate it into a buffer object that can be accessed by image scaling, cropping or compositing operations; S103, for each original video frame, read the color sampling format, resolution, and the operation type of the image processing to be performed. From the pre-stored corresponding records indexed by the format, resolution, and operation type, obtain the first estimated delay for processing in a two-dimensional graphics acceleration mode and the second estimated delay for processing in a general heterogeneous computing mode. Compare the first estimated delay and the second estimated delay, select the processing mode with the lower estimated delay, and perform scaling, cropping, or format conversion on the original video frame to obtain the processed video frame. S104, in response to changes in splicing layout parameters, number of video channels, or resolution of any channel, release the buffer handle in the current buffer pool, recalculate the number of buffers and buffer size according to the changed splicing layout parameters, number of video channels, and channel resolution, and reallocate the buffer pool. S105 performs layer blending and compositing on each processed video frame according to the splicing layout parameters to obtain a composite frame, and outputs the composite frame to the splicing screen.

[0036] To provide a clearer explanation, the steps in the embodiments of the present invention are described in detail below: S101 receives the multiple video streams to be spliced ​​and the splicing layout parameters, binds each video stream to the corresponding processing channel, performs hardware parallel decoding, obtains the original video frames, and stores them in the kernel shared buffer identified by the file descriptor; S101 further includes: Read the input interface type, channel identifier, and display area from the splicing layout parameters for each video stream; Establish a channel binding relationship based on the channel identifier and the display area, and write each video stream into the processing channel corresponding to the channel binding relationship; The video streams in each processing channel are decoded in hardware in parallel, and the decoded raw video frames are written to the kernel shared buffer. Generate a file descriptor for each kernel shared buffer and associate the file descriptor with the channel identifier, frame size, and color sampling format.

[0037] Specifically, in this embodiment of the invention, the multiple video streams to be spliced ​​can come from network cameras, high-definition multimedia interfaces, serial digital interfaces, or other video sources capable of outputting compressed video streams. When each video stream enters the large-screen splicing process, a video stream access record is first formed. This record includes at least the input interface type, channel identifier, input frame time sequence, original encoding format, and display area corresponding to the splicing layout parameters. The splicing layout parameters describe the display position and size of each video stream on the spliced ​​large screen, and may include the coordinates of the upper left corner of the display area, the width of the display area, the height of the display area, and the layer order. When reading the input interface type, if the same channel can access different interfaces, the currently enabled interface is determined according to the channel identifier, and the input data of that interface is written into the corresponding video stream access record. If a certain video stream is not yet connected, an empty input state is generated for that channel, and a preset fill frame or a processed video frame from the previous processing cycle is used to fill the gap during subsequent layer mixing and synthesis.

[0038] The input interface type determines the video stream reading method. The channel identifier maintains a one-to-one correspondence between the video stream and the processing channel, buffer object, processed video frame, and display area throughout the entire process. The display area determines the position of the video stream that needs to be cropped, scaled, or mapped onto the splicing screen. When reading the display area, the splicing layout parameters can be parsed into layout records indexed by the channel identifier. Each layout record stores the display area, display size, and hierarchy order. If the splicing layout parameters contain multiple display areas corresponding to the same channel identifier, the channel identifier can be expanded into multiple display sub-records. Each display sub-record retains its source channel identifier so that the same processed video frame can be copied or mapped to multiple display areas later. If the display areas of multiple channels overlap, the stacking order of subsequent layer blending is determined by the hierarchy order.

[0039] Channel binding relationships are determined by both the channel identifier and the display area. When establishing a channel binding relationship, an available processing channel is located based on the channel identifier, and the display area is written into the channel configuration record of that processing channel. This ensures that the processing channel only receives the video stream corresponding to that channel identifier during the current processing cycle. For newly added video channels, an idle processing channel is allocated and a new channel binding relationship is written; for deleted video channels, their channel binding relationship is marked as pending release; for cases where the display area changes but the video channel remains the same, the original processing channel is retained and the display area in the channel configuration record is updated. After writing the processing channel information, each video stream enters the parallel decoding and shared buffer transfer process, enabling subsequent steps to batch process multiple video frames according to their channel identifiers.

[0040] The video streams in each processing channel are hardware-parallel decoded, and the decoded raw video frames are written to the kernel shared buffer. This kernel shared buffer can be allocated by the embedded system's memory management interface, and the raw video frames are directly written into it. The data format of the raw video frames can be a separate luminance / chrominance format, a luminance / chrominance interleaved format, or an image format with a transparency channel; in this embodiment, it is written as a color sampling format into the frame information. When outputting the raw video frames, the frame size, color sampling format, channel identifier, and frame sequence number are also output. If a video stream fails to decode in the current processing cycle, the decoding status of that channel is written as a failure status, and the available frame identifier from the previous processing cycle for that channel is retained for boundary processing during subsequent synthesis.

[0041] File descriptors are used to identify the kernel shared buffer where the decoded raw video frames reside. After generating a file descriptor, the file descriptor, channel identifier, frame size, and color sampling format are associated to form a frame description record. The frame description record serves as input for subsequent batch import processing. Multiple frame description records can be generated within a single processing cycle, each pointing to a single raw video frame from one video stream. If a file descriptor is successfully generated but the frame size or color sampling format is missing, the frame description record is marked as unimportable, and a missing field flag is output; subsequent steps use the processed video frames from the previous processing cycle or preset fill frames in layer blending. Through the above processing, a raw video frame data stream containing file descriptors, channel identifiers, frame sizes, and color sampling formats is formed.

[0042] S102, Batch read each file descriptor, establish address mapping for the kernel shared buffer, and based on the address mapping, encapsulate it into a buffer object that can be accessed by image scaling, cropping or compositing operations; S102 further includes: Organize multiple file descriptors within the same processing cycle into an import list according to the channel identifier; Verify the access status and frame size of the kernel shared buffer corresponding to each file descriptor in the import list; For file descriptors that pass verification, establish an address mapping between the kernel shared buffer and the image processing address space, and retain the mapping handle; Write the mapping handle, channel identifier, frame size, and color sampling format into the buffer descriptor, and combine multiple buffer descriptors into a set of buffer objects.

[0043] Specifically, in this embodiment of the invention, batch reading of file descriptors refers to the centralized reading of frame description records corresponding to multiple video streams within the same processing cycle. The output frame description records are received, and file descriptors in an importable state within the current processing cycle are added to the import list according to the channel identifier. The import list includes at least the file descriptor, channel identifier, frame size, color sampling format, frame sequence number, and display area index. If a video stream does not generate an importable file descriptor within the current processing cycle, that channel is not added to the import list, and a missing frame flag is stored in the bypass record of the import list. The missing frame flag is used to determine whether to use the processed video frames from the previous processing cycle or preset fill frames for layer blending and compositing.

[0044] Access status verification confirms that the kernel shared buffer pointed to by the file descriptor is still valid, not released, not occupied by other processing cycles and is not in an unreadable state, and has address mapping permissions. Frame size verification confirms that the width and height in the frame description record match the actual size of the kernel shared buffer. For example, the frame size, color sampling format, and line span can be converted into a minimum buffer requirement, and this requirement can be compared with the accessible capacity of the kernel shared buffer; if the accessible capacity is less than the minimum buffer requirement, the file descriptor is marked as verification failed. For file descriptors that fail verification, address mapping is not performed, no buffer description entry is generated, and the corresponding channel identifier is written to the exception record. The channel identifier in the exception record is used to select a replacement frame or a padding frame to avoid undefined regions in the synthesized frame.

[0045] Address mapping maps the kernel shared buffer to the image processing address space. When establishing an address mapping, a file descriptor can be received via the import interface provided by the driver, and a mapping handle is returned. The mapping handle is used to access the original video frames in the same kernel shared buffer during subsequent scaling, cropping, format conversion, and layer blending. This step provides the frame data written by hardware parallel decoding to the processing units accessible to the image processing operations through the mapping relationship. If the mapping interface returns a failure status, the mapping handles established during this mapping process but not yet entered into the buffer object set are released, and the corresponding file descriptors are marked as import failures. If the mapping is successful but the channel identifier does not match the frame description record, the buffer description entry is not written, and the mapping handle is released to avoid erroneous cross-channel access.

[0046] A buffer descriptor describes an accessible buffer object. Each buffer descriptor includes a mapping handle, channel identifier, frame size, color sampling format, frame sequence number, and buffer access status. Multiple buffer descriptors are combined into a buffer object set according to the channel identifier or frame sequence number. The buffer object set can be read channel by channel and serves as input for scaling, cropping, or format conversion of each raw video frame. If multiple available file descriptors exist for the same channel within a processing cycle, the latest frame description record can be added to the buffer object set based on the frame sequence number, or the frame description record matching the current display cycle can be selected based on the display synchronization strategy; unselected mapping handles are released after confirmation that they are no longer referenced. The output buffer object set retains both the access entry for each raw video frame and the frame size and color sampling format required for subsequent processing.

[0047] S103, for each original video frame, read the color sampling format, resolution, and the operation type of the image processing to be performed. From the pre-stored corresponding records indexed by the format, resolution, and operation type, obtain the first estimated delay for processing in a two-dimensional graphics acceleration mode and the second estimated delay for processing in a general heterogeneous computing mode. Compare the first estimated delay and the second estimated delay, select the processing mode with the lower estimated delay, and perform scaling, cropping, or format conversion on the original video frame to obtain the processed video frame. S103 further includes: Combine color sampling format, resolution, and image processing operation type into query conditions; Match the records that are identical to the query conditions in the pre-stored corresponding records, and read the first estimated delay and the second estimated delay; When no matching record is found, the first estimated delay and the second estimated delay are calculated by using the record with the same color sampling format and the same image processing operation type that is adjacent to the resolution, according to the ratio of the number of pixels of the resolution to the resolution of the adjacent record. Compare the first estimated delay and the second estimated delay, select the processing method corresponding to the lower one as the current processing method, and add the buffer object, clipping region, target size and output format to the pending operation queue.

[0048] Further: When the selected processing method is 2D graphics acceleration, the buffer objects, clipping regions, target sizes and output formats in the processing queue will be clipped, scaled or formatted according to the 2D graphics acceleration processing method. When the selected processing method is general heterogeneous computing, the buffer objects in the queue of operations to be processed are mapped to parallel computing inputs, and parallel computing is performed according to the clipping region, target size and output format to complete clipping, scaling or format conversion; The processed video frames are written to the processed video frame queue.

[0049] Specifically, in this embodiment of the invention, the set of output buffer objects is received, and processing mode selection and image processing are performed channel by channel using the channel identifier in the buffer description as the processing unit. When reading the color sampling format, resolution, and image processing operation type to be performed for each original video frame, the color sampling format comes from the buffer description, the resolution can be formed by the width and height in the frame size, and the image processing operation type to be performed is determined by the splicing layout parameters and the original video frame size. For example, if the original video frame size is different from the display area size, the image processing operation type includes scaling; if the display area only needs a local area in the original video frame, the image processing operation type includes cropping; if the output format required for layer blending is different from the color sampling format of the original video frame, the image processing operation type includes format conversion. If the same original video frame requires cropping, scaling, and format conversion simultaneously, these operation types are combined into an operation type set and used as part of the query conditions.

[0050] The query criteria consist of color sampling format, resolution, and image processing operation type. To ensure the query criteria can be matched with pre-stored records, the resolution can be converted into width, height, and number of pixels. The image processing operation types are arranged in the execution order of cropping, scaling, and format conversion. Pre-stored records can be established through pre-run rule configuration, platform adaptation files, or historical run statistics. Each record must include at least the color sampling format, resolution, image processing operation type, first estimated latency, and second estimated latency. The estimated latency can use milliseconds as an example unit, or it can use the timing unit within the processing cycle; as long as the two estimated latency values ​​in the same record use the same measurement benchmark.

[0051] When matching records that match the query conditions, the color sampling format, image processing operation type, and resolution are matched sequentially. Upon successful matching, the first and second estimated delays in the record are read. The reading results serve as the basis for selecting the processing method. If any of the read processing estimated delays is null, non-numerical, or marked as unavailable, the record is not used as the same record, and conversion is performed according to neighboring records. If both processing estimated delays are available but equal, the processing method can be selected according to a preset priority. In this embodiment, the processing method with the lower load in the current processing cycle can be prioritized, and this load status is written to the additional field of the selected method.

[0052] When no matching record is found, records with the same color sampling format and the same image processing operation type that are adjacent to the stated resolution are used. Adjacent records can be determined based on the difference in the number of pixels. This is done by multiplying the width and height of the current resolution to obtain the current number of pixels, multiplying the width and height of the candidate record resolution to obtain the candidate number of pixels, and selecting the candidate record with the smallest difference in the number of pixels as the adjacent record. When converting the two estimated processing delays according to the pixel count ratio, the ratio of the current number of pixels to the number of pixels in the adjacent record can be used as the conversion ratio, and multiplied by the first and second estimated delays in the adjacent records respectively to obtain the two estimated processing delays under the current query conditions. The conversion process can be expressed as follows: ; ; in, This is the first estimated latency obtained by converting the data for processing in a two-dimensional graphics acceleration manner; This is the second estimated latency obtained by processing using a general heterogeneous computing method; The estimated latency for processing neighboring records using a two-dimensional graphics acceleration method; The estimated latency for processing neighboring records using a general heterogeneous computing approach; The number of pixels at the current resolution; This represents the number of pixels at the nearest record resolution.

[0053] If no adjacent records with the same color sampling format and the same image processing operation type exist, the preset default processing method will be used, and the processing record of the original video frame will be marked as the default selection state.

[0054] When comparing two estimated delays, the processing method corresponding to the lower value is selected as the current processing method. This selection includes at least the channel identifier, frame number, selected processing method, estimated processing delay, and source selection. The source selection distinguishes between matching identical records, converting adjacent records, and the default selection. The buffer object, cropping region, target size, and output format of the original video frame are added to the processing queue. The cropping region can be determined by splicing layout parameters, user configuration, or the effective screen area of ​​the original video frame; the target size is determined by the width and height of the display area; the output format is determined by the input format required for layer blending. If the cropping region exceeds the boundaries, it is cropped to within the original video frame boundaries; if the target size is empty, the display area size is used as the target size; if the output format is not configured, the default output format supported by layer blending is used.

[0055] When the selected processing method is 2D graphics acceleration, the buffer object, cropping region, target size, and output format in the processing queue are executed according to the 2D graphics acceleration processing method. During 2D graphics acceleration processing, the processing region is read from the original video frame pointed to by the buffer object based on the cropping region, then scaled according to the target size, and finally converted according to the output format. If the image processing operation type only includes scaling, the cropping region can be equal to the entire area of ​​the original video frame; if it only includes format conversion, the target size can be equal to the original video frame size; if it includes both cropping and scaling, a partial frame is first obtained based on the cropping region, and then the partial frame is scaled to the target size. The output video frame after processing retains the channel identifier and frame sequence number and is written to the processed video frame queue.

[0056] When the selected processing method is general heterogeneous computing, the buffer objects in the processing queue are mapped to parallel computing inputs. Parallel computing inputs include at least the original video frame access address, cropping region, target size, output format, pixel reading rules, and output buffer handle. General heterogeneous computing can read the corresponding input pixel positions in parallel according to the output pixel position and perform sampling and conversion according to the color sampling format. For scaling, one or more input pixels can be read based on the positional mapping relationship between the output pixel coordinates and the cropping region to form the output pixel; for format conversion, pixel components in the target output format can be generated according to the color component conversion relationship. If the buffer object corresponding to the parallel computing input is inaccessible, the processing of that channel fails, a failure status is written to the video frame queue after processing, and the corresponding channel identifier is retained.

[0057] The processed video frames are written to the processed video frame queue. The processed video frame queue includes at least the channel identifier, frame number, processed frame size, output format, processing method, and processed buffer handle. If both processing methods fail, the processed video frame queue retains the failure status, allowing selection of either the processed video frame from the previous processing cycle or a preset fill frame. Through this process, each original video frame is processed according to its color sampling format, resolution, and the type of image processing operation to be performed, selecting the processing method with the lowest estimated latency, thus forming a processed video frame suitable for layer blending and composition.

[0058] S104, in response to changes in splicing layout parameters, number of video channels, or resolution of any channel, release the buffer handle in the current buffer pool, recalculate the number of buffers and buffer size according to the changed splicing layout parameters, number of video channels, and channel resolution, and reallocate the buffer pool. S104 further includes: Monitor the splicing layout parameter version, channel binding relationship, and frame size of each channel; When any monitoring result is inconsistent with the current buffer pool record, the buffer handles in the current buffer pool that are not associated with the output composite frame will be released, and the buffer handles that are still referenced by the current processing cycle will be marked as pending release. Based on the changed stitching layout parameters, the number of video channels and the channel resolution, calculate the number and size of the input buffer, the post-processing buffer and the compositing buffer; Allocate a new buffer pool.

[0059] Further: The changed splicing layout parameters are parsed into the width, height, and hierarchy of each display area; The input buffer size is calculated based on the channel resolution corresponding to each video channel, and the processed buffer size is calculated based on the width and height of the display area corresponding to the video channel. Calculate the size of the composite buffer based on the output size of the spliced ​​large screen display; The number of input buffers and post-processing buffers is determined according to the number of video channels, and the number of compositing buffers is determined according to the output buffer requirements for layer blending and compositing. The calculation results are then written to a new buffer pool.

[0060] Specifically, in this embodiment of the invention, the buffer pool is used to store handles and states of the input buffer, the post-processing buffer, and the compositing buffer. The input buffer is used to receive the original video frames identified by the file descriptor, the post-processing buffer is used to store the output processed video frames, and the compositing buffer is used to store the formed composite frames. Buffer pool reconstruction is triggered when the splicing layout parameters, the number of video channels, or the resolution of any channel change.

[0061] The splicing layout parameter version is used to determine whether the display area, display size, or hierarchy order has changed; the channel binding relationship is used to determine whether the number of video channels has been added, deleted, or the binding has changed; the frame size of each channel is used to determine whether the resolution of any channel has changed. During monitoring, the splicing layout parameter version, channel binding relationship, and frame size of each channel from the previous effective period can be saved in the current buffer pool record. Then, the corresponding data read in the current processing period is compared item by item with the current buffer pool record. If the splicing layout parameter version, channel binding relationship, and frame size of each channel are the same, the current buffer pool continues to be used; if any item is inconsistent, the subsequent steps are performed. For cases where a video stream temporarily has missing frames but the channel resolution has not changed, buffer pool reconstruction is not triggered, and missing frame boundary processing is performed instead.

[0062] When releasing buffer handles in the current buffer pool, it's necessary to distinguish between handles that are no longer referenced and those still referenced by the current processing cycle. Buffer handles unrelated to the output composite frame can be released immediately; buffer handles still referenced by the current processing cycle cannot be released immediately, otherwise it may lead to accessing invalid addresses, so they are marked as pending release. Handles pending release are released only after the read completion status, processing completion status, or the display buffer returns to the read completion status. If a buffer handle is found to be invalid during the release process, it is removed from the current buffer pool record, and the invalidation status is recorded; if release fails, the handle is kept in the pending release list, and the release attempt continues in the next processing cycle or is handled by the resource reclamation thread.

[0063] The number and size of the input buffer, post-processing buffer, and composite buffer are calculated based on the changed splicing layout parameters, the number of video channels, and the channel resolution. The size of the input buffer is related to the channel resolution and color sampling format corresponding to each video channel; the size of the post-processing buffer is related to the width, height, and output format of the display area corresponding to each video channel; and the size of the composite buffer is related to the output size of the spliced ​​large screen display and the output format of the composite frame.

[0064] For example, the calculation of the input buffer size can be expressed as: ; in, This represents the total size of the input buffer for a given channel. This represents the channel resolution width. This represents the channel resolution height. This represents the number of bytes per pixel corresponding to the color sampling format. The number of input buffers allocated to this channel.

[0065] The calculation of the size of the processed buffer can be expressed as: ; in, The total size of the buffer after processing a certain channel; This is the width of the display area corresponding to this channel; This refers to the height of the display area corresponding to this channel; The number of bytes per pixel corresponding to the target output format; The number of post-processing buffers allocated to this channel.

[0066] The calculation of the synthesis buffer size can be expressed as: ; in, This is the total size of the synthesis buffer; To output width for splicing large screen displays; To output the height for splicing large screen displays; The number of bytes per pixel corresponding to the output format of the composite frame; The number of synthesis buffers to allocate.

[0067] If the width or height of a display area is 0, no post-processing buffer will be allocated for that display area, and the corresponding channel identifier will be written to the disabled state. If the channel resolution is missing, the size of the temporary buffer will be calculated using the previous valid resolution, and will be recalculated after the valid frame size of the channel is restored.

[0068] The number of input buffers and post-processing buffers allocated is determined based on the number of video channels. For example, each video channel can be configured with at least one input buffer and at least one post-processing buffer; to support pipeline parallelism between decoding, importing, processing, and compositing, multiple input buffers and multiple post-processing buffers can be configured for each video channel. The number of compositing buffers allocated is determined based on the output buffering requirements for layer blending and compositing. For example, a foreground display buffer and a background compositing buffer can be configured so that while one compositing frame is being displayed and read, another compositing buffer is used for writing in the next processing cycle. After the allocation number and size are calculated, the calculation result is written to a new buffer pool record, which includes at least the buffer type, channel identifier, buffer size, number of buffers, buffer handle, and usage status.

[0069] When allocating a new buffer pool, first request the input buffer, post-processing buffer, and compositing buffer based on the obtained calculation results, and then write the successfully requested handles into the new buffer pool. If the allocation of any type of buffer fails, handles that are no longer referenced in the release list can be released according to priority and then re-allocated; if allocation still fails, the current processing cycle is marked as a buffer reconstruction failure, and the previous available buffer pool is kept for the running processing cycle, pausing the reception of new video frames that would cause the buffer to go out of bounds.

[0070] S105 performs layer blending and compositing on each processed video frame according to the splicing layout parameters to obtain a composite frame, and outputs the composite frame to the splicing screen.

[0071] S105 further includes: The splicing layout parameters are analyzed into the display area, display size, and image hierarchy of each processed video frame on the splicing screen. Based on the display area and display size, the corresponding processed video frames are position-mapped and size-matched to obtain the layer to be composited. Multiple layers to be composited are blended in order of image hierarchy to obtain a composite frame; When any processed video frame is missing, the processed video frame or preset fill frame from the previous processing cycle of the missing path is mapped to the corresponding display area and then participates in the transparency blending.

[0072] Further: Write the composite frame to the display buffer associated with the splicing large screen display interface; After the display buffer returns a read complete status, the display buffer reference corresponding to the composite frame is released; Release the file descriptor references, address mapping handles, and buffer objects corresponding to the original and processed video frames within this processing cycle according to the channel identifier; Return to the batch read file descriptors for the next processing cycle until a termination command is received.

[0073] Specifically, in this embodiment of the invention, a queue of processed video frames and a buffer pool are received. Each processed video frame in the queue has a channel identifier, frame number, processed frame size, output format, processing method identifier, and a processed buffer handle. The splicing layout parameters are used to determine the position and hierarchy of each processed video frame in the spliced ​​large screen.

[0074] The splicing layout parameters are parsed into display area, display size, and layer order. The display area is used to determine the coordinate range of the processed video frame to be written into the composite frame. The display size is used to confirm whether the processed video frame has met the target display size. The layer order is used to determine the blending order when multiple layers to be composited overlap.

[0075] Position mapping maps the pixel coordinates of the processed video frames to the coordinates of the spliced ​​large screen. If the size of the processed video frame matches the display size, the pixel coordinate offset relationship is directly established according to the coordinates of the upper left corner of the display area; if they do not match, supplementary size matching is performed before size matching is completed, adjusting the processed video frame to the display size before mapping. Size matching can be performed during layer blending or by calling an already used processing method. When the layers to be blended are obtained, the channel identifier, layer order, display area, layer pixel data, and transparency parameters are saved for each layer. The transparency parameter can come from the splicing layout parameters or use the default opaque state if not configured. If the display area exceeds the output size of the spliced ​​large screen, the excess part is cropped, and only the area within the output size of the spliced ​​large screen is retained for subsequent transparency blending.

[0076] The opacity blending of multiple layers to be composited is performed in hierarchical order. Opacity blending can be performed from bottom to top; first, the bottom layer is written to the composition buffer, and then the top layer is blended with the existing pixels in the composition buffer according to its opacity parameter. For example, when the opacity parameter of a certain top layer is... When the pixel values ​​are mixed, they can be represented as: ; in, These are the blended pixel values; The pixel value of the upper layer; The values ​​are the pixel values ​​already present in the lower layer; This is the transparency parameter for the upper layer, with a value ranging from 0 to 1.

[0077] when When the value is 1, the corresponding position is covered by the pixels of the upper layer; when When the value is 0, the existing pixels in the lower layer are retained.

[0078] If multiple layers to be composited do not overlap, they are written directly to the composite buffer according to their respective display areas.

[0079] When any processed video frame is missing, the processed video frame from the previous processing cycle or a preset fill frame from the missing channel is mapped to the corresponding display area and then used for transparency blending. Missing processed video frames can be caused by decoding failure, import failure, or image processing failure. If a processed video frame corresponding to the same channel identifier exists in the previous processing cycle, and the display area of ​​that processed video frame is compatible with the current splicing layout parameters, then the processed video frame from the previous processing cycle is used first. If no usable frame from the previous processing cycle exists, or if the current splicing layout parameters have caused the frame size to be mismatched, then a preset fill frame is used. The preset fill frame can be a monochrome frame, a transparent frame, or a frame containing a channel deactivation message. This boundary processing ensures that the composite frame can still form a complete output even when one input channel is abnormal, without affecting the entry of processed video frames from other channels into the composite frame.

[0080] The composite frame is written to the display buffer associated with the splicing large screen display interface. The display buffer can be part of the step composite buffer or a display buffer separately requested by the display interface. During writing, the output width, output height, output format, and buffer handle of the composite frame are submitted to the display interface. If the display interface requires a synchronization signal or refresh cycle control, the composite frame is submitted when writing is allowed in the current display cycle; if the display interface is temporarily unwritable, the composite frame is retained in the composite buffer, waiting for the next writable state. If the display buffer write fails, the reference to the display buffer requested for this write is released, the error status is written to the operation log, and the composite frame handle of the current processing cycle is retained for subsequent retry or discard processing.

[0081] Resource release is performed based on reference status. After the display buffer returns to a read-complete status, the display buffer reference corresponding to the composite frame is released; if the display buffer has not yet returned to a read-complete status, its reference is not released, and it is marked as display occupied. File descriptor references, address-mapped handles, and buffer objects corresponding to the original and processed video frames within this processing cycle are released according to channel identifiers. After a file descriptor reference is released, that file descriptor no longer participates in subsequent processing cycles; after an address-mapped handle is released, the image processing address space no longer accesses the corresponding kernel shared buffer; after a buffer object is released, the corresponding buffer descriptor entry is removed from the current processing cycle queue. If a buffer handle has been marked as pending release, it is released after confirming that its reference count is 0.

[0082] The process returns to batch read file descriptors for the next processing cycle until a termination command is received. Upon returning, it can continue to decode multiple video streams for the next processing cycle in parallel, batch import file descriptors according to the new processing cycle, select processing methods for the original video frames of the next processing cycle, determine whether to rebuild the buffer pool based on layout and channel changes, and output the next composite frame. If a termination command is received, it stops reading new file descriptors, waits for the currently displayed composite frame to return a read completion status, releases the still-referenced file descriptors, address-mapped handles, buffer objects, and buffer pool handles, and then ends the large-screen splicing process.

[0083] Another embodiment of the present invention will be described below with reference to an application scenario: The key terms used in this embodiment are explained below: MPP (Media Processing Platform) is the multimedia processing engine for embedded SoC platforms.

[0084] Embedded SoC platform: refers to a system-on-a-chip platform that integrates a central processing unit, hardware video decoder, 2D graphics acceleration unit and general heterogeneous computing unit, including but not limited to Rockchip RK series and HiSilicon Hi series.

[0085] Hardware video decoder: A dedicated video decoding circuit module integrated inside the chip, used to decode compressed video streams into raw image formats.

[0086] Two-dimensional graphics acceleration unit: A hardware module integrated inside the chip dedicated to two-dimensional graphics operations such as image scaling, rotation, format conversion, and color space transformation, including but not limited to RGA, GC, and DPU.

[0087] General-purpose heterogeneous computing units: Hardware computing units that support general-purpose parallel computing frameworks such as OpenCL and Vulkan Compute, including but not limited to computing cores in ARM Mali GPUs.

[0088] Dmabuf: A file descriptor mechanism provided by the kernel for sharing DMA buffers across devices and processes.

[0089] Zero-copy transfer: A technique in which data is transferred directly between different hardware units without being copied in memory by the CPU, through shared handles or address mappings.

[0090] Dynamic buffer management: A memory management strategy that dynamically allocates, rebuilds, and releases buffers based on the layout parameters, number of channels, and resolution information of the large-screen splicing scenario.

[0091] VPSS: Video Post-Processing Subsystem, used for preprocessing of multiple video streams such as scaling, cropping, and compositing.

[0092] This embodiment uses the Rockchip RK3588 embedded SoC platform as an example: Initialize the hardware context on the embedded SoC platform.

[0093] In this embodiment, the embedded SoC platform is Rockchip RK3588, which integrates an MPP hardware video decoder, an RGA 2D graphics acceleration unit, and an ARM Mali GPU general-purpose heterogeneous computing unit supporting OpenCL. Before starting the large-screen splicing process, the above hardware units are initialized. Initializing the hardware context includes: configuring the MPP decoding capability parameters to support H.264 and H.265 encoding formats and NV12 and YUYV output formats; configuring the RGA working mode to support scaling, cropping, and format conversion operations; and configuring the OpenCL runtime environment, including loading the OpenCL runtime library, creating a command queue, and compiling the kernel program. After initialization, each hardware unit is in a schedulable ready state.

[0094] The system receives multiple video streams to be spliced ​​and their splicing layout parameters, binds each video stream to its corresponding processing channel, performs hardware parallel decoding, obtains the original video frames, and stores them in a kernel shared buffer identified by a file descriptor.

[0095] In this embodiment, four network camera video streams with a resolution of 3840×2160, a frame rate of 30fps, an H.265 encoding format, and an NV12 color sampling format are accessed. The splicing layout parameters are a 2×2 four-screen uniformly divided layout, with each display area having a resolution of 1920×1080. The four video streams are each bound to one of four VPSS processing channels, and each channel is decoded in parallel by an MPP hardware video decoder. The decoded raw video frames are written to a kernel shared buffer identified by a dmabuf file descriptor.

[0096] Each file descriptor is read in batches, an address mapping is established for the kernel shared buffer, and based on the address mapping, it is encapsulated into a buffer object that can be accessed by image scaling, cropping or compositing operations.

[0097] In this embodiment, within the same processing cycle, the file descriptors corresponding to the four decoded frames are imported in batches using the dmabuf mechanism. This establishes an address mapping between the kernel shared buffer and the image processing address space, retains the mapping handles, and encapsulates these handles into a set of buffer objects. This process involves no data copying between the CPU and GPU, achieving zero-copy transmission of multiple video frames.

[0098] For each original video frame, the color sampling format, resolution, and type of image processing operation to be performed are read. From the pre-stored corresponding records indexed by the format, resolution, and operation type, the first estimated latency for processing in a two-dimensional graphics acceleration mode and the second estimated latency for processing in a general heterogeneous computing mode are obtained respectively. The first estimated latency and the second estimated latency are compared, and the processing mode with the lower estimated latency is selected. The original video frame is then scaled, cropped, or converted in format to obtain the processed video frame.

[0099] In this embodiment, the color sampling format of all four video frames is NV12, and the resolution is 3840×2160. The image processing operation to be performed is scaling, scaling from 3840×2160 to 1920×1080. At the same time, the NV12 format needs to be converted to the ARGB8888 format required for layer blending and composition. That is, the operation type is scaling plus format conversion.

[0100] The query conditions—NV12, 3840×2160, scaling + format conversion—are entered and searched in the pre-stored corresponding records. The first estimated latency for processing this query condition using 2D graphics acceleration (RGA) is 22ms, and the second estimated latency using general heterogeneous computing (OpenCL) is 15ms. After comparison, the estimated latency of OpenCL is approximately 32% lower than that of RGA; therefore, general heterogeneous computing is chosen. The buffer object, cropped region (full frame), target size (1920×1080), and output format (ARGB8888) are added to the pending operation queue.

[0101] After scaling and format conversion are performed by OpenCL in parallel computing, the four original video frames are all processed into 1920×1080 ARGB8888 format processed video frames and written to the processed video frame queue.

[0102] In response to changes in splicing layout parameters, number of video channels, or resolution of any channel, release the buffer handles in the current buffer pool, recalculate the number and size of buffers according to the changed splicing layout parameters, number of video channels, and channel resolution, and reallocate the buffer pool.

[0103] In this embodiment, if the user switches the splicing layout from a 2×2 four-screen layout to a 1+3 picture-in-picture layout during operation, with one main screen occupying a 1920×2160 area on the left and three auxiliary screens arranged vertically on the right, then the buffer pool is rebuilt. The buffer handles corresponding to the original layout are released, and the buffer sizes after processing each channel are recalculated according to the new layout. The main screen channel is 1920×2160, and the auxiliary screen channels are 1920×720. The buffer pool is then reallocated.

[0104] According to the splicing layout parameters, the processed video frames of each channel are layered and composited to obtain a composite frame, which is then output to the splicing screen.

[0105] In this embodiment, each channel is mapped to a new layout and its transparency is mixed before the composite frame is output to the splicing screen.

[0106] For the purpose of simplicity, the method steps disclosed in the above embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0107] like Figure 2 As shown, the present invention also provides a large-screen splicing system, comprising: The video access decoding module 201 is configured to access multiple video streams to be spliced ​​and splicing layout parameters, bind each video stream to the corresponding processing channel, perform hardware parallel decoding, obtain the original video frames, and store them in the kernel shared buffer identified by the file descriptor. The buffer import module 202 is configured to read each file descriptor in batches, establish an address mapping for the kernel shared buffer, and encapsulate it into a buffer object that can be accessed by image scaling, cropping or compositing operations based on the address mapping. The accelerated selection processing module 203 is configured to read the color sampling format, resolution, and image processing operation type for each original video frame, obtain the first estimated delay for processing in a two-dimensional graphics acceleration mode and the second estimated delay for processing in a general heterogeneous computing mode from the pre-stored corresponding records indexed by the format, resolution, and operation type, compare the first estimated delay and the second estimated delay, select the processing mode with the lower estimated delay, and perform scaling, cropping, or format conversion on the original video frame to obtain the processed video frame. The buffer pool reconstruction module 204 is configured to release the buffer handles in the current buffer pool in response to changes in the splicing layout parameters, the number of video channels, or the resolution of any channel, and recalculate the number and size of the buffers according to the changed splicing layout parameters, the number of video channels, and the channel resolution, and reallocate the buffer pool. The composite output module 205 is configured to perform layer blending and compositing on each processed video frame according to the splicing layout parameters to obtain a composite frame, and output the composite frame to the splicing screen.

[0108] It is worth noting that although only some basic functional modules are disclosed in the embodiments of this invention, it does not mean that the composition of this system is limited to the above-mentioned basic functional modules. On the contrary, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technology to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. The fact that this embodiment only discloses a few basic functional modules should not be considered as the scope of protection of this invention being limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above devices are described separately according to their functions as various units and modules. Of course, in implementing this invention, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for splicing large screens, characterized in that, include: The system receives multiple video streams to be spliced ​​and splicing layout parameters, binds each video stream to its corresponding processing channel, performs hardware parallel decoding, obtains the original video frames, and stores them in the kernel shared buffer identified by the file descriptor. Batch read each file descriptor, establish an address mapping for the kernel shared buffer, and based on the address mapping, encapsulate it into a buffer object that can be accessed by image scaling, cropping or compositing operations; For each original video frame, the color sampling format, resolution, and image processing operation type to be performed are read. From the pre-stored corresponding records indexed by the format, resolution, and operation type, the first estimated delay for processing in a two-dimensional graphics acceleration mode and the second estimated delay for processing in a general heterogeneous computing mode are obtained respectively. The first estimated delay and the second estimated delay are compared, and the processing mode with the lower estimated delay is selected. The original video frame is then scaled, cropped, or converted in format to obtain the processed video frame. When the splicing layout parameters, the number of video channels, or the resolution of any channel change, the buffer handles in the current buffer pool are released, the number of buffers and the buffer size are recalculated according to the changed splicing layout parameters, the number of video channels, and the channel resolution, and the buffer pool is reallocated. According to the splicing layout parameters, the processed video frames of each channel are layered and composited to obtain a composite frame, which is then output to the splicing screen.

2. The large-screen splicing method according to claim 1, characterized in that, The system receives multiple video streams to be spliced ​​and their splicing layout parameters, binds each video stream to its corresponding processing channel, performs hardware parallel decoding to obtain the original video frames, and stores them in a kernel shared buffer identified by a file descriptor. Further steps include: Read the input interface type, channel identifier, and display area from the splicing layout parameters for each video stream; Establish a channel binding relationship based on the channel identifier and the display area, and write each video stream into the processing channel corresponding to the channel binding relationship; The video streams in each processing channel are decoded in hardware in parallel, and the decoded raw video frames are written to the kernel shared buffer. Generate a file descriptor for each kernel shared buffer and associate the file descriptor with the channel identifier, frame size, and color sampling format.

3. The large-screen splicing method according to claim 1, characterized in that, Batch reading of each file descriptor, establishing an address mapping for the kernel shared buffer, and based on the address mapping, encapsulating it into a buffer object accessible for image scaling, cropping, or compositing operations, further including: Organize multiple file descriptors within the same processing cycle into an import list according to the channel identifier; Verify the access status and frame size of the kernel shared buffer corresponding to each file descriptor in the import list; For file descriptors that pass verification, establish an address mapping between the kernel shared buffer and the image processing address space, and retain the mapping handle; Write the mapping handle, channel identifier, frame size, and color sampling format into the buffer descriptor, and combine multiple buffer descriptors into a set of buffer objects.

4. The large-screen splicing method according to claim 1, characterized in that, From the pre-stored corresponding records indexed by the aforementioned format, resolution, and operation type, a first estimated latency for processing using a 2D graphics acceleration method and a second estimated latency for processing using a general heterogeneous computing method are obtained respectively. The first estimated latency and the second estimated latency are compared, and the processing method with the lower estimated latency is selected. This further includes: Combine color sampling format, resolution, and image processing operation type into query conditions; Match the records that are identical to the query conditions in the pre-stored corresponding records, and read the first estimated delay and the second estimated delay; When no matching record is found, the first estimated delay and the second estimated delay are calculated by using the record with the same color sampling format and the same image processing operation type that is adjacent to the resolution, according to the ratio of the number of pixels of the resolution to the resolution of the adjacent record. Compare the first estimated delay and the second estimated delay, select the processing method corresponding to the lower one as the current processing method, and add the buffer object, clipping region, target size and output format to the pending operation queue.

5. A large-screen splicing method according to claim 4, characterized in that, Selecting a processing method with lower estimated latency, scaling, cropping, or format conversion is performed on the original video frame to obtain a processed video frame, further including: When the selected processing method is 2D graphics acceleration, the buffer objects, clipping regions, target sizes and output formats in the processing queue will be clipped, scaled or formatted according to the 2D graphics acceleration processing method. When the selected processing method is general heterogeneous computing, the buffer objects in the queue of operations to be processed are mapped to parallel computing inputs, and parallel computing is performed according to the clipping region, target size and output format to complete clipping, scaling or format conversion; The processed video frames are written to the processed video frame queue.

6. The large-screen splicing method according to claim 1, characterized in that, In response to changes in stitching layout parameters, number of video channels, or resolution of any channel, the buffer handles in the current buffer pool are released. The number and size of buffers are recalculated according to the changed stitching layout parameters, number of video channels, and channel resolution, and the buffer pool is reallocated. Further steps include: Monitor the splicing layout parameter version, channel binding relationship, and frame size of each channel; When any monitoring result is inconsistent with the current buffer pool record, the buffer handles in the current buffer pool that are not associated with the output composite frame will be released, and the buffer handles that are still referenced by the current processing cycle will be marked as pending release. Based on the changed stitching layout parameters, the number of video channels and the channel resolution, calculate the number and size of the input buffer, the post-processing buffer and the compositing buffer; Allocate a new buffer pool.

7. The large-screen splicing method according to claim 1, characterized in that, The processed video frames from each stream are layer-blended and composited according to the splicing layout parameters to obtain a composite frame, which further includes: The splicing layout parameters are analyzed into the display area, display size, and image hierarchy of each processed video frame on the splicing screen. Based on the display area and display size, the corresponding processed video frames are position-mapped and size-matched to obtain the layer to be composited. Multiple layers to be composited are blended in order of image hierarchy to obtain a composite frame; When any processed video frame is missing, the processed video frame or preset fill frame from the previous processing cycle of the missing path is mapped to the corresponding display area and then participates in the transparency blending.

8. The large-screen splicing method according to claim 1, characterized in that, Outputting the composite frame to the video wall further includes: Write the composite frame to the display buffer associated with the splicing large screen display interface; After the display buffer returns a read complete status, the display buffer reference corresponding to the composite frame is released; Release the file descriptor references, address mapping handles, and buffer objects corresponding to the original and processed video frames within this processing cycle according to the channel identifier; Return to the batch read file descriptors for the next processing cycle until a termination command is received.

9. A large-screen splicing method according to claim 6, characterized in that, Based on the changed stitching layout parameters, the number of video channels, and the channel resolution, the number and size of the input buffer, the post-processing buffer, and the compositing buffer are calculated, further including: The changed splicing layout parameters are parsed into the width, height, and hierarchy of each display area; The input buffer size is calculated based on the channel resolution corresponding to each video channel, and the processed buffer size is calculated based on the width and height of the display area corresponding to the video channel. Calculate the size of the composite buffer based on the output size of the spliced ​​large screen display; The number of input buffers and post-processing buffers is determined according to the number of video channels, and the number of compositing buffers is determined according to the output buffer requirements for layer blending and compositing. The calculation results are then written to a new buffer pool.

10. A large-screen splicing system, characterized in that, include: The video access decoding module is configured to access multiple video streams to be spliced ​​and splicing layout parameters, bind each video stream to the corresponding processing channel, perform hardware parallel decoding, obtain the original video frames, and store them in the kernel shared buffer identified by the file descriptor; The buffer import module is configured to read each file descriptor in batches, establish an address mapping for the kernel shared buffer, and encapsulate it into a buffer object that can be accessed by image scaling, cropping or compositing operations based on the address mapping. The accelerated selection processing module is configured to read the color sampling format, resolution, and image processing operation type for each original video frame, obtain the first estimated delay for processing in a two-dimensional graphics acceleration mode and the second estimated delay for processing in a general heterogeneous computing mode from the pre-stored corresponding records indexed by the format, resolution, and operation type, compare the first estimated delay and the second estimated delay, select the processing mode with the lower estimated delay, and perform scaling, cropping, or format conversion on the original video frame to obtain the processed video frame. The buffer pool reconstruction module is configured to release the buffer handles in the current buffer pool in response to changes in the splicing layout parameters, the number of video channels, or the resolution of any channel, and recalculate the number and size of buffers according to the changed splicing layout parameters, the number of video channels, and the channel resolution, and reallocate the buffer pool. The composite output module is configured to perform layer blending and compositing on each processed video frame according to the splicing layout parameters to obtain a composite frame, which is then output to the splicing screen.

Images