Multi-split rendering method and apparatus

Abstract

The application relates to a multi-split screen rendering method and device. The method comprises the following steps: acquiring an input view and an output view of a video processor; determining at least one source rectangular region used for constituting a multi-split screen picture and a target rectangular region corresponding to each source rectangular region on a target surface; for each source rectangular region, acquiring pixel data in the source rectangular region from an original texture surface according to the input view, and performing hardware scaling and color space conversion on the pixel data in the source rectangular region to obtain a processing result of the source rectangular region; and writing the processing result of each source rectangular region into the corresponding target rectangular region according to the output view, so as to synthesize the multi-split screen picture on the target surface.

Description

Technical Field

[0001] This application relates to the field of video image processing technology, and in particular to a multi-screen rendering method and apparatus. Background Technology

[0002] In applications such as video surveillance, video conferencing, and streaming media playback, it is often necessary to simultaneously display a complete panoramic view and several zoomed-in areas of interest from a high-resolution video source within the same display window. There are three main technical approaches to achieving this display requirement: First, software decoding via the CPU (Central Processing Unit), followed by CPU-based cropping, scaling, and color space conversion, before copying and uploading the entire frame's RGB data to the GPU (Graphics Processing Unit) for display. Second, hardware decoding using the GPU, reading the decoded texture data back to system memory, and then re-uploading it to the GPU as a regular texture, which is then sampled and synthesized multiple times using pixel shaders. Third, starting a separate decoder instance for each region of interest, obtaining images of each region through multiple hardware decodings, and then stitching them together.

[0003] None of the above solutions can render different regions of the same frame onto the corresponding positions on the screen without leaving the GPU memory. Instead, they require CPU processing, copying or reading back the hardware-decoded textures before processing, or decoding each region separately. Without exception, these methods introduce additional data copying and round-trip transfers between GPU memory and system memory, resulting in increased overall system processing latency and making it difficult to meet the real-time requirements of high-resolution scenes. Summary of the Invention

[0004] Therefore, it is necessary to provide a multi-screen rendering method and device that can render different areas of the same frame onto different positions on the screen without leaving the GPU memory, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a multi-screen rendering method applied to a video processor, comprising:

[0006] Obtain the input view and output view of the video processor; wherein, the input view is constructed based on the original texture surface of the decoded video image stored in the graphics processor's video memory, and is used to indicate the storage location of the pixel data to be processed in the original texture surface; the output view is constructed based on the target surface of the multi-screen image to be rendered allocated in the graphics processor's video memory, and is used to indicate the writing location of the processing result in the target surface.

[0007] Determine at least one source rectangular region for constituting the multi-screen display, and the target rectangular region corresponding to each of the source rectangular regions on the target surface.

[0008] Without releasing or rebuilding the input view, for each source rectangular region, pixel data in the source rectangular region is obtained from the original texture surface according to the input view, and hardware scaling and color space conversion are performed on the pixel data in the source rectangular region to obtain the processing result of the source rectangular region;

[0009] Based on the output view, the processing results of each of the source rectangular regions are written into the corresponding target rectangular region to synthesize the multi-screen image on the target surface.

[0010] Secondly, this application also provides a multi-screen rendering apparatus, comprising:

[0011] An acquisition module is used to acquire an input view and an output view; wherein, the input view is constructed based on the original texture surface of the decoded video image stored in the graphics processor's video memory, and is used to indicate the storage location of the pixel data to be processed in the original texture surface; the output view is constructed based on the target surface of the multi-screen image to be rendered allocated in the graphics processor's video memory, and is used to indicate the writing location of the processing result in the target surface.

[0012] The determining module is used to determine at least one source rectangular region for constituting the multi-screen image, and the target rectangular region corresponding to each of the source rectangular regions on the target surface.

[0013] The processing module is used to obtain pixel data from the original texture surface of each source rectangular region according to the input view without releasing or rebuilding the input view, and to perform hardware scaling and color space conversion on the pixel data in the source rectangular region to obtain the processing result of the source rectangular region.

[0014] The writing module is used to write the processing results of each of the source rectangular regions into the corresponding target rectangular regions according to the output view, so as to synthesize the multi-screen image on the target surface.

[0015] The aforementioned multi-screen rendering method and apparatus have decoded video images that remain on the original texture surface of the GPU memory. Based on the same input view, multiple source rectangular regions are subjected to hardware scaling and color space conversion, and then directly written to their respective target rectangular regions on the target surface. There is no need to read the data back to the system memory. Thus, different regions of the same frame can be rendered to their corresponding positions on the screen without leaving the GPU memory. At the same time, it reduces the additional overhead caused by round-trip copying from the GPU memory to the main memory, texture copying, and multiple sampling of the shader, thus meeting the real-time rendering requirements of high-resolution scenes. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application environment diagram of a multi-screen rendering method in one embodiment;

[0018] Figure 2 This is a flowchart illustrating a multi-screen rendering method in one embodiment;

[0019] Figure 3 This is a schematic diagram of the processing flow of pixel data in the source rectangular region in one embodiment;

[0020] Figure 4 This is a schematic diagram of the pixel data capture process in one embodiment;

[0021] Figure 5 This is a structural block diagram of a multi-screen rendering device in one embodiment;

[0022] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0025] The multi-screen rendering method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown can be a terminal 102 with video decoding and graphics processing capabilities. Terminal 102 can communicate with server 104 via a network to obtain video streams from server 104; alternatively, terminal 102 can read video files from local storage media for decoding and rendering. The multi-screen rendering method provided in this embodiment is executed by a video processor in terminal 102. The video processor first obtains its required input and output views. The input view points to the original texture surface where the decoded video image is located in the GPU memory; the output view points to the target surface allocated for the multi-screen display in the GPU memory. Further, the video processor determines each source rectangular region required to constitute the multi-screen display, and the target rectangular region corresponding to each source rectangular region on the target surface. For each source rectangular region, the video processor extracts the pixel data of that region from the original texture surface according to the input view, performs image processing, and then writes the processing result into the corresponding target rectangular region according to the output view. Terminal 102 can house the aforementioned video processor. Terminal 102 can be, but is not limited to, a network video recorder (NVR), video decoder, video conferencing terminal, personal computer, laptop, smartphone, tablet, drone, low-altitude aircraft, IoT device (such as smart TV, projector, smart vehicle device), or portable wearable device (such as smartwatch, smart bracelet, head-mounted device, including virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses), or any other device with video playback and display capabilities. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, used to provide video streams or store video data to terminal 102.

[0026] In one exemplary embodiment, such as Figure 2 As shown, a multi-screen rendering method is provided, which is applied to... Figure 1 The following steps, 201 to 204, are used as an example of a video processor in a computer.

[0027] Step 201: Obtain the input view and the output view; wherein, the input view is constructed based on the original texture surface of the decoded video image stored in the graphics processor's video memory, and is used to indicate the storage location of the pixel data to be processed in the original texture surface; the output view is constructed based on the target surface of the multi-screen image to be rendered allocated in the graphics processor's video memory, and is used to indicate the writing location of the processing result in the target surface.

[0028] In this context, the raw texture surface refers to a two-dimensional buffer in GPU memory that stores decoded video pixel data. The raw texture surface is output by the hardware decoder and resides in GPU memory. The input view is an access handle or interface pointing to this raw texture surface, used to describe to the video processor where to read data. The target surface refers to a buffer allocated in GPU memory used to hold the final multi-screen display. The output view is an access handle pointing to this target surface, used to describe where the processing results will be written.

[0029] After the hardware decoder completes decoding of a frame (the current decoding frame) of the decoded video image and stores the pixel data to be processed into the original texture surface, the video processor obtains the resource pointer of the original texture surface and calls the interface for creating the input view, passing in the texture pointer and the corresponding view description information, thereby obtaining the input view. Further, the video processor calls the interface for creating the output view based on the target surface to obtain the output view.

[0030] In other embodiments, the output view may be created once during rendering session initialization and reused in subsequent frame processing, while the input view is updated as decoded frames are updated. However, regardless of the creation method, the construction of the input and output views does not involve copying or transferring the pixel data to be processed.

[0031] Step 202: Determine at least one source rectangular region for constituting the multi-screen display, and the target rectangular region corresponding to each source rectangular region on the target surface.

[0032] The source rectangle region refers to a rectangular area defined on the original texture surface. This rectangle is defined by the coordinates of its top-left corner and width and height (or equivalently, the coordinates of its top-left and bottom-right corners) in pixel coordinates. The source rectangle region indicates where in the original image the data is extracted for this processing. The target rectangle region refers to a rectangular area defined on the target surface, also defined by pixel coordinates. Each source rectangle region uniquely corresponds to one target rectangle region, and together they form a source-target mapping relationship, used to instruct the video processor where to place the processed pixel data on the target surface.

[0033] The video processor calculates and generates the above mapping relationship based on the currently selected display mode and multi-screen layout configuration of the electronic pan-tilt unit. Specifically, if the display mode is a single-screen mode, then based on the multi-screen layout configuration, a source rectangular area and its corresponding target rectangular area (calculated as the center area according to the proportional adaptive parameters) are determined. If the current mode is a panoramic plus partial zoom mode, then based on the multi-screen layout configuration, two source rectangular areas (corresponding to the global area and the user-selected area of ​​interest, respectively) and two corresponding target rectangular areas (corresponding to the global display position and the partial zoom display position, respectively) are determined.

[0034] Step 203: Without releasing or rebuilding the input view, for each source rectangular region, obtain the pixel data in the source rectangular region from the original texture surface according to the input view, and perform hardware scaling and color space conversion on the pixel data in the source rectangular region to obtain the processing result of the source rectangular region.

[0035] Hardware scaling and color space conversion refer to the operations performed by the video processor on pixel data within a source rectangular region. Specifically, color space conversion involves converting the pixel data of the original textured surface into pixel data of the format required by the target surface; image scaling refers to enlarging or reducing the image to fit the size of the target rectangle when the size of the source rectangular region differs from that of the target rectangular region.

[0036] First, for each source rectangle region, the video processor calls the source rectangle region setting interface to set the rectangle parameters of the source rectangle region to the coordinates of the source rectangle region, in order to determine which specific range of the original texture surface pointed to by the input view should be read pixel data.

[0037] Secondly, the video processor calls the target rectangle setting interface to set the rectangle parameters of the source rectangle region to the coordinates of the corresponding target rectangle region, so as to determine which specific range in the target surface pointed to by the output view should be written to the processed data.

[0038] Subsequently, the video processor reads the pixel data at the corresponding position from the original texture surface through the input view according to the set source rectangle parameters. Then, the color space conversion module inside the video processor converts the format of the pixel data, while the internal scaling module enlarges or reduces the size according to the size ratio of the source rectangle area and the target rectangle area. Finally, the pixel data after the above image processing is temporarily stored in the output buffer of the video processor as the processing result of the source rectangle area.

[0039] In some embodiments, when the video processor begins pipelined processing of multiple source rectangular regions, it avoids the significant overhead of recreating resource views for each region of interest in traditional solutions by reusing the already created input view. Specifically, before executing the first video processing call, the video processor creates an input view pointing to the current decoded frame via a graphics interface. After processing the first source rectangular region (e.g., a panoramic region) and completing its corresponding hardware scaling, color space conversion, and writing operations, the video processor does not release the underlying resource pointer of the input view, nor does it re-call the view creation interface. Instead, it directly modifies the source rectangle and target rectangle attribute parameters in the video processor's stream state. After modification, the video processor immediately reuses the existing input view handle, triggering the next video processing call to process the next source rectangular region (e.g., a zoomed-in region).

[0040] Throughout the entire processing flow, for N source rectangular regions, the video processor performs only one input view creation operation and N rectangle parameter updates and video processing calls. Since the creation of the input view involves resource binding and memory page table mapping at the driver layer, it incurs significant overhead, while modifying the rectangle parameters only involves register-level numerical updates, resulting in less overhead. Through this one-time creation and multiple reuse mechanism, this embodiment reduces the number of interactions with the graphics driver layer when rendering multi-screen images, thereby reducing CPU utilization and rendering latency, making it particularly suitable for real-time monitoring scenarios that simultaneously display multiple magnified local areas.

[0041] Step 204: Based on the output view, write the processing results of each source rectangular area into the corresponding target rectangular area to synthesize a multi-screen image on the target surface.

[0042] In some embodiments, image processing and writing operations are performed synchronously by the same video processor call. After performing color space conversion and scaling, the video processor's output is directly connected to the video memory address space of the target surface; based on the rectangle parameters of the target rectangular region, the video processor directly stores the processing results in the output buffer into the corresponding location on the target surface pointed to by the output view. During this process, pixel data is transferred entirely on the GPU's internal bus without passing through the CPU or system memory.

[0043] In one embodiment, before performing hardware scaling and color space conversion on the first source rectangular region, the video processor first performs a screen clearing operation on the target surface. For example, the video processor calls a screen clearing function of the graphics interface to fill all pixels on the target surface pointed to by the output view with black. The purpose of the screen clearing operation is to provide a clean background for writing subsequent processing results. After the video processor subsequently writes the processing results of each source rectangular region to its corresponding target rectangular region, areas on the target surface that have not been written (such as black borders at the top, bottom, left, or right due to scaling) will remain black, thus forming a correct multi-screen visual effect. Once all source rectangular regions have been processed and written, a complete multi-screen image composed of multiple sub-images is synthesized on the target surface.

[0044] In this embodiment, the decoded video image always resides on the original texture surface of the GPU memory. Based on the same input view, multiple source rectangular regions are subjected to hardware scaling and color space conversion respectively, and then directly written to their respective target rectangular regions on the target surface. There is no need to read the data back to the system memory. Thus, different regions of the same frame can be rendered to their corresponding positions on the screen without leaving the GPU memory. At the same time, the additional overhead caused by round-trip copying from the GPU memory to the main memory, texture copying, and multiple sampling of the shader is reduced, which meets the real-time rendering requirements of high-resolution scenes.

[0045] Figure 3 This is a schematic diagram illustrating the processing flow of pixel data in the source rectangular region in one embodiment. Please refer to [link / reference]. Figure 3 In one exemplary embodiment, for each source rectangular region, pixel data within the source rectangular region is obtained from the original texture surface according to the input view, and hardware scaling and color space conversion are performed on the pixel data within the source rectangular region to obtain the processing result of the source rectangular region, including:

[0046] Step 301: For each source rectangular region, determine the rectangle parameters of the source rectangular region in the original texture surface.

[0047] In this context, the rectangle parameters refer to a set of values ​​that uniquely describe the position and size of a rectangular region in a two-dimensional pixel coordinate system. These values ​​include at least the coordinates of the top-left corner of the rectangle in both the horizontal and vertical directions, as well as its width and height in the same directions. The rectangle parameters can be expressed as top-left corner coordinates and width / height, or top-left corner coordinates and bottom-right corner coordinates.

[0048] In some embodiments, before processing each source rectangular region, the video processor first reads the rectangle parameters of the source rectangular region from a parameter buffer in an internal register or memory. For example, the video processor sequentially reads four sets of coordinate values ​​from the parameter buffer: the x-coordinate of the top-left corner of the source rectangle, the y-coordinate of the top-left corner of the source rectangle, the x-coordinate of the bottom-right corner of the source rectangle, and the y-coordinate of the bottom-right corner of the source rectangle. These four sets of values ​​collectively define the position and size of the source rectangular region on the original texture surface.

[0049] Step 302: Based on the input view and rectangle parameters, extract the pixel data corresponding to the source rectangle region from the original texture surface to obtain the extracted pixel data.

[0050] In some embodiments, the video processor accesses the original texture surface via an input view in response to the loaded source rectangle parameters. The internal read control logic of the video processor calculates the starting storage address of the data to be extracted in the original texture surface and the length of each row of data based on the source rectangle parameters. Starting from this starting address, the video processor reads pixel data row by row until all rows covered by the rectangle height have been read. During this process, the video processor only reads pixel data within the range defined by the rectangle parameters; pixels outside the rectangle's range are not read.

[0051] It should be noted that when the video processor performs the cropping operation, it does not write the cropped data to system memory, nor does it allocate an additional buffer in the GPU memory to store the cropping result. The cropped data only flows within the video processor in a pipeline manner, entering the subsequent scaling and color space conversion stages, in order to reduce the number of off-chip memory accesses and reduce bandwidth consumption.

[0052] Step 303: Perform hardware scaling and color space conversion on the captured pixel data to obtain the processing result of the source rectangular region.

[0053] As mentioned earlier, hardware scaling refers to the process by which the scaling module inside the video processor hardware transforms the size of the captured pixel data from the source rectangular region to the target rectangular region. When the width and height of the source rectangular region are inconsistent with the width and height of the target rectangular region, the scaling module resamples the pixels using a multi-tap filter to enlarge or reduce the image. Color space conversion refers to the process by which the color conversion module inside the video processor hardware maps pixels from one color space to another. Both hardware scaling and color space conversion are performed by dedicated circuits inside the video processor hardware and do not involve pixel-level calculations by the CPU.

[0054] In some embodiments, the video processor sequentially sends the captured pixel data to a hardware scaling module and a color space conversion module. The hardware scaling module first reads the source rectangle size and the corresponding target rectangle size of the captured pixel data. The scaling module compares the proportional relationship between the source and target sizes. If the source width is greater than the target width, a scaling down operation is performed; if the source width is less than the target width, a scaling up operation is performed. Similarly, the same judgment and operation are performed in the height direction. The filter inside the scaling module selects the optimal filter coefficients according to the scaling ratio and performs horizontal and vertical resampling on the captured pixel data to generate scaled pixel data that conforms to the target rectangle size. For scaling up, the scaling module uses an interpolation algorithm to generate new pixel values ​​between adjacent pixels; for scaling down, the scaling module uses a downsampling algorithm to merge multiple pixels into one pixel.

[0055] After scaling, the video processor sends the scaled pixel data to the color space conversion module. The color space conversion module selects an appropriate conversion matrix based on the input and output pixel formats. In one example, for the conversion from a luminance-chrominance separated color encoding method to RGB (Red-Green-Blue), the color space conversion module performs matrix multiplication on the luminance and chrominance components of each pixel to generate the corresponding red, green, and blue component values, and assembles them into complete RGB pixels according to the target format. The converted pixel data is arranged in rows according to the width of the target rectangular area, resulting in two-dimensional pixel data.

[0056] The two-dimensional pixel data obtained after hardware scaling and color space conversion of the pixel data extracted from the source rectangular region is the processing result of the source rectangular region. The video processor then determines the rectangle parameters of the next source rectangular region on the original texture surface, continuing this process until all source rectangular regions have been processed.

[0057] In this embodiment, the video processor sequentially performs rectangle parameter determination, pixel data cropping, hardware scaling, and color space conversion for each source rectangular region. All operations are completed in a pipeline manner within the video processor hardware. The cropped pixel data does not leave the video processor's internal cache and does not need to be written to GPU memory or system memory, reducing off-chip storage access and lowering bandwidth consumption and latency.

[0058] Figure 4 For a schematic diagram of the pixel data capture process in one embodiment, please refer to [link / reference]. Figure 4 In one exemplary embodiment, the rectangle parameters include position information and size information.

[0059] The location information refers to the coordinates of the top-left corner or any reference point of the source rectangular region in the pixel coordinate system corresponding to the original texture surface, used to determine the starting position of the rectangular region on the original texture surface. The location information includes at least horizontal and vertical coordinate values. The size information refers to the number of pixels covered by the source rectangular region in the horizontal and vertical directions, including at least width and height values, used to determine the size of the rectangular region.

[0060] The pixel data corresponding to the source rectangular region is extracted from the original texture surface to obtain the extracted pixel data, including:

[0061] Step 401: Determine the cut-off range in the original textured surface based on the location and size information.

[0062] In some embodiments, after obtaining the rectangular parameters of the source rectangular region, the video processor parses the position and size information from the rectangular parameters. Based on the position and size information, the video processor determines the cropping range within the original texture surface. Specifically, the video processor uses the upper left corner of the original texture surface as the origin (0, 0), with the horizontal direction to the right as the positive x-axis and the vertical direction downwards as the positive y-axis. The video processor uses the x-coordinate (X) and y-coordinate (Y) as the starting points of the cropping range, and the width (W) and height (H) as the extension lengths of the cropping range in the horizontal and vertical directions, respectively. Thus, the video processor determines the upper left corner coordinates of the cropping range as (X, Y), and the lower right corner coordinates as (X+W-1, Y+H-1), or equivalently, the cropping range is a rectangular pixel region defined by the x-coordinate from X to X+W and the y-coordinate from Y to Y+H.

[0063] Step 402: Extract pixel data corresponding to the extraction range from the original texture surface to obtain the extracted pixel data.

[0064] After determining the cropping range, the video processor accesses the original texture surface through the input view and reads the pixel data corresponding to the cropping range from the original texture surface. The input view holds a resource reference to the original texture surface, and the video processor obtains attribute information such as the storage address, pixel format, and line stride (i.e., the number of bytes occupied by each row of pixels in the image) of the original texture surface through the input view. The video processor calculates the actual storage location to be read based on the starting address and line stride of the cropping range. Starting from the storage address corresponding to the top left corner (X, Y), the video processor reads the pixel data of each row sequentially. For each row, starting from the storage position corresponding to the starting horizontal coordinate X, the video processor continuously reads the amount of pixel data corresponding to the width W until the data of that row is completely read. Further, the video processor moves the reading position to the starting address of the next row (i.e., the starting address of the current row plus the line stride) and continues to read the data of the next row. The above process is repeated until the number of rows read reaches the number of rows corresponding to the height H, thus obtaining the cropped pixel data.

[0065] In this embodiment, the video processor defines the cropping range in the original texture surface based on the position and size information in the rectangle parameters, and reads the pixel data within the cropping range line by line from the original texture surface. The CPU does not need to participate in any pixel-level operations. The cropped pixel data flows within the video processor without going through off-chip storage, which reduces bandwidth consumption and access latency.

[0066] In one exemplary embodiment, the processing results of each source rectangular region are written into the corresponding target rectangular region according to the output view, including: determining the rectangular parameters of the target rectangular region corresponding to the source rectangular region on the target surface for the processing results of each source rectangular region; and writing the processing results of the source rectangular regions into the position corresponding to the target rectangular region on the target surface according to the rectangular parameters of the target rectangular region and the output view.

[0067] Before writing the processing result of each source rectangle region, the video processor first reads the rectangle parameters of the target rectangle region corresponding to the source rectangle region from the parameter buffer in the internal register or memory. For example, the video processor reads the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the width, and the height of the target rectangle from the parameter buffer. These four sets of values ​​together define the precise position and size of the current target rectangle region in the pixel coordinate system of the target surface. The pixel coordinate system of the target surface has its origin (0, 0) at the top-left corner of the target surface, with the positive x-axis pointing to the right and the positive y-axis pointing downwards. The video processor loads the read target rectangle parameters into the target rectangle parameter register inside the video processor for use in subsequent write operations.

[0068] After determining the rectangular parameters of the target rectangular region, the video processor accesses the target surface through the output view. The output view holds a resource reference pointing to the target surface, and the video processor obtains attribute information such as the target surface's storage address, pixel format, and line stride through the output view.

[0069] The video processor calculates the write start address on the target surface based on the rectangular parameters of the target rectangular region. For a pixel with coordinates (x, y) on the target surface, its storage address is calculated as: target surface base address + y × target surface row stride + x × number of bytes per pixel. The video processor substitutes the coordinates (X, Y) of the top-left corner of the target rectangle into the above formula to calculate the write start address.

[0070] The processing results for each source rectangular region of the video processor are read from its internal output buffer and written to the target surface line by line, starting from the write start address. For each line, the video processor reads the pixel data corresponding to width W from the output buffer and writes it continuously to the corresponding storage area on the target surface, starting from the storage location corresponding to the write start address. After one line is written, the video processor moves the write address to the corresponding position of the next line on the target surface (i.e., the current line start address plus the target surface line step) and continues writing the next line of data. This process is repeated until the number of lines written reaches the number of lines corresponding to height H, thus completely writing the processing results to the target rectangular region of the target surface.

[0071] It should be noted that when the video processor performs the write operation, the original pixel data in the target rectangular area on the target surface is completely overwritten by the processing result. Since the pixel format in this embodiment can be RGBA (Red-Green-Blue-Alpha) format, and the Alpha channel value of each pixel is set to opaque, the written pixel completely covers the original background pixels at the written location, ensuring that the sub-screens in the multi-screen display do not interfere with each other.

[0072] In this embodiment, the video processor calculates the start address and write range on the target surface based on the position and size information in the rectangle parameters of the target rectangular region. The processing result is directly written to the specified position on the target surface through the output view. The writing process is completed entirely inside the GPU, without the need to read the data back to the system memory and then write it back, thus avoiding additional data transmission overhead.

[0073] In one exemplary embodiment, determining at least one source rectangular region for constituting a multi-screen display, and a target rectangular region corresponding to each source rectangular region on a target surface, includes: determining at least one source rectangular region and display mode type information, the display mode type information including a panoramic plus local mode; when the display mode type information is a panoramic plus local mode, determining a first source rectangular region corresponding to a panoramic region of the decoded video image and a second source rectangular region corresponding to at least one local region in the decoded video image; determining a first target rectangular region corresponding to the first source region on the target surface and a second target rectangular region corresponding to the second source region on the target surface; determining the first source region and the second source region as at least one source rectangular region, and determining the first target rectangular region and the second target rectangular region as the target rectangular regions corresponding to each source rectangular region.

[0074] The display mode type information refers to configuration parameters indicating the display layout category of the current multi-screen display. It includes at least a panoramic plus local mode, where the multi-screen display consists of one global region and at least one local region. The global region refers to the complete frame of the decoded video image, i.e., the entire frame of image content carried by the original texture surface. The local region refers to one or more user-specified local sub-regions in the decoded video image, corresponding to specific targets or fields of view that the user needs to focus on in the panoramic view.

[0075] In some embodiments, when the video processor begins the rendering process for the current frame, it first reads the display mode type information from an internal register or a parameter buffer in memory. The display mode type information is set by the upper-layer application based on user operation and then passed to the video processor. The display mode type information includes at least two types: panoramic plus partial mode and single-view mode. In panoramic plus partial mode, the multi-screen display consists of one panoramic view and at least one magnified partial view; in single-view mode, the multi-screen display contains only one complete video view.

[0076] The video processor determines the subsequent region calculation strategy based on the display mode type information. If the display mode type information is a panoramic plus local mode, the video processor determines the number of at least one source rectangular regions. The number of source rectangular regions depends on the number of local regions. For example, if the user selects only one region of interest, the number of source rectangular regions is 2 (one panoramic region plus one local region); if the user selects three regions of interest, the number of source rectangular regions is 4 (one panoramic region plus three local regions). The video processor reserves corresponding storage space for each source rectangular region in its internal parameter buffer according to the number of local regions set by the user.

[0077] The video processor reads the encoded width and height of the decoded video image from its internal parameters, sets the top-left corner coordinates of the first source rectangle to (0, 0), the width to the encoded width, and the height to the encoded height. The first source rectangle is the complete pixel range covering the original texture surface. In one embodiment, if the user sets a cropped area of ​​the panoramic image through an upper-layer application (e.g., the effective field of view after distortion correction of a fisheye image), the video processor uses this cropped area as the first source rectangle, rather than the complete frame range. The video processor determines the second source rectangle as corresponding to at least one local region in the decoded video image. The local region is set by the user through an upper-layer application; the user can select one or more regions of interest in the preview screen by dragging with a mouse, touch operation, or coordinate input. Each local region is defined by a set of rectangular parameters, including the coordinates of the top-left corner of the local region in the original texture surface, as well as the width and height of the local region. The video processor reads these local region parameters from the parameter buffer and uses them as the second source rectangle. When multiple local regions exist, the video processor determines multiple second source rectangles, each local region corresponding to an independent second source rectangle.

[0078] It should be noted that both the first and second source rectangular regions point to the same original texture surface, i.e., the same decoded video image. The difference between the panoramic region and the local region lies only in their coverage area on the original texture surface; the panoramic region covers the entire area, while the local region covers a local sub-area.

[0079] The video processor determines the first target rectangular area as the display position corresponding to the panoramic view. In the preset layout, the panoramic view occupies the main area of ​​the target surface. The video processor determines the second target rectangular area as the display position corresponding to the magnified partial view. In the preset layout, the partial view occupies a secondary area of ​​the target surface. When determining the first and second target rectangular areas, the video processor also corrects the coordinates of the target rectangular areas with reference to the currently selected aspect ratio to ensure the correct display ratio of each sub-view. In an optional implementation, the video processor also adjusts the coordinates of the target rectangular areas based on whether corridor mode is currently enabled. In corridor mode, the target rectangular area is limited to a vertical strip area.

[0080] After completing the above calculations, the video processor determines the first source rectangle region and all second source rectangle regions together as at least one source rectangle region, and determines the corresponding first target rectangle region and all second target rectangle regions together as the target rectangle region corresponding to each source rectangle region. These source rectangle regions and target rectangle regions are stored in a one-to-one correspondence in the parameter buffer inside the video processor for use in subsequent hardware scaling and color space conversion stages.

[0081] In this embodiment, the entire process of determining the source rectangular region and the target rectangular region in the panoramic plus local mode is completed by the video processor, which performs coordinate calculations. The CPU does not need to participate in pixel-level calculations. Furthermore, the panoramic region and all local regions point to the same original texture surface, ensuring the uniqueness of the data source and avoiding multiple decoding or copying of the texture.

[0082] In one exemplary embodiment, determining at least one source rectangular region for constituting a multi-screen display, and a corresponding target rectangular region on a target surface for each source rectangular region, includes: acquiring display mode type information, which includes a single-screen mode; if the display mode type information is a single-screen mode, determining a source rectangular region corresponding to a panoramic area of ​​a decoded video image; determining the size information of the source rectangular region on the original texture surface, the size information of the decoded video image, and the size information of the target surface; and determining the target rectangular region corresponding to the source rectangular region based on the size information of the source rectangular region on the original texture surface, the size information of the decoded video image, and the size information of the target surface.

[0083] In single-screen mode, the multi-screen view contains only one complete video feed; the electronic pan-tilt function is disabled, neither cropping local areas nor performing multi-screen stitching. The dimensions of the source rectangular region on the original texture surface refer to its width in the horizontal direction and its height in the vertical direction. In single-screen mode, the source rectangular region corresponds to the panoramic area of ​​the decoded video image, and its dimensions are consistent with the dimensions of the decoded video image. The dimensions of the decoded video image refer to the number of pixels in the complete video image carried by the original texture surface in the horizontal and vertical directions, i.e., the encoded width and encoded height. The dimensions of the target surface refer to the number of pixels in the target surface in the horizontal and vertical directions, i.e., the target surface width and height.

[0084] In some embodiments, after reading the display mode type information, the video processor determines whether the current mode is single-screen mode. If the display mode type information indicates single-screen mode, it means that the electronic pan-tilt function is off, and the video processor does not need to capture any local area or perform multi-screen mosaicking; it only needs to display the complete decoded video image on the target surface. In single-screen mode, the video processor determines the source rectangular region corresponding to the panoramic area of ​​the decoded video image. Specifically, the video processor reads the encoded width and encoded height of the decoded video image from internal parameters, sets the upper left corner coordinates of the source rectangular region to (0, 0), sets the width to the encoded width, and sets the height to the encoded height. The source rectangular region covers the complete pixel range of the original texture surface, i.e., the panoramic area.

[0085] The video processor determines the size information of the source rectangular region on the original texture surface. Since the source rectangular region corresponds to the complete panoramic area, its width is the encoded width and its height is the encoded height. The video processor determines the size information of the decoded video image, which is exactly the same as the size information of the source rectangular region in this embodiment, namely, the encoded width and encoded height. It should be noted that in single-view mode, the size of the source rectangular region is equal to the size of the decoded video image, therefore the above two sets of size information are numerically consistent.

[0086] The video processor determines the size information of the target surface. The width and height of the target surface are determined when the target surface is assigned, and the video processor reads these two values ​​from the target surface attributes associated with the output view. In a display scene, the target surface is a back buffer of the swap chain, and its size is the same as the size of the display window; in a snapshot scene, the target surface is a separate rendering texture, and its size is determined by the snapshot resolution.

[0087] The video processor first calculates the aspect ratio of the decoded video image, where aspect ratio = encoded width / encoded height. Next, it calculates the aspect ratio of the target surface, where the target surface aspect ratio = target surface width / target surface height. The video processor compares the image aspect ratio and the target surface aspect ratio to determine the calculation strategy for the target rectangular region. If the image aspect ratio is greater than the target surface aspect ratio, it means the decoded video image is "wider" relative to the target surface. In this case, the video processor sets the width of the target rectangular region to be equal to the target surface width, and then calculates the height of the target rectangular region as its width / image aspect ratio. Since the calculated target height is less than the target surface height, the video processor centers the target rectangular region vertically, with the top-left corner's ordinate being (target surface height - target rectangular region height) / 2, and the top-left corner's abscissa being 0.

[0088] If the aspect ratio of the image is smaller than that of the target surface, it means that the decoded video image is "taller" than the target surface. In this case, the video processor uses the height of the target surface as a reference, sets the height of the target rectangular region to be equal to the height of the target surface, and then calculates the width of the target rectangular region as the height of the target rectangular region multiplied by the image aspect ratio. Since the calculated target width is smaller than the target surface width, the video processor centers the target rectangular region horizontally, with the x-coordinate of the top-left corner of the target rectangular region being (the width of the target surface - the width of the target rectangular region) / 2, and the y-coordinate of the top-left corner being 0.

[0089] If the aspect ratio of the image is equal to that of the target surface, it means that the decoded video image and the target surface are in the same proportion. In this case, the size of the target rectangular region is equal to the size of the target surface, the target rectangular region covers the entire target surface, and the coordinates of the upper left corner are (0, 0).

[0090] After the video processor completes the above calculations, it obtains the rectangular parameters of the target rectangular region corresponding to the source rectangular region, including the coordinates of the top left corner, width, and height of the target rectangular region.

[0091] In this embodiment, the video processor automatically calculates the target rectangular area in single-screen mode based on the size information of the decoded video image and the size information of the target surface, ensuring that the decoded video image always maintains its original aspect ratio on the target surface and will not be stretched or compressed due to the inconsistency between the target surface ratio and the image ratio.

[0092] In one exemplary embodiment, the method further includes: obtaining a sub-resource index corresponding to the decoded video image in the original texture surface; the sub-resource index is used to identify the target sub-resource storing the decoded video image among multiple sub-resources contained in the original texture surface; and creating an input view pointing to the target sub-resource based on the original texture surface and the sub-resource index.

[0093] In this context, a sub-resource index is a unique identifier or number used to uniquely identify a specific sub-resource within a texture resource that contains multiple independent data storage units (i.e., sub-resources). The original texture surface can be a texture array, where each array element is a sub-resource, and each sub-resource can store a complete frame of video pixel data. The target sub-resource refers to the specific storage unit pointed to by the sub-resource index, which contains the pixel data of the currently decoded video image to be processed.

[0094] In some embodiments, after decoding a video frame, the hardware decoder writes the decoded raw pixel data into the raw texture surface in the GPU memory. The raw texture surface is configured during allocation to contain multiple sub-resources (e.g., a texture array, where each array element corresponds to a decoded frame buffer) to support multi-frame parallel decoding or circular buffering of decoded frames. After decoding, the hardware decoder passes a resource pointer to the raw texture surface and the index of the sub-resource where the currently decoded frame is stored to the video processor. The sub-resource index indicates which sub-resource of the raw texture surface the currently decoded video image is stored in. For example, in a texture array, the sub-resource index is the array slice index, with values ​​of 0, 1, 2, ..., corresponding to the 0th, 1st, 2nd, etc., layers of the texture array, respectively. The video processor obtains the sub-resource index from the parameters passed by the decoder.

[0095] After obtaining the resource pointer and sub-resource index of the original texture surface, the video processor calls the interface for creating an input view, passing in the original texture resource pointer and a view description structure. In the view description structure, the video processor sets the sub-resource index field, setting its value to the sub-resource index value obtained from the decoder. When the video processor calls the interface for creating the input view, the graphics driver locates the original texture surface based on the passed original texture resource pointer and locates the target sub-resource among the multiple sub-resources contained in the original texture surface based on the sub-resource index. The input view is configured to point only to the storage area corresponding to the target sub-resource, that is, to only overwrite the data of the decoded video image of the frame stored in the target sub-resource. When the video processor subsequently reads data through this input view, the read operation is restricted to the scope of the target sub-resource, preventing accidental reading of other frame data stored in other sub-resources in the original texture surface.

[0096] After the video processor completes the creation of the input view, it is bound to the target sub-resource in the original texture surface. When the video processor subsequently reads data from the original texture surface based on the input view, the actual data source is the decoded video image stored in the target sub-resource. Throughout the creation process, the input view does not copy any pixel data; it only establishes an access handle to the target sub-resource, thus minimizing creation overhead.

[0097] In this embodiment, only an access handle is created when the input view is created; no pixel data is copied, resulting in minimal creation overhead and no impact on real-time rendering performance. Furthermore, when the decoder outputs a new frame and switches sub-resource indices, the video processor only needs to recreate the input view; there is no need to rebuild the video processor or output view, resulting in high switching efficiency.

[0098] In one exemplary embodiment, writing the processing results of each of the source rectangular regions into the corresponding target rectangular region includes: performing a screen clearing operation on the target surface to fill all pixels of the target surface with a preset background color; writing the processing results of each of the source rectangular regions into the corresponding target rectangular region, and overwriting the original pixel data at the corresponding position of the target surface with the opacity channel value of each processing result.

[0099] To avoid "ghosting" or "screen tearing" in multi-screen displays due to residual data on the target surface, and to ensure color consistency in the peripheral areas of the screen when switching between different display ratios, the video processor performs strict surface preprocessing and overwrite strategies before the first write operation. Specifically, after acquiring the output view and before calling the video processing call for the first source rectangular area, the video processor first calls the graphics interface to perform a screen clearing operation on the target surface pointed to by the output view. The screen clearing operation fills all pixels on the target surface with a preset background color (e.g., pure black, RGB value 0, 0, 0).

[0100] Subsequently, during hardware scaling and color space conversion for each source rectangular region, the output pixel format within the video processor hardware is configured to include an alpha channel. While performing color space conversion, the video processor forces the alpha channel value of the output pixels to be opaque (i.e., alpha = 1.0). When the video processor writes the processing results to the target surface, the hardware performs a pixel overlay operation rather than a transparency blending operation. Because each target rectangular region has non-overlapping coordinate ranges on the target surface, and the written pixels are opaque, each write operation directly overwrites the original background color within the target rectangular region without color blending with the surrounding black background or other already written sub-frames.

[0101] In this embodiment, the problem of black borders caused by the adaptation of the image edge ratio is solved, and the clear boundaries of multiple magnified images are ensured when they are stitched together, without the need for the CPU to participate in additional pixel mixing calculations.

[0102] In one exemplary embodiment, for each of the source rectangular regions, obtaining pixel data in the source rectangular region from the original texture surface according to the input view includes: for each of the source rectangular regions, performing a pixel alignment operation on the coordinate information and size information of the source rectangular region to obtain a pixel-aligned source rectangular region; the pixel alignment operation is used to adjust the coordinate information and size information to an alignment value that conforms to the hardware access granularity of the video processor; the pixel alignment operation includes performing a 16-pixel alignment bitwise operation correction on the coordinate values ​​and size values ​​of the source rectangular region; and obtaining pixel data in the pixel-aligned source rectangular region from the original texture surface according to the input view.

[0103] To ensure the video processor can access video memory with maximum efficiency and prevent software rollback in the driver layer due to misaligned memory addresses, the video processor performs alignment correction on the rectangular parameters of the source rectangular region after determining it and before extracting pixel data from the original texture surface based on the input view. Specifically, the video processor first reads the original parameters of the source rectangular region to be processed from its internal parameter buffer, including the x-coordinate of the top-left corner (X), the y-coordinate of the top-left corner (Y), the width (W), and the height (H). Then, the video processor calls a safe bitwise operation function from the hardware instruction set to perform a 16-pixel alignment correction on the above four parameters. For the coordinate value X, the bitwise operation ((~0 - 1) & X) is used to correct X to a value no greater than the current X and the nearest multiple of 16. For the width value W, the bitwise operation ((~0 - 1) & (W + 15)) is used to correct W to a value that makes the right boundary (X + W) of the extracted range meet the 16-pixel alignment requirement. Similarly, the same correction operation is performed on the y-coordinate Y and the height H. This bitwise operation avoids the use of complex division or modulo instructions and can be completed within just a few clock cycles inside the video processor. The pixel-aligned source rectangle region obtained after the above 16-pixel alignment correction has a starting memory address and a read width per row that meet the alignment requirements of the hardware video processor's DMA burst transfer. When the video processor accesses the original texture surface through the input view based on the rectangle parameters of the pixel-aligned source rectangle region, the hardware can directly initiate efficient continuous burst read requests without requiring the driver layer to split unaligned requests or rearrange data through system memory.

[0104] In this embodiment, the cropping, scaling, and color space conversion processes always run on a pure hardware-accelerated path, effectively maintaining the real-time frame rate in high-resolution video processing scenarios.

[0105] In a detailed embodiment, when the video processor begins processing the current frame, it first acquires its required input and output views. The input view is constructed based on the original texture surface of the decoded video image stored in the graphics processor's video memory, indicating the storage location of the pixel data to be processed within the original texture surface. Specifically, the input view is created by acquiring the sub-resource index corresponding to the decoded video image in the original texture surface and, based on the original texture surface and the sub-resource index, pointing to the target sub-resource. The sub-resource index is used to identify the target sub-resource storing the current decoded video image among the multiple sub-resources contained in the original texture surface, ensuring that the input view only points to the storage area corresponding to the target sub-resource. The output view is constructed based on the target surface of the multi-screen image to be rendered, allocated in the graphics processor's video memory, indicating the write location of the processing results on the target surface. After acquiring the input and output views, the video processor first performs a screen clearing operation on the target surface, for example, by calling a screen clearing function of the graphics interface, filling all pixels of the target surface pointed to by the output view with black to provide a clean composite background.

[0106] Subsequently, the video processor determines at least one source rectangular region for constructing the multi-screen image, and the corresponding target rectangular region on the target surface for each source rectangular region. Specifically, the video processor calculates this based on information such as the current display mode type, user-defined electronic pan-tilt parameters (e.g., the position and size of the local area of ​​interest), aspect ratio, and whether corridor mode is enabled. For example, if the display mode type is panorama plus local mode, the video processor determines a first source rectangular region corresponding to the panoramic area of ​​the decoded video image, and a second source rectangular region corresponding to one or more local areas selected by the user; and correspondingly determines a first target rectangular region on the target surface for the first source region, and a second target rectangular region on the target surface for the second source region. If the display mode type is single-screen mode, the video processor determines a single source rectangular region corresponding to the panoramic area of ​​the decoded video image, and calculates the corresponding target rectangular region based on the size information of the source rectangular region, the size information of the decoded video image, and the size information of the target surface, according to a preset aspect ratio adaptive strategy (e.g., maintaining the original aspect ratio and centering the display). For the target rectangular region in panoramic plus partial mode, the video processor determines it based on a preset layout template (e.g., the panoramic region occupies the upper 2 / 3 of the image, and the partial region is placed side by side in the lower 1 / 3) and in combination with the aspect ratio type and corridor mode configuration. In some embodiments, when determining the rectangular parameters of all source and target rectangular regions, the video processor can perform pixel alignment operations on the rectangular parameters (such as position coordinates and size values), for example, by converting each coordinate value into a 16-pixel aligned value through bitwise operations, to avoid software rollback processing that the hardware video processor may cause for misaligned coordinates, thereby ensuring hardware acceleration throughout the process.

[0107] After identifying all source rectangular regions and their corresponding target rectangular regions, the video processor begins processing and writing operations region by region. Specifically, for each source rectangular region, the video processor first determines its rectangular parameters (including position and size information) on the original texture surface. Next, based on the input view and the rectangular parameters, the video processor extracts the pixel data corresponding to the source rectangular region from the original texture surface. Specifically, it determines the extraction range on the original texture surface based on the position and size information, and then extracts the pixel data corresponding to the extraction range, obtaining the extracted pixel data. The extraction operation only reads pixels within the range defined by the rectangular parameters and does not write the extracted data to system memory or allocate additional GPU memory buffers; instead, it flows through the video processor in a pipelined manner. Subsequently, the video processor performs hardware scaling and color space conversion on the extracted pixel data to obtain the processing result for the source rectangular region. Both hardware scaling and color space conversion are performed by dedicated circuitry within the video processor hardware and do not involve pixel-level calculations by the CPU. After obtaining the processing results for the source rectangular regions, the video processor writes the processing results for each source rectangular region to the corresponding target rectangular region according to the output view. Specifically, for the processing result of each source rectangular region, the rectangular parameters of the target rectangular region corresponding to that source rectangular region are determined on the target surface; then, based on the rectangular parameters of the target rectangular region and the output view, the processing results of the source rectangular regions are directly written to the positions corresponding to the target rectangular regions on the target surface. The writing operation is completed internally by the GPU. The video processor calculates the write start address on the target surface based on the rectangular parameters of the target rectangular region, reads the processing results from the internal output buffer, and writes them line by line to the designated storage area on the target surface. The written pixels overwrite the original background pixels at the written positions.

[0108] The video processor sequentially performs the above-described sequence of operations—capturing, hardware scaling, color space conversion, and writing—on each source rectangular region. Throughout this process, the input view remains constant, always pointing to the same original texture surface containing the decoded video image; only the source and target rectangle parameters are switched between different source rectangular regions. Once all source rectangular regions have been processed and written, a complete multi-screen image, composed of multiple sub-images, is composited on the target surface. In some embodiments, if the current frame is a snapshot path, the video processor forces the display mode type information to single-screen mode, ensuring the source rectangular regions cover the entire frame, and writes the processing result to the target surface, thereby obtaining a snapshot image containing the complete image. This achieves automatic bypass of the electronic gimbal effect under snapshot paths, guaranteeing that the screenshot is a complete image. For normal rendering outside of snapshot paths, after the multi-screen image is composited on the target surface, the video processor presents the target surface (e.g., the back buffer of the swap chain) to the display device via a Present operation, thus displaying the multi-screen image on the screen.

[0109] In one example, during the initialization phase, the Direct3D 11-based hardware video processor first obtains the D3D11 device, video device, and video context shared with FFmpeg through the av_hwdevice_ctx_create interface and specifies the AV_HWDEVICE_TYPE_D3D11VA type, thereby establishing direct communication with the graphics card driver layer. Subsequently, the processor creates a swap chain with attributes of 16-pixel aligned width and height, B8G8R8A8_UNORM pixel format, DISCARD swapping effect, and a maximum frame latency of 1, ensuring real-time rendering in a minimum-latency single-buffered mode.

[0110] Next, the D3D11_VIDEO_PROCESSOR_CONTENT_DESC structure is used to describe the input (NV12) and output (BGRA) formats and playback scene types. A video processor enumerator, VideoProcessorEnumerator, is created, and finally, an ID3D11VideoProcessor core hardware processing instance is created based on this enumerator, preparing for subsequent hardware-accelerated BLT operations.

[0111] In each frame rendering loop, the video processor first acquires the back buffer of the swap chain, creates a render target view for it, and performs a screen clearing operation to fill the entire target surface with black to provide a clean composite background. For the input video frame (AVFrame), the video processor directly uses its texture pointer (frame->data[0]) and array slice index (frame->data[1]) to create the input view (ID3D11VideoProcessorInputView). This process is entirely based on referencing the original texture surface in video memory and does not involve copying any pixel data. At the same time, the output view (ID3D11VideoProcessorOutputView) points directly to the same back buffer, ensuring that the processing result is directly written to the final display target. The video processor calculates the source rectangle area based on the display rectangle (m_SourceRect) set by the user; if it is currently in snapshot save mode (m_bSaveFrame is true), it forces the bypass of the electronic gimbal effect (eptzMode = NONE) so that the source rectangle covers the entire frame range. Based on the different branches of the electronic pan-tilt mode (TYPE1 corresponds to a panoramic view plus one local area, TYPE2 corresponds to a panoramic view plus up to three local areas, and NONE corresponds to a single frame), the video processor constructs corresponding source rectangle lists and target rectangle lists. It then adjusts the target rectangle areas according to the current aspect ratio (m_ratioType, such as 32:9, 16:9, 4:3) to ensure correct display proportions. Subsequently, it repeatedly calls VideoProcessorSetRect to set parameters for each source-target rectangle group and calls the VideoProcessorBlt interface to trigger the video processor to perform actual scaling, color space conversion, and pixel writing operations, directly drawing the image data of each source rectangle area to the corresponding target rectangle position in the back buffer. If snapshot saving is enabled, SaveFrameToJPG is called to save it as a JPEG file after Blt is completed; otherwise, OSD (On-Screen Display) information is overlaid on the composite image.

[0112] Finally, the contents of the back buffer are flushed to the display device via the Present1 interface with the DXGI_PRESENT_DO_NOT_WAIT flag specified, and references to the input view, output view, and back buffer are released after rendering to reclaim resources.

[0113] Regarding the dynamic updating of electronic gimbal parameters, the video processor provides the SetEPTZParam interface, which accepts a structure parameter containing the mode type and a local rectangle array (eptzRect[3]). The video processor performs an even-number alignment operation on all received rectangular components (i.e., the coordinates and dimensions are forcibly converted to even numbers through bitwise operations of ((~0 - 1) & x)) to ensure that they meet the strict requirements of the video processor for pixel alignment and avoid triggering software rollback. The updated parameters take effect immediately when the next frame is rendered. The entire switching process does not require rebuilding the decoder instance or swap chain resources, thereby achieving seamless and low-latency electronic gimbal parameter adjustment.

[0114] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0115] Based on the same inventive concept, this application also provides a multi-screen rendering apparatus for implementing the multi-screen rendering method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more multi-screen rendering apparatus embodiments provided below can be found in the limitations of the multi-screen rendering method described above, and will not be repeated here.

[0116] In one exemplary embodiment, such as Figure 5 As shown, a multi-screen rendering device 500 is provided, comprising:

[0117] The acquisition module 501 is used to acquire an input view and an output view; wherein, the input view is constructed based on the original texture surface of the decoded video image stored in the graphics processor's video memory, and is used to indicate the storage location of the pixel data to be processed in the original texture surface; the output view is constructed based on the target surface of the multi-screen image to be rendered allocated in the graphics processor's video memory, and is used to indicate the writing location of the processing result in the target surface.

[0118] The determining module 502 is used to determine at least one source rectangular region for constituting a multi-screen display, and the target rectangular region corresponding to each source rectangular region on the target surface.

[0119] The processing module 503 is used to obtain pixel data in the source rectangular region from the original texture surface according to the input view for each source rectangular region without releasing or rebuilding the input view, and to perform hardware scaling and color space conversion on the pixel data in the source rectangular region to obtain the processing result of the source rectangular region.

[0120] The writing module 504 is used to write the processing results of each source rectangular area into the corresponding target rectangular area according to the output view, so as to synthesize a multi-screen image on the target surface.

[0121] In one exemplary embodiment, the processing module 503 is configured to:

[0122] For each source rectangular region, determine the rectangle parameters of the source rectangular region on the original texture surface;

[0123] Based on the input view and rectangle parameters, the pixel data corresponding to the source rectangle region is extracted from the original texture surface to obtain the extracted pixel data;

[0124] Hardware scaling and color space conversion are performed on the captured pixel data to obtain the processing result of the source rectangular region.

[0125] In one exemplary embodiment, the rectangle parameters include position information and size information; the processing module 503 is used for:

[0126] Based on the location and size information, determine the cut-off range in the original textured surface;

[0127] Extract the pixel data corresponding to the selected area from the original texture surface to obtain the extracted pixel data.

[0128] In one exemplary embodiment, the writing module 504 is used to:

[0129] For each source rectangular region, determine the rectangle parameters of the target rectangular region on the target surface corresponding to the source rectangular region based on the processing result.

[0130] Based on the rectangle parameters of the target rectangle region and the output view, the processing result of the source rectangle region is written into the target surface at the position corresponding to the target rectangle region.

[0131] In one exemplary embodiment, the determining module 502 is used to:

[0132] Determine at least one source rectangular region and display mode type information, which includes panoramic plus local modes;

[0133] When the display mode type information is panorama plus local mode, a first source rectangle region corresponding to the panorama region of the decoded video image and a second source rectangle region corresponding to at least one local region in the decoded video image are determined.

[0134] Determine the first target rectangular region corresponding to the first source region on the target surface, and the second target rectangular region corresponding to the second source region on the target surface;

[0135] The first source region and the second source region are defined as at least one source rectangular region, and the first target rectangular region and the second target rectangular region are defined as the target rectangular regions corresponding to each source rectangular region.

[0136] In one exemplary embodiment, the determining module 502 is used to:

[0137] Obtain display mode type information, which includes single-screen mode;

[0138] When the display mode type information is single-screen mode, determine the source rectangular region corresponding to the panoramic area of ​​the decoded video image;

[0139] Determine the size information of the source rectangular region in the original texture surface, the size information of the decoded video image, and the size information of the target surface;

[0140] Based on the size information of the source rectangular region in the original texture surface, the size information of the decoded video image, and the size information of the target surface, the target rectangular region corresponding to the source rectangular region is determined.

[0141] In one exemplary embodiment, it further includes:

[0142] The sub-resource index acquisition module is used to acquire the sub-resource index corresponding to the decoded video image in the original texture surface; the sub-resource index is used to identify the target sub-resource storing the decoded video image among multiple sub-resources contained in the original texture surface;

[0143] Create a module that generates an input view pointing to the target sub-resource based on the original texture surface and sub-resource index.

[0144] In one exemplary embodiment, the writing module 504 is used to:

[0145] Perform a screen clearing operation on the target surface, filling all pixels of the target surface with a preset background color;

[0146] The processing results of each of the source rectangular regions are written into the corresponding target rectangular region, and the original pixel data at the corresponding position on the target surface is overwritten with the opaque channel value of each processing result.

[0147] In one exemplary embodiment, the processing module 503 is configured to:

[0148] For each of the source rectangular regions, a pixel alignment operation is performed on the coordinate and size information of the source rectangular region to obtain a pixel-aligned source rectangular region; the pixel alignment operation is used to adjust the coordinate and size information to an alignment value that conforms to the hardware access granularity of the video processor; the pixel alignment operation includes performing a 16-pixel alignment bitwise operation correction on the coordinate and size values ​​of the source rectangular region.

[0149] Pixel data in the pixel-aligned source rectangular region is obtained from the original texture surface according to the input view.

[0150] The modules in the aforementioned multi-screen rendering device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0151] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores configuration parameters, including display mode type information, rectangle parameters for each source rectangle region, rectangle parameters for each target rectangle region, target surface dimensions, and decoded video image dimensions. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a multi-screen rendering method.

[0152] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0154] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0155] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0156] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0157] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0158] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-screen rendering method, characterized in that, Applied to video processors, including: Obtain an input view and an output view; wherein, the input view is constructed based on the original texture surface of the decoded video image stored in the graphics processor's video memory, and is used to indicate the storage location of the pixel data to be processed in the original texture surface; the output view is constructed based on the target surface of the multi-screen image to be rendered allocated in the graphics processor's video memory, and is used to indicate the writing location of the processing result in the target surface. Determine at least one source rectangular region for constituting the multi-screen display, and the target rectangular region corresponding to each of the source rectangular regions on the target surface. Without releasing or rebuilding the input view, for each source rectangular region, pixel data in the source rectangular region is obtained from the original texture surface according to the input view, and hardware scaling and color space conversion are performed on the pixel data in the source rectangular region to obtain the processing result of the source rectangular region; Based on the output view, the processing results of each of the source rectangular regions are written into the corresponding target rectangular region to synthesize the multi-screen image on the target surface.

2. The method according to claim 1, characterized in that, For each of the source rectangular regions, hardware scaling and color space conversion are performed on the pixel data within the source rectangular region based on the input view to obtain the processing result of the source rectangular region, including: For each of the source rectangular regions, determine the rectangular parameters of the source rectangular region in the original texture surface; Based on the input view and the rectangle parameters, pixel data corresponding to the source rectangle region is extracted from the original texture surface to obtain the extracted pixel data; Hardware scaling and color space conversion are performed on the extracted pixel data to obtain the processing result of the source rectangular region.

3. The method according to claim 2, characterized in that, The rectangular parameters include position information and size information; The pixel data corresponding to the source rectangular region is extracted from the original texture surface to obtain the extracted pixel data, including: Based on the location information and the size information, the cut-off range is determined in the original textured surface; The pixel data corresponding to the cut-out range is extracted from the original texture surface to obtain the cut-out pixel data.

4. The method according to claim 1, characterized in that, Based on the output view, the processing results of each of the source rectangular regions are written into the corresponding target rectangular region, including: For each source rectangular region, the rectangular parameters of the target rectangular region corresponding to the source rectangular region in the target surface are determined; Based on the rectangle parameters of the target rectangular region and the output view, the processing result of the source rectangular region is written into the target surface at the position corresponding to the target rectangular region.

5. The method according to claim 1, characterized in that, Determining at least one source rectangular region for constituting the multi-screen display, and the corresponding target rectangular region on the target surface for each of the source rectangular regions, includes: Determine at least one source rectangular region and display mode type information, wherein the display mode type information includes panoramic plus local mode; When the display mode type information is panorama plus local mode, a first source rectangle region corresponding to the panorama region of the decoded video image and a second source rectangle region corresponding to at least one local region in the decoded video image are determined in the at least one source rectangle region. Determine a first target rectangular region corresponding to the first source region on the target surface, and a second target rectangular region corresponding to the second source region on the target surface; The first source region and the second source region are determined as at least one source rectangular region, and the first target rectangular region and the second target rectangular region are determined as the target rectangular regions corresponding to each of the source rectangular regions.

6. The method according to claim 1, characterized in that, Determining at least one source rectangular region for constituting the multi-screen display, and the corresponding target rectangular region on the target surface for each of the source rectangular regions, includes: Obtain display mode type information, including single-screen mode; When the display mode type information is single-screen mode, the source rectangular region corresponding to the panoramic area of ​​the decoded video image is determined; Determine the size information of the source rectangular region in the original texture surface, the size information of the decoded video image, and the size information of the target surface; The target rectangular region corresponding to the source rectangular region is determined based on the size information of the source rectangular region in the original texture surface, the size information of the decoded video image, and the size information of the target surface.

7. The method according to any one of claims 1-6, characterized in that, Also includes: Obtain the sub-resource index corresponding to the decoded video image in the original texture surface; The sub-resource index is used to identify the target sub-resource storing the decoded video image among the multiple sub-resources contained in the original texture surface; Based on the original texture surface and the sub-resource index, create the input view pointing to the target sub-resource.

8. The method according to claim 1, characterized in that, The processing results of each of the source rectangular regions are written into the corresponding target rectangular region, including: Perform a screen clearing operation on the target surface, filling all pixels of the target surface with a preset background color; The processing results of each of the source rectangular regions are written into the corresponding target rectangular region, and the original pixel data at the corresponding position on the target surface is overwritten with the opaque channel value of each processing result.

9. The method according to claim 1, characterized in that, For each of the source rectangular regions, pixel data within the source rectangular region is obtained from the original texture surface according to the input view, including: For each of the source rectangular regions, a pixel alignment operation is performed on the coordinate and size information of the source rectangular region to obtain a pixel-aligned source rectangular region; the pixel alignment operation is used to adjust the coordinate and size information to an alignment value that conforms to the hardware access granularity of the video processor; the pixel alignment operation includes performing a 16-pixel alignment bitwise operation correction on the coordinate and size values ​​of the source rectangular region. Pixel data in the pixel-aligned source rectangular region is obtained from the original texture surface according to the input view.

10. A multi-screen rendering device, characterized in that, The device includes: An acquisition module is used to acquire an input view and an output view; wherein, the input view is constructed based on the original texture surface of the decoded video image stored in the graphics processor's video memory, and is used to indicate the storage location of the pixel data to be processed in the original texture surface; the output view is constructed based on the target surface of the multi-screen image to be rendered allocated in the graphics processor's video memory, and is used to indicate the writing location of the processing result in the target surface. The determining module is used to determine at least one source rectangular region for constituting the multi-screen image, and the target rectangular region corresponding to each of the source rectangular regions on the target surface. The processing module is used to obtain pixel data from the original texture surface of each source rectangular region according to the input view without releasing or rebuilding the input view, and to perform hardware scaling and color space conversion on the pixel data in the source rectangular region to obtain the processing result of the source rectangular region. The writing module is used to write the processing results of each of the source rectangular regions into the corresponding target rectangular regions according to the output view, so as to synthesize the multi-screen image on the target surface.

Images