A splicing display method and device based on video image processing and a medium

Abstract

The application discloses a splicing display method and device based on video image processing and a medium, relates to the technical field of image processing, and comprises the following steps: through a cross-screen image block group, tracing a blocking propagation relationship and a cross-screen association relationship caused by a video window change, determining a video window and a corresponding display processing node that need to be jointly updated, and constructing a common update object set; checking a video frame version and a window state version of the cross-screen image block in the common update object set, determining an overall display switching time in combination with a ready state of the display processing node, executing overall display switching on each video window layer in the common update object set, updating the video window outside the common update object set according to a respective version, and forming a cross-screen splicing display picture. The application jointly predicts a cross-screen migration range and pre-fetches an image in advance, simultaneously constructs the common update object set to implement version checking and local synchronous submission, and reduces cross-screen delay, picture tearing and irrelevant window blocking.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a splicing display method, device and medium based on video image processing. Background Technology

[0002] With the development of ultra-high-definition acquisition, network video transmission, distributed decoding, and multi-screen display control technologies, splicing display devices have been widely used in scenarios such as command and dispatch, security monitoring, industrial production monitoring, digital exhibitions, and converged media. Existing splicing display methods typically establish a unified display canvas based on the arrangement of physical displays, mapping the position, size, and layer order of video windows to corresponding display areas. Different display processing nodes then perform video reception, decoding, image cropping, scaling, and output adaptation. For video windows spanning multiple physical displays, the image content is generally segmented according to the overlap between the window and each display, and the segmentation results are sent to the corresponding nodes for parallel display. As the display scale, the number of video windows, and the frequency of interactive operations continue to increase, the movement, scaling, layer adjustment, and content changes of video windows exhibit greater dynamism. Splicing display control has gradually shifted from fixed screen allocation to real-time collaborative processing oriented towards dynamic changes across multiple nodes, windows, and screens.

[0003] However, existing splicing display technologies still have certain shortcomings. Most methods mainly rely on changes in the position or display boundaries of video windows to redistribute cross-screen tasks, failing to comprehensively utilize the external motion state of video windows and the internal motion state of video image content to predict the image range that may enter adjacent physical displays in advance. This easily leads to problems such as delays, short-term gaps, or discontinuous boundary connections in cross-screen images. Existing multi-node display updates are usually submitted separately for a single video window, a single physical display, or a single display processing node, making it difficult to fully identify the propagation effect of window occlusion changes between multiple video windows. It is also difficult to ensure that the image blocks of the same cross-screen window on different physical displays use the same video frame version and window state version simultaneously. When some nodes complete the update in advance while other nodes retain the original display state, cross-screen tearing, layer misalignment, or local mixing of old and new states may occur. If all window updates are paused to avoid this problem, it will affect the normal display of video windows unrelated to the current change. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a splicing display method based on video image processing to solve the problems of delayed cross-screen migration preparation and difficulty in synchronizing updates between associated video windows and display processing nodes.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a splicing display method based on video image processing, comprising: acquiring the arrangement relationship of physical display screens, the video to be displayed, and the corresponding video window; establishing a logical display canvas according to the arrangement relationship of the physical display screens; mapping the video window to the logical display canvas; identifying the actual visible range of the video window; determining the physical display screen and display processing node corresponding to the actual visible range; generating a visible mapping record; analyzing the external motion trend of the video window based on the visible mapping record; extracting the internal motion trend of the video image to be displayed; predicting the cross-screen migration range based on the external motion trend of the video window; determining the pre-fetched image range corresponding to the cross-screen migration range in combination with the internal motion trend of the image; determining the migration pre-fetch node based on the correspondence between the cross-screen migration range and the physical display screen; and generating a migration pre-fetch record. Based on the visible mapping record and migration prefetch record, the video to be displayed is transmitted to the display processing node and the migration prefetch node. The video to be displayed is decoded, and image cropping and display adaptation are performed according to the actual visible range and cross-screen migration range to form a cross-screen image block group. Through the cross-screen image block group, the occlusion propagation relationship and cross-screen association relationship caused by the change of video window are traced to determine the video window that needs to be updated together and the corresponding display processing node, and a common update object set is constructed. The video frame version and window state version of the cross-screen image blocks in the common update object set are verified. Combined with the readiness state of the display processing node, the overall display switching time is determined. The overall display switching is performed on each video window layer in the common update object set, and the video windows outside the common update object set are updated according to their respective versions to form a cross-screen spliced ​​display screen.

[0007] As a preferred embodiment of the video image processing-based splicing display method of the present invention, the specific steps for generating the visible mapping record are as follows: Based on the arrangement, resolution, and splicing boundaries of each physical display screen, a unified coordinate direction and coordinate origin are determined. The canvas offset and size conversion ratio of each physical display screen relative to the coordinate origin are obtained. The display boundaries of each physical display screen are converted into the corresponding canvas range, and a logical display canvas is established according to the distribution boundaries of the entire canvas range. The video window associated with the video to be displayed is used as the display object in the logical display canvas. The canvas position, display size and layer order corresponding to the video window are read. The window mapping range is formed according to the canvas position and display size. The common coverage part of the window mapping range and each canvas range is determined. The overlapping part displayed by other video windows is deducted according to the layer order. The retained image range is determined as the actual visible range. The actual visible area is divided according to the canvas area. The divided visible area is converted from canvas coordinates to the in-screen coordinates of the corresponding physical display screen, and associated with the display processing node responsible for the corresponding physical display screen to generate a visible mapping record.

[0008] As a preferred embodiment of the video image processing-based splicing display method of the present invention, the specific steps for predicting the cross-screen migration range are as follows: Extract the canvas position, display size, and generation time of the same video window from the visible mapping record. Identify the most recent continuous change process based on the center displacement and boundary stretching direction of the adjacent window states. Preserve the window states in the most recent continuous change process to form the external motion trend of the video window. Motion vectors of each image region are extracted from consecutive image frames of the video to be displayed. The motion vectors are aggregated according to the consistency of their direction and the proportion of the corresponding image region. The motion direction and amplitude of the image content are determined by combining the inter-frame time of adjacent image frames, thus forming the motion trend inside the image. Based on the external motion trend of the video window, predict the center position and display size of the video window to form a migration prediction range. The part of the migration prediction range that enters the canvas range of the adjacent physical display screen but is not within the actual visible range is determined as the cross-screen migration range.

[0009] As a preferred embodiment of the video image processing-based splicing display method of the present invention, the specific steps for generating migration prefetch records are as follows: The cross-screen migration range is mapped to the canvas range of each physical display screen in the logical display canvas. The migration sub-ranges are divided according to the part of the cross-screen migration range that enters different canvas ranges. Based on the image cropping ratio and display ratio of the video window, each migration sub-range is converted into the pre-fetched image range in the video to be displayed. Based on the external motion trend of the video window and the canvas distance from the video window boundary to each migration sub-range, the entry order of each migration sub-range is determined. The display processing node responsible for the corresponding physical display screen is determined as the migration prefetch node. The migration prefetch node, the prefetch image range and the entry order are associated to generate a migration prefetch record.

[0010] As a preferred embodiment of the video image processing-based splicing display method of the present invention, the specific steps for forming a cross-screen image block group are as follows: Combining the actual visible range in the visible mapping record and the prefetched image range in the migration prefetch record, the actual visible range is converted in reverse into the visible source image range in the video to be displayed, according to the position, display size and image cropping ratio of the video window in the logical display canvas, and the target in-screen range corresponding to the visible source image range and the prefetched image range are determined respectively. Based on the correspondence between the source image range and the target screen range, the processing tasks corresponding to the actual visible range are assigned to the display processing node, the processing tasks corresponding to the prefetched image range are assigned to the migration prefetch node, the source image ranges that are repeatedly covered are merged, and the adjacent target screen ranges are corrected according to the splicing boundary of the physical display screen. The same video to be displayed is decoded by the display processing node and the migration prefetch node, the image is cropped according to the allocated source image range, and the size is scaled and the display format is adapted according to the target screen range. The processed image is associated with the corresponding physical display screen, screen position, video frame version and window state version to form a cross-screen image block group.

[0011] As a preferred embodiment of the video image processing-based splicing display method of the present invention, the specific steps for tracing the occlusion propagation relationship and cross-screen correlation relationship caused by changes in the video window are as follows: Compare the state versions of adjacent windows and compare the video frame versions of cross-screen image blocks according to the video window identifier. The video windows that have changed at least one of the following: canvas position, display size, layer order, and actual visible range; the video windows whose video frame versions have changed and are distributed on different physical displays; and the video windows that have not yet completed their first display are identified as the starting windows of the change. For windows whose states have changed or whose initial display has not yet been completed, based on the video window identifier, canvas position, and layer order in the cross-screen image block group, the increase and decrease in coverage between the initial window and other video windows before and after the change are calculated. The affected video windows are then traced layer by layer along the display ownership relationship determined by the layer order to form an occlusion propagation relationship. According to the window identifier, cross-screen image blocks distributed on different physical displays but belonging to the same video window are grouped into the same cross-screen association range. The video frame version and window state version of each cross-screen image block are retained, and the cross-screen association range is associated with the splicing boundary and the corresponding display processing node to form a cross-screen association relationship.

[0012] As a preferred embodiment of the video image processing-based splicing display method of the present invention, the specific steps for constructing the common update object set are as follows: Using each change-starting window as the starting point for expansion, the video windows affected by the window state changes are traced along the occlusion propagation relationship, and the same video window distributed on different physical displays and its corresponding cross-screen image blocks, splicing boundaries and display processing nodes are traced along the cross-screen association relationship to form an update candidate range; The newly added video windows within each update candidate range are used as the starting point for the next round of expansion. The update candidate range is then expanded along the occlusion propagation relationship and cross-screen association relationship until the number of video windows, cross-screen image blocks, and display processing nodes before and after the expansion no longer increases. The update candidate range after the expansion stops is then determined as the common update object set.

[0013] As a preferred embodiment of the video image processing-based splicing display method of the present invention, the specific steps for outputting the cross-screen splicing display image are as follows: Based on the video windows and display processing nodes within the common update object set, the video frame version and window state version of the cross-screen image blocks corresponding to each video window are collected. The cross-screen image blocks belonging to the same video window but distributed on different physical displays are paired for version matching. The video frame version and window state version of each group of cross-screen image blocks are verified respectively. When the verification results of each group are consistent, the common update object set is marked as a version consistent state. Under the same version status, a unified commit identifier is generated for the common update object set. The unified commit identifier is sent to each display processing node in the common update object set. After each display processing node completes the preparation for display of the corresponding cross-screen image block, it reports the ready status. When all display processing nodes have reached the ready status, each video window layer in the common update object set is submitted synchronously. When at least one of the following situations occurs: inconsistent cross-screen image block versions or not all display processing nodes are ready, the display state of each video window in the common update object set before this display switch is retained, and the cross-screen image blocks that have not been submitted are kept in the pending display state. Video windows that are not included in the common update object set are displayed and switched independently according to their respective video frame versions and window state versions, forming a cross-screen spliced ​​display screen.

[0014] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the video image processing-based splicing display method described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the video image processing-based splicing display method described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By analyzing the external motion trend of the video window and the internal motion trend of the image of the video to be displayed, the cross-screen migration range that the video window may enter into adjacent physical displays is predicted, and the migration prefetching node, prefetching image range and entry sequence are determined accordingly. This realizes the transformation of cross-screen display tasks from passive redistribution after changes occur to active preparation before changes occur, enabling the target display processing node to complete video reception, decoding and image cropping in advance, reducing screen delay, short-term gaps and missing boundary content during cross-screen migration. By tracing the occlusion propagation relationship formed by changes in the video window and the cross-screen association relationship of the same video window between different physical displays, a common update object set containing only the actually affected video window, cross-screen image block and display processing node is constructed to realize local collaborative update of associated display content, avoid cross-screen tearing, layer misalignment and mixing of old and new states, and reduce the blocking of local changes on the update of the entire spliced ​​image. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a video image processing-based splicing display method.

[0019] Figure 2 This is a schematic diagram of video window cross-screen migration prediction and prefetching.

[0020] Figure 3 A schematic diagram for constructing a set of objects for joint updates.

[0021] Figure 4 This is a diagram illustrating cross-screen image block version verification and overall display switching. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a splicing display method based on video image processing, including the following steps: S1. Obtain the layout relationship of the physical display screens, the video to be displayed and the corresponding video window, establish a logical display canvas according to the layout relationship of the physical display screens, map the video window to the logical display canvas, identify the actual visible range of the video window, determine the physical display screen and display processing node corresponding to the actual visible range, and generate a visible mapping record.

[0026] S1.1 Determine a unified coordinate direction and coordinate origin based on the arrangement position, display resolution, and splicing boundary of each physical display screen. Obtain the canvas offset and size conversion ratio of each physical display screen relative to the coordinate origin. Convert the display boundary of each physical display screen into the corresponding canvas range and establish a logical display canvas according to the distribution boundary of the entire canvas range.

[0027] The arrangement of physical displays is derived from the screen topology configuration formed during the installation and commissioning phase of the video wall. This configuration records the physical display identifier, arrangement position, adjacency relationship, video wall output refresh rate, installation direction, and corresponding display processing node for each display. The display resolution is read through the display interface connected to the display processing node; if the resolution is not found, it is obtained from the screen topology configuration or the physical display device parameters. The video wall boundary is determined based on the effective display boundary of each physical display, the spacing between adjacent screens, and the border compensation. The effective display boundary and the spacing between adjacent screens are derived from the physical display device parameters and the video wall installation dimensions, while the border compensation is derived from the image continuity calibration results formed during the installation and commissioning phase. The video wall output refresh rate is determined during the installation and commissioning phase from the refresh rates commonly supported by all physical displays forming the same logical display canvas. If a physical display does not support a corresponding video wall output refresh rate, it is not included in the same logical display canvas.

[0028] The origin of the coordinate system is determined based on the reference screen position, video wall installation reference position, or canvas reference position recorded in the screen topology configuration, and two mutually perpendicular coordinate directions are determined according to the arrangement direction of the physical display screens and the image output direction. When no independent reference position is configured, the effective display boundary start point of the physical display screen ranked first in the screen topology configuration is taken as the origin of the coordinate system. When the physical display screens are rotated or have different image output directions, the corresponding in-screen coordinates are converted to a unified canvas coordinate direction.

[0029] The canvas coordinate precision originates from the canvas configuration saved during the installation and debugging phase of the video wall, and is used to represent the actual display length corresponding to a canvas coordinate distance. The ratio of the effective display width to the horizontal resolution is used as the horizontal pixel display scale, and the ratio of the effective display height to the vertical resolution is used as the vertical pixel display scale. The ratio of the horizontal pixel display scale to the canvas coordinate precision is used as the horizontal size conversion ratio, and the ratio of the vertical pixel display scale to the canvas coordinate precision is used as the vertical size conversion ratio. When the effective display size or display resolution of each physical display screen is different, the corresponding horizontal and vertical size conversion ratios are determined separately.

[0030] The canvas offset is determined based on the installation position of each physical display screen relative to the coordinate origin. When the screen topology configuration only records the adjacent relationship, the effective display size after conversion, the spacing between adjacent screens, and the border compensation amount are accumulated sequentially from the reference position along the adjacent relationship to determine the canvas offset of each physical display screen.

[0031] The in-screen coordinates within the effective display boundaries of each physical display screen are converted according to the size conversion ratio, and the corresponding canvas offset is added to obtain the canvas range of each physical display screen in a unified canvas coordinate reference. A logical display canvas is established according to the distribution boundary that can accommodate the entire canvas range, and the correspondence between the canvas range and the physical display screen, splicing boundary, and display processing node is saved; when there are areas on the physical display screens that are not configured with display devices, the corresponding areas are marked as non-displayable ranges and are not associated with display processing nodes.

[0032] S1.2. Take the video window associated with the video to be displayed as the display object in the logical display canvas, read the canvas position, display size and layer order corresponding to the video window, form the window mapping range according to the canvas position and display size, determine the common coverage part of the window mapping range and each canvas range, and deduct the overlapping part displayed by other video windows according to the layer order, and determine the retained image range as the actual visible range.

[0033] It should be noted that the video to be displayed originates from the network video stream or local video signal accessed by the video input terminal; the video input terminal configures the video to be displayed identifier for the accessed video and determines the complete image range of the video to be displayed based on the image width and image height of the video to be displayed; the video window is created by the splicing display control terminal to carry the corresponding video to be displayed, and a correspondence is established between the video window identifier and the video to be displayed identifier.

[0034] The canvas position, display size, layer order, and image cropping range of the video window are derived from the window creation, movement, scaling, layer adjustment, and image cropping adjustment commands issued by the splicing display control terminal; when no image cropping adjustment commands are issued, the complete image range of the video to be displayed is used as the image cropping range.

[0035] The splicing display control terminal generates a window state version when a video window is created. The window state version is updated when the canvas position, display size, layer order, or image cropping range of the video window changes. After calculating the actual visible range, the current actual visible range is compared with the actual visible range associated with the previous window state version of the corresponding video window. The window state version of the corresponding video window is updated when at least one of the following changes: the number of ranges, boundary coordinates, or range connectivity relationships. The window state version of the affected video window is also updated when the change in the actual visible range is caused by the creation, closing, moving, scaling, or layer order adjustment of other video windows.

[0036] When the actual visible range corresponding to the same window state is distributed across multiple physical displays, each visible mapping record uses the same window state version.

[0037] The positioning position of the window boundary is determined based on the canvas position of the video window, and the extension distance of the window boundary along the two canvas coordinate directions is determined based on the display size. The window boundary forms the window mapping range. The parts of the window mapping range that exceed the logical display canvas and fall into the non-displayable range are excluded. The common coverage part of the window mapping range and the corresponding canvas range of each physical display screen is determined respectively, forming the candidate visible part of the video window within each canvas range.

[0038] When video windows overlap, the display assignment of the overlapping parts is determined according to the layer order. For each video window, the overlapping parts displayed by other video windows in the candidate visible parts are identified as the occlusion range, and the occlusion range is subtracted from the candidate visible parts. When multiple occlusion ranges overlap, the overlapping parts are merged first and then the subtraction is performed. When the subtraction results in mutually separated ranges, the canvas coordinates and boundaries of each range are preserved.

[0039] The actual visible range is determined by the image range retained after the logical display canvas boundary is clipped, the undisplayable range is excluded, and the occlusion range is deducted. When the candidate visible part is completely covered by the occlusion range, the actual visible range is determined as an empty range, and the corresponding video window is marked as completely occluded. When other video windows change and re-form a non-empty actual visible range, the completely occluded state of the corresponding video window is canceled, and the window state version is updated according to the re-formed actual visible range.

[0040] S1.3. Divide the actual visible area according to the canvas range, convert the divided visible area from canvas coordinates to the in-screen coordinates of the corresponding physical display screen, and associate it with the display processing node responsible for the corresponding physical display screen to generate a visible mapping record.

[0041] It should be noted that the actual visible range is compared with the canvas range corresponding to each physical display screen. The common coverage area that is simultaneously located within the actual visible range and the canvas range is determined as the visible sub-range. When the actual visible range spans multiple canvas ranges, visible sub-ranges are formed according to the boundaries of each canvas range. When there are mutually separated visible parts within the same canvas range, the corresponding visible sub-ranges are retained respectively. Parts falling into the undisplayable range are not included in the division.

[0042] For each visible sub-range, the canvas offset of the corresponding physical display screen is used as the position conversion reference. Based on the horizontal and vertical size conversion ratios, the canvas coordinates of the visible sub-range are converted into the in-screen coordinates of the corresponding physical display screen. The in-screen coordinates are used to represent the starting position, occupied width, and occupied height of the visible sub-range within the effective display boundary. When the conversion result exceeds the effective display boundary, it is clipped according to the effective display boundary.

[0043] When the actual visible area is not empty, the corresponding physical display screen is determined according to the canvas area to which the visible sub-area belongs, and the display processing node responsible for the physical display screen is read from the screen topology configuration; a corresponding relationship is established between the video window identifier, the video identifier to be displayed, the canvas position of the video window, the display size, the layer order, the image cropping area, the window mapping area, the actual visible area, the canvas coordinates of the visible sub-area, the in-screen coordinates, the physical display screen identifier, the display processing node identifier, the generation time, and the window state version to form a visible mapping record.

[0044] When the actual visible area is empty, no visible sub-areas are divided, no conversion from canvas coordinates to screen coordinates is performed, and no association is made with the physical display screen and display processing node. A correspondence is established between the video window identifier, the video identifier to be displayed, the canvas position of the video window, the display size, the layer order, the image cropping range, the window mapping range, the empty actual visible area, the complete occlusion state, the generation time, and the window state version. The physical display screen identifier and the display processing node identifier are recorded as empty, forming a visible mapping record.

[0045] S2. Analyze the external motion trend of the video window based on the visible mapping record, extract the internal motion trend of the video image to be displayed, predict the cross-screen migration range based on the external motion trend of the video window, determine the prefetch image range corresponding to the cross-screen migration range in combination with the internal motion trend of the image, determine the migration prefetch node based on the correspondence between the cross-screen migration range and the physical display screen, and generate the migration prefetch record.

[0046] S2.1 Extract the canvas position, display size and generation time of the same video window from the visible mapping record, identify the most recent continuous change process based on the center displacement and boundary stretching direction of the adjacent window states, retain the window state in the most recent continuous change process, and form the external motion trend of the video window.

[0047] It should be noted that, based on the canvas position and display size in each window state, the center position, horizontal half-size, and vertical half-size of the video window are determined, and the horizontal and vertical displacements of the window center, as well as the changes in the horizontal and vertical half-sizes, are obtained by comparing window states generated at adjacent times.

[0048] Displacement and half-size changes smaller than the minimum coordinate unit of the logical display canvas are defined as invalid changes; when neither the horizontal nor vertical displacement components of the window center undergo a positive-negative direction change, the center movement direction of adjacent window states is defined as consistent; when the half-size of the window does not change between continuous increase and continuous decrease, the stretching direction of the corresponding window boundary is defined as consistent.

[0049] The splicing display control terminal records the canvas position and display size of the video window according to the window state sampling period, and determines the window state sampling period as the window state sampling interval. Starting from the window state with the latest generation time, the generation time interval of adjacent window states does not exceed two window state sampling intervals, and the center movement direction and the window boundary expansion and contraction direction are consistent. When these conditions are not met, the tracing stops.

[0050] The most recent continuous change process contains at least two window states with different generation times and effective positional or dimensional changes; if the corresponding conditions are not met, trend extrapolation is not performed, and the latest generation window state is used as the prediction basis. The generation time, window center position, horizontal half-size, and vertical half-size of each window state in the most recent continuous change process are correlated to form the external motion trend of the video window.

[0051] S2.2 Extract motion vectors of each image region from the continuous image frames of the video to be displayed, aggregate them according to the consistency of the direction of the motion vectors and the proportion of the corresponding image region, and combine the inter-frame time of adjacent image frames to determine the motion direction and motion amplitude of the image content, forming the motion trend inside the image.

[0052] It should be noted that the video to be displayed is parsed in the order of video frames to obtain continuous image frames, and the first frame number, last frame number, first frame time, last frame time and inter-frame time of adjacent image frames are recorded for motion analysis.

[0053] The image cropping area is divided into rectangular image regions of the same size. Parts located at the boundary of the image cropping area but not large enough to be fully divided are incorporated into adjacent image regions. For each image region in the previous image frame, a search range is established with the same position of the image region in the next image frame as the center. The normalized grayscale difference between each candidate image region and the corresponding image region within the search range is obtained. The candidate image region with the smallest grayscale difference is determined as the candidate matching image region.

[0054] Perform reverse matching from the candidate matching image region to the previous image frame. If the reverse matching yields the original image region, the two are determined as the matching image regions. If the reverse matching does not yield the original image region, or if there are multiple candidate matching image regions with the same gray level difference, cancel the corresponding matching relationship.

[0055] Based on the difference in center coordinates of the matched image region in adjacent image frames, the horizontal and vertical image coordinate displacements are determined. Combined with the corresponding inter-frame time, the horizontal and vertical motion per unit time are obtained to form the motion vector of the matched image region.

[0056] Motion vectors are grouped according to the positive and negative combinations of horizontal and vertical unit time motion quantities, and the area ratio of each matched image region in the image cropping range is determined as the contribution ratio. The unit time motion quantities of horizontal positive, horizontal negative, vertical positive, and vertical negative directions are aggregated according to the contribution ratio, without directly subtracting the motion quantities in the opposite direction.

[0057] When the total area of ​​the image regions that form a valid matching relationship in adjacent image frames is less than half of the area of ​​the image cropping range, the corresponding adjacent image frames are determined as the screen switching positions, and motion vectors are not aggregated across the screen switching positions.

[0058] By establishing a correlation between the aggregated motion direction, the amount of motion per unit time in each image coordinate direction, the corresponding image region proportion, the first frame number, the last frame number, and the inter-frame time, an internal motion trend of the image is formed.

[0059] S2.3. Predict the center position and display size of the video window based on the external movement trend of the video window to form a migration prediction range. The part of the migration prediction range that enters the canvas range of the adjacent physical display screen but does not belong to the actual visible range is determined as the cross-screen migration range.

[0060] It should be noted that the states of each window in the most recent continuous change process are read from the external motion trend of the video window and arranged according to the generation time; based on the canvas position and display size in each window state, the position of the center of the video window in the two canvas coordinate directions and the half size of the video window in the two canvas coordinate directions are determined respectively.

[0061] Read the bitrate, image resolution, and encoding format of the video to be displayed, and read the available transmission bandwidth, decoding frame rate, and image processing frame rate of each display processing node associated with the logical display canvas; calculate the processing time required for each display processing node to complete the transmission, decoding, image cropping, display adaptation, and preparation for display of an image frame, and convert the maximum value of each processing time upwards into an integer number of splicing output refresh cycles to form an extrapolated time period.

[0062] If the extrapolation period is less than one splicing output refresh cycle, it is processed as one splicing output refresh cycle; if the extrapolation period exceeds the duration of the most recent continuous change process, it is processed as the duration of the most recent continuous change process; if the duration of the most recent continuous change process is less than one splicing output refresh cycle, window state extrapolation is not performed.

[0063] The predicted state quantity at the end of the extrapolated period is calculated using the following expression: ; in, Indicates the first The generation time of each window state This represents the average time for generating the state of each window. Indicates the first The number of states to be predicted in each window state. This represents the average value of the corresponding state variable. Indicates the latest window state generated. The corresponding state variables in the data. Indicates the state from the latest generation window. The extrapolation period that begins to extend backward. This represents the predicted state quantity at the end of the extrapolation period; This indicates that the center of the video window is horizontal. Position in the canvas coordinate direction Indicates vertical direction Position in the canvas coordinate direction; Indicates the horizontal orientation of the video window Half size Indicates the vertical orientation of the video window Half size.

[0064] Based on the generation time, window center position, and window half-size of each window state in the most recent continuous change process, obtain the predicted window center position and predicted window half-size at the end of the extrapolation period; if the most recent continuous change process does not meet the extrapolation conditions, use the window center position and window half-size of the latest generated window state as the prediction result.

[0065] Using the center position of the prediction window as the range center, the boundary of the prediction window is determined according to the half size of the prediction window to form the migration prediction range. According to the splicing output refresh cycle, the period from the latest generated window state to the end of the extrapolation time period is divided into continuous prediction time points. The center position and half size of the video window corresponding to each prediction time point are obtained respectively, and the window ranges corresponding to each prediction time point are merged to form the migration process coverage range.

[0066] The common coverage area of ​​the migration process and the corresponding canvas area of ​​each physical display screen are determined separately. The actual visible area and the undisplayable area in the common coverage area are excluded, and the retained area is determined as the cross-screen migration range. When the migration process coverage area passes through multiple physical display screens continuously, the cross-screen migration range corresponding to each physical display screen is retained respectively.

[0067] S2.4. Match the cross-screen migration range with the canvas range of each physical display screen in the logical display canvas, divide the migration sub-range according to the part of the cross-screen migration range entering different canvas ranges, and convert each migration sub-range into the pre-fetched image range in the video to be displayed according to the image cropping ratio and display ratio of the video window.

[0068] It should be noted that the cross-screen migration range is compared with the canvas range corresponding to each physical display screen. The common coverage area that is simultaneously located in the cross-screen migration range and the canvas range is determined as the migration sub-range, so that each migration sub-range corresponds to one physical display screen. When there are mutually separate common coverage areas within the same canvas range, the corresponding migration sub-ranges are retained respectively.

[0069] From the continuous prediction time points, find the prediction time point where the video window range first co-occurs with the migration sub-range, and read the prediction window center position and prediction display size corresponding to that prediction time point; based on the positional relationship between the migration sub-range and the corresponding prediction window range, determine the relative position and occupied size of the migration sub-range in the prediction window range.

[0070] The predicted display ratio is formed based on the correspondence between the predicted display size and the image cropping range size. According to the predicted display ratio, the relative position and occupied size of the migration sub-range are converted into the image position and image size in the image cropping range. The starting position of the image cropping range in the complete image range of the video to be displayed is superimposed to form the basic prefetched image range.

[0071] The unit time motion amount corresponding to the horizontal positive, horizontal negative, vertical positive, and vertical negative directions is read from the internal motion trend of the image. The product of this value and the corresponding extrapolated time period is used as the boundary expansion amount in each image coordinate direction. The right boundary of the pre-fetched image range is expanded according to the horizontal positive motion amount, the left boundary is expanded according to the horizontal negative motion amount, the lower boundary is expanded according to the vertical positive motion amount, and the upper boundary is expanded according to the vertical negative motion amount.

[0072] When the boundary expansion result exceeds the range of the complete image of the video to be displayed, it is cropped according to the range of the complete image of the video to be displayed, and the image range retained after cropping is determined as the prefetch image range; when there is no effective internal motion trend of the image, the basic prefetch image range is directly determined as the prefetch image range; and a correspondence is established between the migration sub-range, the prefetch image range and the physical display screen identifier.

[0073] S2.5. Based on the external motion trend of the video window and the canvas distance from the video window boundary to each migration sub-range, determine the entry order of each migration sub-range, identify the display processing node responsible for the corresponding physical display screen as the migration prefetch node, associate the migration prefetch node, the prefetch image range and the entry order, and generate a migration prefetch record.

[0074] It should be noted that, for each migration sub-range, the window boundary facing that migration sub-range in the video window is determined according to the distribution direction of the migration sub-range relative to the predicted position of the video window, and the changes of the window center position, horizontal half-size and vertical half-size with the generation time are read according to the external motion trend of the video window.

[0075] Based on the predicted changes in the center position and display size of the video window during the extrapolation period, the position change process of the corresponding window boundary is determined, and the expected entry time corresponding to the first common coverage of the window boundary and the migration sub-range is obtained; the migration sub-ranges are arranged from earliest to latest according to the expected entry time to form an entry sequence relationship; when multiple migration sub-ranges correspond to the same expected entry time, the corresponding migration sub-ranges are grouped into the same entry sequence.

[0076] Based on the physical display screen identifier corresponding to the migration sub-range, the display processing node responsible for that physical display screen is read from the screen topology configuration, and this display processing node is identified as the migration prefetch node. When the migration prefetch node has already received the same video to be displayed and is handling the actual visible range of the video window, the existing video transmission and decoding results are reused, and a cropping task corresponding to the prefetched image range is added to the existing processing tasks; when the migration prefetch node has not yet received the video to be displayed, the corresponding video transmission and prefetch processing tasks are established.

[0077] Establish a correspondence between the video window identifier, the video to be displayed identifier, the physical display screen identifier, the migration sub-range, the prefetched image range, the migration prefetch node identifier, the canvas distance, the expected entry time period number, and the entry sequence relationship, and generate a migration prefetch record.

[0078] S3. Based on the visible mapping record and migration prefetch record, the video to be displayed is transmitted to the display processing node and the migration prefetch node. The video to be displayed is decoded, and image cropping and display adaptation are performed according to the actual visible range and cross-screen migration range to form a cross-screen image block group.

[0079] S3.1 Combining the actual visible range in the visible mapping record and the prefetched image range in the migration prefetch record, according to the position, display size and image cropping ratio of the video window in the logical display canvas, the actual visible range is converted inversely into the visible source image range in the video to be displayed, and the target in-screen range corresponding to the visible source image range and the prefetched image range are determined respectively.

[0080] It should be noted that visible mapping records and migration prefetch records are collected according to the video window identifier; the visible sub-range, canvas position of the video window, display size and image cropping range are read from the visible mapping records, and the prefetch image range and migration sub-range are read from the migration prefetch records.

[0081] Based on the positional relationship between the visible sub-range and the video window canvas position, determine its relative position and occupied size within the video window; based on the correspondence between the video window display size and the image cropping range size, convert the relative position and occupied size into local image coordinates within the image cropping range, and then superimpose the starting coordinates of the image cropping range within the complete image range of the video to be displayed to form the visible source image range.

[0082] The target screen range corresponding to the visible source image range is read from the visible mapping record. The target screen range corresponding to the prefetched image range is determined based on the migration sub-range, canvas offset, and size conversion ratio; when the conversion result exceeds the effective display boundary, it is truncated according to the effective display boundary.

[0083] Establish the correspondence between the visible source image range and the prefetched image range and the corresponding target screen range respectively; when the actual visible range of the video window is empty and in a completely occluded state, no corresponding visible source image range and display processing task are formed, the video input end continues to receive the video to be displayed, and retains the most recent keyframe and the image frames after the most recent keyframe.

[0084] After the complete occlusion is lifted, the visible source image range and the target screen range are reconstructed based on the updated visible mapping record. The corresponding processing tasks are assigned to the display processing node, and decoding is resumed from the most recent keyframe. After forming the display image block, the change start window recognition is performed.

[0085] S3.2. Based on the correspondence between the source image range and the target screen range, the processing tasks corresponding to the actual visible range are assigned to the display processing node, the processing tasks corresponding to the prefetched image range are assigned to the migration prefetch node, the source image ranges that are repeatedly covered are merged, and the adjacent target screen ranges are corrected according to the splicing boundary of the physical display screen.

[0086] It should be noted that, based on the display processing node identifier in the visible mapping record, the processing task corresponding to the visible source image range is assigned to the display processing node; based on the migration prefetch node identifier in the migration prefetch record, the processing task corresponding to the prefetch image range is assigned to the migration prefetch node. Each processing task carries the window state version, layer order, source image range, target in-screen range, and task type from the corresponding record.

[0087] When the same node processes the same video to be displayed, it merges overlapping source image ranges and retains the correspondence between the merged source image range and the range within each target screen; source image ranges assigned to different nodes are not merged.

[0088] For target screen areas located on both sides of the same splicing boundary and belonging to the same video window, the undisplayable parts are excluded based on the adjacent screen spacing and border compensation amount, and the boundaries of the target screen areas on both sides are corrected to make the corresponding source image positions continuously connected.

[0089] S3.3 The display processing node and the migration prefetch node decode the same video to be displayed, perform image cropping according to the allocated source image range, and complete size scaling and display format adaptation according to the target screen range. The processed image is associated with the corresponding physical display screen, screen position, video frame version and window state version to form a cross-screen image block group.

[0090] It should be noted that the display processing node and the migration prefetch node receive the video to be displayed according to the video identifier in the processing task, and decode it according to the video frame order; for each complete image frame obtained, the source frame number and source display timestamp are read from the video to be displayed, and the video identifier to be displayed, the source frame number, and the source display timestamp are used together to determine the video frame version.

[0091] Each node crops the image content according to the source image range in the processing task, and performs size scaling and display format adaptation according to the target screen range. The output parameters used for display format adaptation are read through the display interface. If the output parameters are not read, they are obtained from the screen topology configuration or physical display device parameters.

[0092] The image portion formed by the processing task corresponding to the actual visible range is marked as the display image block, and the image portion formed by the migration prefetch task is marked as the migration prefetch image block. The task type corresponding to each image block is recorded. The display image block enters the ready-to-display state, and the migration prefetch image block remains in the prefetch state and does not participate in the construction of cross-screen association relationships and common update object sets.

[0093] When the actual visible area of ​​the video window enters the migration sub-area, the source image range, target screen range, video frame version, and window state version of the migration prefetch image block are checked against the updated visible mapping record. If they match, the migration prefetch image block is converted into a display image block. If they do not match, image cropping and display adaptation are re-executed according to the updated visible mapping record.

[0094] For each image segment, a cross-screen image block identifier is generated, and the video window identifier, the video to be displayed identifier, the physical display screen identifier, the target screen range, the corresponding canvas coordinates, the source image range, the display processing node identifier, the task type, the video frame version, and the window state version are recorded. At the same time, the canvas position, display size, layer order, image cropping range, window mapping range, actual visible range, and generation time of the corresponding window state version are associated to form a cross-screen image block group.

[0095] S4. By using cross-screen image block groups, trace the occlusion propagation relationship and cross-screen association relationship caused by changes in video windows, determine the video windows that need to be updated together and their corresponding display processing nodes, and construct a set of common update objects.

[0096] S4.1. Compare the state versions of adjacent windows and compare the video frame versions of cross-screen image blocks according to the video window identifier. Determine the video windows whose canvas position, display size, layer order, and actual visible range have changed, the video windows whose video frame versions have changed and are distributed on different physical displays, and the video windows that have not yet completed their first display as the starting windows of the change.

[0097] It should be noted that cross-screen image blocks with the task type of display image block are selected from the cross-screen image block group. The window state version and video frame version associated with the selected cross-screen image blocks are collected according to the video window identifier, and arranged according to the generation time of the window state version and the source display timestamp corresponding to the video frame version. When the same window state version corresponds to multiple cross-screen image blocks, the canvas position, display size, layer order, window mapping range and actual visible range with the same window state version are merged into a group of window states.

[0098] For the same video window, select two adjacent window state versions for comparison; compare the canvas position, display size, layer order, and actual visible area in the two window state versions, and if at least one of them changes, the corresponding video window is determined as the video window with the changed window state.

[0099] When the window state version remains unchanged, the video frame versions of two adjacent sets of cross-screen image blocks in the same video window are compared according to the source display timestamp. When the video frame version changes and the cross-screen image blocks of the corresponding video window are distributed across two or more physical displays, the corresponding video window is identified as the video window for cross-screen video frame updates. When the video window is distributed across only one physical display, the corresponding video window is not included in the common update object set due to changes in the video frame version, and is updated independently by the corresponding display processing node according to the video frame version.

[0100] If a video window has only one window state version and has not yet completed its first display, the corresponding video window will be identified as the first video window to be displayed. The window mapping range and the actual visible range before the video window is created will be determined as empty ranges, and the window state version formed after creation will be taken as the window state version with the later generation time. If the first display has been completed and there is only one window state version, the first display will not be identified again due to insufficient number of window state versions.

[0101] The video window whose window state changes, the video window that updates cross-screen video frames, and the video window that is displayed for the first time are identified as the starting windows of the change. The change types corresponding to the window state change, cross-screen video frame update, and first display are recorded respectively. When the window state changes, the two window state versions with adjacent generation times are recorded. When the window is displayed for the first time, the empty range state before creation and the first window state version are recorded. When the cross-screen video frame is updated, the window state version and the video frame versions before and after the change are recorded.

[0102] S4.2 For windows whose states have changed or whose initial display has not yet been completed, based on the video window identifier, canvas position, and layer order in the cross-screen image block group, calculate the increase and decrease in coverage between the initial window and other video windows before and after the change, and trace the affected video windows layer by layer along the display ownership relationship determined by the layer order to form an occlusion propagation relationship.

[0103] It should be noted that the processing method for determining the occlusion propagation relationship based on the change type corresponding to the starting window is as follows: when the change type is a window state change, read the two window state versions with adjacent generation times and count the coverage increase and decrease range before and after the window state change; when the change type is the first display, take the empty range state before the video window is created as the window state with the earlier generation time, and take the first window state version as the window state with the later generation time, and count the coverage increase range formed by the first display.

[0104] When the change type is only cross-screen video frame update and the window state version has not changed, the statistics of coverage increase and coverage decrease will not be performed, the occlusion propagation relationship of the corresponding video window will not be formed, and the corresponding video window will be directly used for subsequent cross-screen association tracing.

[0105] Based on the video window identifier of the starting window of the change, read the window state version generated earlier and the window state version generated later, and obtain the corresponding window mapping range, actual visible range and layer order respectively; when the cross-screen image block is distributed across multiple physical displays, merge the canvas coordinates corresponding to the same video window identifier and window state version to restore the complete range of the video window in the logical display canvas.

[0106] The window mapping range of the change starting window in the two window state versions is merged into the coverage search range, and other video windows that have a common coverage area with the coverage search range in either window state version are found; the common coverage area between the change starting window and each video window is determined in the two window state versions, and which video window displays the common coverage area according to the corresponding layer order.

[0107] The portion displayed by the change-starting window in a later window state version but by other video windows in an earlier window state version is identified as the coverage increase range; the portion displayed by the change-starting window in an earlier window state version but by other video windows in a later window state version is identified as the coverage decrease range; the portion where the window mapping range remains overlapping but the display ownership shifts due to changes in layer order is classified into the coverage increase range or coverage decrease range according to the direction of the shift in display ownership.

[0108] The occluded video window corresponding to the increased coverage area and the redisplayed video window corresponding to the decreased coverage area are identified as the directly affected video windows; when the decreased coverage area does not fall within the window mapping range of other video windows, no occlusion propagation relationship is formed between video windows.

[0109] The directly affected video window and the changed portion of its actual visible area are used as the next layer of tracing objects. The common coverage area where its display ownership changes with other video windows is determined, and the source of the impact, the affected video window, the area of ​​increased or decreased coverage, the direction of the display ownership change, and the corresponding window state version are recorded. Tracing is repeated until no new occlusion impact records are formed. All records are then associated according to the order of impact transmission to form an occlusion propagation relationship.

[0110] S4.3 According to the window identifier, cross-screen image blocks distributed on different physical displays but belonging to the same video window are grouped into the same cross-screen association range. The video frame version and window state version of each cross-screen image block are retained, and the cross-screen association range is associated with the splicing boundary and the corresponding display processing node to form a cross-screen association relationship.

[0111] It should be noted that cross-screen image blocks with the task type of display image blocks are selected from the cross-screen image block group. The selected cross-screen image blocks are then aggregated according to the video window identifier. In the same video window aggregation result, the cross-screen image block identifier, physical display screen identifier, target screen range, corresponding canvas coordinates, display processing node identifier, video frame version, and window state version of each cross-screen image block are retained. If there are differences in the video frame version or window state version, the video window aggregation result is not split so that the version of cross-screen image blocks on different physical displays can be verified before display submission.

[0112] Based on the corresponding canvas coordinates of each cross-screen image block, determine its distribution range in the logical display canvas; when cross-screen image blocks belonging to the same video window are located in the corresponding canvas ranges of two or more physical displays, and the corresponding canvas ranges are connected sequentially through splicing boundaries, the canvas ranges covered by these cross-screen image blocks are classified into the same cross-screen associated range.

[0113] When cross-screen image blocks belonging to the same video window are located in unconnected canvas areas, cross-screen associated areas are formed according to the connection relationship between canvas areas, and spatially discontinuous cross-screen image blocks are not grouped into the same area; when the video window is distributed on only one physical display screen, the processing relationship between the corresponding cross-screen image block and the display processing node is retained, but no connection relationship is formed across the splicing boundary.

[0114] Read the splicing boundaries that each cross-screen association range passes through, and determine the physical display screen and corresponding display processing node connected to the splicing boundary; establish a correspondence between the cross-screen association range and the video window identifier, cross-screen image block identifier, physical display screen identifier, splicing boundary, display processing node identifier, video frame version and window state version to form a cross-screen association relationship.

[0115] S4.4. Using each change starting window as the starting point for expansion, trace the video windows affected by the window state changes along the occlusion propagation relationship, and trace the same video window distributed on different physical displays and the corresponding cross-screen image blocks, splicing boundaries and display processing nodes along the cross-screen association relationship to form an update candidate range.

[0116] It should be noted that update candidate ranges are established according to the change start window identifier, and the corresponding change start window is included in the corresponding update candidate range; for each change start window, occlusion impact records with the change start window as the source of impact are searched in the occlusion propagation relationship, and the directly affected video windows are included in the corresponding update candidate range, while the occlusion impact relationship between the change start window and the affected video windows is preserved.

[0117] According to the video window identifier in each update candidate range, the cross-screen association relationship is searched. The cross-screen image blocks belonging to the corresponding video window, the splicing boundary, and the display processing node responsible for the corresponding cross-screen image block are included in the corresponding update candidate range. The subordinate relationship between the video window and the cross-screen image block, the processing relationship between the cross-screen image block and the display processing node, and the splicing boundary connection relationship between the cross-screen image blocks are preserved.

[0118] The video window identifier and cross-screen image block identifier in each update candidate range are compared; when two update candidate ranges contain the same video window identifier or the same cross-screen image block identifier, the video window, cross-screen image block and display processing node in the two update candidate ranges are merged, and the traceability source and association relationship corresponding to each change starting window are retained.

[0119] If two update candidate ranges contain only the same display processing node, but not the same video window and cross-screen image block, the two update candidate ranges are not merged, and the processing relationship between the display processing node and the corresponding cross-screen image block is preserved respectively.

[0120] S4.5. Take the newly added video windows in each update candidate range as the starting point for the next round of expansion, and continue to expand the update candidate range along the occlusion propagation relationship and cross-screen association relationship until the video windows, cross-screen image blocks and display processing nodes before and after the expansion no longer increase. The update candidate range after the expansion stops is determined as the common update object set.

[0121] It should be noted that before each round of expansion begins, the video window identifier, cross-screen image block identifier, and display processing node identifier within each update candidate range are recorded respectively, and the video window newly included in the previous round within each update candidate range is used as the starting point for the current round of expansion of the corresponding update candidate range.

[0122] For each update candidate range, starting from the current expansion point, search for occlusion impact records in the occlusion propagation relationship that use the corresponding video window as the source of impact, and include the affected video windows that have not yet entered the corresponding update candidate range into the update candidate range; search for cross-screen associations according to the newly included video window identifiers, and include the corresponding cross-screen image blocks and display processing nodes into the update candidate range.

[0123] After each round of expansion, the video window identifiers and cross-screen image block identifiers in each update candidate range are cross-compared. When two update candidate ranges contain the same video window identifier or the same cross-screen image block identifier, the two update candidate ranges are merged, and the newly included video window after the merger is used as the starting point for the next round of expansion. When two update candidate ranges only contain the same display processing node, the two update candidate ranges are kept independent of each other.

[0124] If a new video window, cross-screen image block, or display processing node is added, or if there is a merging of update candidate ranges, the next round of expansion continues; if no more of the three types of objects are added within each update candidate range, and no more merging occurs between update candidate ranges, the expansion stops.

[0125] Each independent update candidate range after the expansion stops, along with the occlusion influence relationships, subordinate relationships, processing relationships, and splicing boundary connection relationships retained therein, are respectively determined as a common update object set.

[0126] For each set of common update objects, arrange the video windows according to their video window identifiers, and record the window state version, layer order, actual visible range, and the physical display screen and display processing node corresponding to the actual visible range of each video window in sequence to form a common update layout snapshot.

[0127] The splicing display control terminal generates a canvas layout version based on the generation time and layout sequence number of the common update layout snapshot, and establishes a correspondence between the canvas layout version and the common update object set identifier, video window identifier, cross-screen image block identifier, and display processing node identifier; when any record in the common update layout snapshot changes, the canvas layout version is updated.

[0128] S5. Verify the video frame version and window state version of the cross-screen image blocks within the common update object set, determine the overall display switching timing based on the readiness status of the display processing node, perform overall display switching on each video window layer within the common update object set, and update the video windows outside the common update object set according to their respective versions to form a cross-screen spliced ​​display.

[0129] S5.1 According to the video windows and display processing nodes in the common update object set, gather the video frame version and window state version of the cross-screen image blocks corresponding to each video window, match the cross-screen image blocks that belong to the same video window but are distributed on different physical displays, verify the video frame version and window state version of each group of cross-screen image blocks respectively, and mark the common update object set as version consistent when the verification results of each group are consistent.

[0130] It should be noted that the splicing display control terminal finds the corresponding video window, cross-screen image block and display processing node according to the common update object set, and sends a version verification request to each display processing node; the version verification request carries the common update object set identifier, canvas layout version, as well as the cross-screen image block identifier, video frame version and window state version assigned to the corresponding display processing node.

[0131] Each display processing node checks the cross-screen image block stored in its own node according to the version verification request; when the cross-screen image block identifier, video frame version and window state version are consistent with the version verification request, the corresponding cross-screen image block is associated with the canvas layout version in the version verification request, and the cross-screen image block identifier, video frame version, window state version and canvas layout version are fed back.

[0132] The feedback results of each display processing node are collected according to the video window identifier and the cross-screen association range to form a version verification group; for version verification groups that span multiple physical displays, the cross-screen image blocks on both sides are paired according to the splicing boundary connection relationship; when the video frame version and window state version of each pair are the same, the corresponding pairing result is recorded as consistent.

[0133] If all version verification groups within the shared update object set are consistent, and the canvas layout versions returned by each display processing node are the same as the canvas layout versions recorded in the shared update object set, the shared update object set will be marked as version consistent. If there are cross-screen image blocks that have not received feedback, inconsistent video frame versions, inconsistent window state versions, or inconsistent canvas layout versions, the shared update object set will be marked as version inconsistent, and the inconsistent cross-screen image blocks and display processing nodes will be recorded. Video frame version values ​​are not compared between different video windows.

[0134] S5.2 Under the condition of consistent version, a unified commit identifier is generated for the common update object set. The unified commit identifier is sent to each display processing node in the common update object set. After each display processing node completes the preparation for display of the corresponding cross-screen image block, it reports the ready status. When all display processing nodes reach the ready status, each video window layer in the common update object set is submitted synchronously.

[0135] It should be noted that when the common update object set is in a version-consistent state, the splicing display control terminal generates a unified submission identifier based on the generation time and switching sequence number of the overall display switch, and establishes a correspondence between the video window identifier, cross-screen image block identifier, display processing node identifier, video frame version, window state version, target screen range and layer order in the common update object set and the unified submission identifier.

[0136] The splicing display control terminal reads the splicing output refresh frequency from the screen topology configuration and sends continuous frame synchronization signals according to the splicing output refresh frequency, configuring sequentially increasing frame synchronization sequence numbers for each frame synchronization signal. Each display processing node inputs the frame synchronization signal to the corresponding display interface and reads the frame synchronization lock status returned by the display interface; after the display interface reaches the frame synchronization lock status, it matches the received frame synchronization sequence number with the display refresh process of the physical display screen frame by frame, and enters the frame synchronization lock status after continuously completing the matching.

[0137] Each display processing node locates the corresponding cross-screen image block based on the unified submission identifier, verifies the cross-screen image block identifier, video frame version, window state version, target screen range, and layer order, and writes the cross-screen image blocks that match the verification into the display buffer according to the layer order; when all corresponding cross-screen image blocks are ready to be displayed and the display processing node is in the frame synchronization locked state, it feeds back the unified submission identifier, the display processing node identifier, and the locked frame synchronization sequence number.

[0138] The splicing display control terminal collects the locked frame synchronization sequence number fed back by each display processing node in the common update object set, and reads the round-trip time of the control command measured when each display processing node is connected; obtains the ratio of the maximum value of each control command round-trip time to the splicing output refresh cycle, rounds it up, and then adds 1 to form the command advance frame number.

[0139] The maximum value of the locked frame synchronization sequence number fed back by each display processing node is added to the command advance frame number to determine the target frame synchronization sequence number. The unified submission identifier and the target frame synchronization sequence number are then sent to each display processing node. After each display processing node confirms that the target frame synchronization sequence number is later than the frame synchronization sequence number already locked by this node, the corresponding display buffer is marked as pending switching.

[0140] When the target frame synchronization number arrives, each display processing node will uniformly submit the identifier associated with the pending display cache to be switched to the display cache, update the layers of each video window in the common update object set according to the target screen range and layer order, and report the actual frame synchronization number of the display switch; the splicing display control terminal collects the actual frame synchronization numbers of the display switch reported by each display processing node; when the frame synchronization numbers reported by each display processing node are the same as the target frame synchronization number, the overall display switch is determined to be completed.

[0141] S5.3 When at least one of the following situations occurs: inconsistent cross-screen image block versions or not all display processing nodes have reached the ready state, the display state of each video window in the common update object set before this display switch is retained, and the cross-screen image blocks that have not been submitted are kept in the pending display state. Video windows that are not included in the common update object set are displayed and switched independently according to their respective video frame versions and window state versions to form a cross-screen spliced ​​display screen.

[0142] It should be noted that when the splicing display device starts up, the initial background image recorded in the screen topology configuration is written to the display cache of each physical display screen; if no initial background image is configured, a blank image is written to the display cache; when a newly created video window has not yet completed its first overall display switch, the display state before this display switch consists of the initial background image and other video window layers that have completed the display switch; when a video window has completed its overall display switch, the display state before this display switch is the video window layer saved in the display cache after the most recent completion of the display switch.

[0143] When the common update object set is in a state of version inconsistency, a unified commit identifier is not generated, and each display processing node does not switch the corresponding cross-screen image block; when the video frame versions are inconsistent, the video frame versions saved in each display processing node for the same video window are aggregated, and the video frame version that has been saved by each display processing node and has the latest source display timestamp is determined as the common video frame version, and version verification is re-executed according to the common video frame version; when there is no common video frame version, wait for the missing video frame to arrive.

[0144] When the window state versions are inconsistent, the window state version corresponding to the latest visible mapping record saved by the splicing display control terminal is used as the target window state version. Cross-screen image blocks corresponding to other window state versions that have not yet been displayed are marked as invalid, and cross-screen image blocks are re-formed according to the target window state version. When the canvas layout versions are inconsistent, the canvas layout version is regenerated according to the latest common update layout snapshot, and the version verification request is resent to each display processing node in the common update object set.

[0145] After the jointly updated object set reaches a version consistency state and a unified commit identifier is generated, if not all display processing nodes have reached the ready state, the unified commit identifier and the cross-screen image blocks that have been written to the display cache are retained. Display processing nodes that have not yet reached the ready state continue to perform cross-screen image block verification, layer arrangement and display cache writing.

[0146] Based on the video bitrate, image resolution, available transmission bandwidth, decoding frame rate, and image processing frame rate of each display processing node, calculate the processing time required for each display processing node to complete the video transmission, decoding, image cropping, display adaptation, and writing to the display buffer. Add the maximum value of each processing time to the maximum round-trip time of the control command, and add two splicing output refresh cycles to form the maximum waiting time.

[0147] When a set of jointly updated objects is marked as being in a version inconsistency state, the version waiting time is accumulated from the moment the version inconsistency state is formed. After the set of jointly updated objects reaches a version consistency state and generates a unified commit identifier, if there are any display processing nodes that have not yet reached the ready state, the node ready waiting time is accumulated from the moment the unified commit identifier is generated. The version waiting timer stops after the version is completed, and the node ready waiting timer stops after all display processing nodes have reached the ready state.

[0148] If the cumulative waiting time has not reached the maximum waiting time, after the video frame version, window state version, or canvas layout version is completed, the video frame version, window state version, and canvas layout version of the cross-screen image block in the common update object set are re-verified; if a unified submission identifier has been generated, after the display processing node reaches the ready state again, the frame synchronization sequence number fed back by each display processing node is collected, the target frame synchronization sequence number is determined, and the overall display switch is performed.

[0149] If the cumulative waiting time reaches the maximum waiting time and the version is still inconsistent, delete the cross-screen image blocks that have not formed a consistent version, do not generate a unified submission identifier, and re-execute the version verification after the new video frame or window state arrives; if the cumulative waiting time reaches the maximum waiting time and a unified submission identifier has been generated, but the display processing node has not yet reached the ready state, cancel the corresponding unified submission identifier, delete the cross-screen image blocks associated with the unified submission identifier that have not yet completed the display switch.

[0150] When a display processing node reaches its maximum waiting time and fails to reach the ready state or has disconnected, the corresponding display processing node is marked as an abnormal display processing node. The allocation of new processing tasks to the abnormal display processing node is suspended, and the displayable state of the corresponding physical display screen in the screen topology configuration is updated. The actual visible range and visible mapping records of the affected video window are recalculated, and cross-screen image block groups, cross-screen associations, and common update object sets are re-formed accordingly. After the abnormal display processing node reconnects and reaches the frame synchronization lock state again, the displayable state of the corresponding physical display screen is restored.

[0151] For video windows not included in the common update object set, the video window is treated as an independent update object. The video frame version and window state version of the display image block belonging to the same video window are verified. After the verification is consistent, the independent display switch is performed according to the target screen range and layer order. The display state before this display switch, which is retained in the corresponding area of ​​the common update object set, is output together with the video window layer that has completed the independent display switch outside the common update object set to form a cross-screen spliced ​​display.

[0152] This embodiment also provides a computer device applicable to the video image processing-based splicing display method, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the video image processing-based splicing display method proposed in the above embodiment.

[0153] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0154] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the video image processing-based splicing display method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0155] In summary, this invention analyzes the external motion trend of the video window and the internal motion trend of the image of the video to be displayed, predicts the cross-screen migration range that the video window may enter into adjacent physical displays, and determines the migration prefetching node, prefetching image range, and entry sequence accordingly. This transforms the cross-screen display task from passive redistribution after the change occurs to active preparation before the change occurs, enabling the target display processing node to complete video reception, decoding, and image cropping in advance, reducing screen delay, short-term gaps, and missing boundary content during the cross-screen migration process. By tracing the occlusion propagation relationship formed by the video window change and the cross-screen association relationship of the same video window between different physical displays, a common update object set containing only the actually affected video window, cross-screen image block, and display processing node is constructed to achieve local collaborative update of associated display content, thereby avoiding cross-screen tearing, layer misalignment, and mixing of old and new states, and reducing the blocking of the update of the entire spliced ​​image by local changes.

[0156] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A splicing display method based on video image processing, characterized in that, include: Obtain the layout relationship of the physical display screens, the video to be displayed and the corresponding video window, establish a logical display canvas according to the layout relationship of the physical display screens, map the video window to the logical display canvas, identify the actual visible range of the video window, determine the physical display screen and display processing node corresponding to the actual visible range, and generate a visible mapping record. The external motion trend of the video window is analyzed based on the visible mapping record, and the internal motion trend of the video image to be displayed is extracted. The cross-screen migration range is predicted based on the external motion trend of the video window, and the prefetch image range corresponding to the cross-screen migration range is determined by combining the internal motion trend of the image. The migration prefetch node is determined based on the correspondence between the cross-screen migration range and the physical display screen, and the migration prefetch record is generated. Based on the visible mapping record and the migration prefetch record, the video to be displayed is transmitted to the display processing node and the migration prefetch node, the video to be displayed is decoded, and image cropping and display adaptation are performed according to the actual visible range and cross-screen migration range to form a cross-screen image block group; By using cross-screen image block groups, we can trace the occlusion propagation relationship and cross-screen association relationship caused by changes in video windows, determine the video windows that need to be updated together and their corresponding display processing nodes, and build a set of common update objects. The system verifies the video frame version and window state version of the cross-screen image blocks within the common update object set, determines the timing of the overall display switch based on the readiness status of the display processing node, performs an overall display switch on each video window layer within the common update object set, and updates the video windows outside the common update object set according to their respective versions, thus forming a cross-screen spliced ​​display.

2. The splicing display method based on video image processing as described in claim 1, characterized in that, The specific steps for generating the visible mapping record are as follows: Based on the arrangement, resolution, and splicing boundaries of each physical display screen, a unified coordinate direction and coordinate origin are determined. The canvas offset and size conversion ratio of each physical display screen relative to the coordinate origin are obtained. The display boundaries of each physical display screen are converted into the corresponding canvas range, and a logical display canvas is established according to the distribution boundaries of the entire canvas range. The video window associated with the video to be displayed is used as the display object in the logical display canvas. The canvas position, display size and layer order corresponding to the video window are read. The window mapping range is formed according to the canvas position and display size. The common coverage part of the window mapping range and each canvas range is determined. The overlapping part displayed by other video windows is deducted according to the layer order. The retained image range is determined as the actual visible range. The actual visible area is divided according to the canvas area. The divided visible area is converted from canvas coordinates to the in-screen coordinates of the corresponding physical display screen, and associated with the display processing node responsible for the corresponding physical display screen to generate a visible mapping record.

3. The splicing display method based on video image processing as described in claim 2, characterized in that, The specific steps for predicting the cross-screen migration range are as follows: Extract the canvas position, display size, and generation time of the same video window from the visible mapping record. Identify the most recent continuous change process based on the center displacement and boundary stretching direction of the adjacent window states. Preserve the window states in the most recent continuous change process to form the external motion trend of the video window. Motion vectors of each image region are extracted from consecutive image frames of the video to be displayed. The motion vectors are aggregated according to the consistency of their direction and the proportion of the corresponding image region. The motion direction and amplitude of the image content are determined by combining the inter-frame time of adjacent image frames, thus forming the motion trend inside the image. Based on the external motion trend of the video window, predict the center position and display size of the video window to form a migration prediction range. The part of the migration prediction range that enters the canvas range of the adjacent physical display screen but is not within the actual visible range is determined as the cross-screen migration range.

4. The splicing display method based on video image processing as described in claim 3, characterized in that, The specific steps for generating migration prefetch records are as follows: The cross-screen migration range is mapped to the canvas range of each physical display screen in the logical display canvas. The migration sub-ranges are divided according to the part of the cross-screen migration range that enters different canvas ranges. Based on the image cropping ratio and display ratio of the video window, each migration sub-range is converted into the pre-fetched image range in the video to be displayed. Based on the external motion trend of the video window and the canvas distance from the video window boundary to each migration sub-range, the entry order of each migration sub-range is determined. The display processing node responsible for the corresponding physical display screen is determined as the migration prefetch node. The migration prefetch node, the prefetch image range and the entry order are associated to generate a migration prefetch record.

5. The video image processing-based splicing display method as described in claim 1 or 4, characterized in that, The specific steps for forming a cross-screen image block group are as follows: Combining the actual visible range in the visible mapping record and the prefetched image range in the migration prefetch record, the actual visible range is converted in reverse into the visible source image range in the video to be displayed, according to the position, display size and image cropping ratio of the video window in the logical display canvas, and the target in-screen range corresponding to the visible source image range and the prefetched image range are determined respectively. Based on the correspondence between the source image range and the target screen range, the processing tasks corresponding to the actual visible range are assigned to the display processing node, the processing tasks corresponding to the prefetched image range are assigned to the migration prefetch node, the source image ranges that are repeatedly covered are merged, and the adjacent target screen ranges are corrected according to the splicing boundary of the physical display screen. The same video to be displayed is decoded by the display processing node and the migration prefetch node, the image is cropped according to the allocated source image range, and the size is scaled and the display format is adapted according to the target screen range. The processed image is associated with the corresponding physical display screen, screen position, video frame version and window state version to form a cross-screen image block group.

6. The splicing display method based on video image processing as described in claim 5, characterized in that, The specific steps for tracing the occlusion propagation relationship and cross-screen correlation caused by changes in the video window are as follows: Compare the state versions of adjacent windows and compare the video frame versions of cross-screen image blocks according to the video window identifier. The video windows that have changed at least one of the following: canvas position, display size, layer order, and actual visible range; the video windows whose video frame versions have changed and are distributed on different physical displays; and the video windows that have not yet completed their first display are identified as the starting windows of the change. For windows whose states have changed or whose initial display has not yet been completed, based on the video window identifier, canvas position, and layer order in the cross-screen image block group, the increase and decrease in coverage between the initial window and other video windows before and after the change are calculated. The affected video windows are then traced layer by layer along the display ownership relationship determined by the layer order to form an occlusion propagation relationship. According to the window identifier, cross-screen image blocks distributed on different physical displays but belonging to the same video window are grouped into the same cross-screen association range. The video frame version and window state version of each cross-screen image block are retained, and the cross-screen association range is associated with the splicing boundary and the corresponding display processing node to form a cross-screen association relationship.

7. The splicing display method based on video image processing as described in claim 6, characterized in that, The specific steps for constructing the common update object set are as follows: Using each change-starting window as the starting point for expansion, the video windows affected by the window state changes are traced along the occlusion propagation relationship, and the same video window distributed on different physical displays and its corresponding cross-screen image blocks, splicing boundaries and display processing nodes are traced along the cross-screen association relationship to form an update candidate range; The newly added video windows within each update candidate range are used as the starting point for the next round of expansion. The update candidate range is then expanded along the occlusion propagation relationship and cross-screen association relationship until the number of video windows, cross-screen image blocks, and display processing nodes before and after the expansion no longer increases. The update candidate range after the expansion stops is then determined as the common update object set.

8. The splicing display method based on video image processing as described in claim 7, characterized in that, The specific steps for outputting the cross-screen spliced ​​display are as follows: Based on the video windows and display processing nodes within the common update object set, the video frame version and window state version of the cross-screen image blocks corresponding to each video window are collected. The cross-screen image blocks belonging to the same video window but distributed on different physical displays are paired for version matching. The video frame version and window state version of each group of cross-screen image blocks are verified respectively. When the verification results of each group are consistent, the common update object set is marked as a version consistent state. Under the same version status, a unified commit identifier is generated for the common update object set. The unified commit identifier is sent to each display processing node in the common update object set. After each display processing node completes the preparation for display of the corresponding cross-screen image block, it reports the ready status. When all display processing nodes have reached the ready status, each video window layer in the common update object set is submitted synchronously. When at least one of the following situations occurs: inconsistent cross-screen image block versions or not all display processing nodes are ready, the display state of each video window in the common update object set before this display switch is retained, and the cross-screen image blocks that have not been submitted are kept in the pending display state. Video windows that are not included in the common update object set are displayed and switched independently according to their respective video frame versions and window state versions, forming a cross-screen spliced ​​display screen.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the video image processing-based splicing display method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the video image processing-based splicing display method according to any one of claims 1 to 8.

Images