Multi-view stitching rendering and cross-view interaction methods and systems

By using a cross-view spatiotemporal prediction model and a candidate hypothesis evaluation mechanism, the problems of discontinuity in the image and breakage of interactive logic caused by changes in the pose of the main camera and user operations in multi-view stitching rendering are solved, achieving high-efficiency image quality and interactive response.

CN123123522APending Publication Date: 2026-11-17CHIMO FANTASY (CHENGDU) TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611453996.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing multi-view stitching rendering, when the pose of the main camera changes, the user operates across views, or the rendering load fluctuates, it is difficult to adjust the projection splitting, seam treatment, and resource allocation in a timely manner, resulting in problems such as discontinuous images, inconsistent colors between views, interrupted event routing, or incorrect touch attribution.

Method used

The projection parameters, seam status, interaction intent, and resource requirements of future multiple frames are predicted by a cross-view spatiotemporal prediction model. Candidate view configuration hypotheses are generated, and the master hypothesis is selected through cost evaluation. The projection segmentation, seam processing, and resource allocation of the view are pre-configured, and adjustments are made in real time to adapt to changes. The actual rendering results are collected to update the prediction model.

Benefits of technology

It achieves continuous multi-view stitching during changes in the main camera's pose and user operations, reducing screen jumps and interruptions in interactive logic, and improving interactive response efficiency and image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure FT_1
    Figure FT_1
  • Figure SMS_1
    Figure SMS_1
  • Figure SMS_13
    Figure SMS_13
Patent Text Reader

Abstract

This invention provides a method and system for multi-view stitching rendering and cross-view interaction, belonging to the field of multi-view stitching rendering and interaction technology. The method acquires the main camera pose, user interaction state, rendering load of each view, and historical prediction deviation; predicts future multi-frame projection parameters, seam states, interaction intentions, and resource requirements through a cross-view spatiotemporal prediction model; generates candidate view configuration hypotheses and evaluates their costs, selecting the primary hypothesis and retaining alternative hypotheses; pre-configures projection segmentation, seam processing, resource allocation, and event routing before rendering; renders each view in parallel, routing input events according to pointer identifiers and source physical view identifiers and locking touch affiliation; collects actual deviations to update the prediction model and hypothesis space, switching alternative hypotheses and transitioning rendering when switching conditions are met. This method can improve the continuity of multi-view stitching, the continuity of cross-view interaction, and resource adaptability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of multi-view stitching rendering and interaction technology, and more specifically, to a method and system for multi-view stitching rendering and cross-view interaction. Background Technology

[0002] In existing technologies, multi-view stitching rendering typically handles the projection segmentation, seam processing, and resource allocation of each view separately, while cross-view interaction largely relies on fixed coordinate mapping and event dispatch rules. When the main camera pose changes, users perform cross-view operations, or rendering load fluctuates, prediction and configuration often struggle to adjust in a timely manner, easily leading to problems such as discontinuities at seams, inconsistencies in color or geometry between views, interrupted event routing, or incorrect touch attribution. Furthermore, the lack of coordination between rendering resource allocation and interactive intent makes it difficult to ensure both image quality and interactive responsiveness. Therefore, a solution is needed that can integrate cross-view prediction, configuration, rendering, interaction, and feedback correction. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method for multi-view splicing rendering and cross-view interaction, the method comprising: Acquire the status information of the multi-view stitching scene, including the main camera pose, user interaction status, rendering load of each view, and historical prediction deviation; Based on the aforementioned state information, the projection parameters, seam status, interaction intent, and resource requirements of each view in future multiple frames are predicted using a cross-view spatiotemporal prediction model. Based on the prediction results, multiple candidate view configuration hypotheses are generated. Each candidate view configuration hypothesis includes projection offset, seam overlap parameters, resource allocation, and interaction route configuration. Evaluate the cost of each candidate view configuration hypothesis, select the master hypothesis, and retain alternative hypotheses; For the current frame to be rendered in the next few frames, pre-configure the projection segmentation, seam processing, resource allocation and event routing of each view according to the current master assumption before rendering; Perform multi-view parallel rendering, with each view rendered independently to its corresponding physical display unit. Input events are routed according to the pointer identifier and the source physical view identifier, and the touch attribute is locked. Collect the deviation between the actual rendering results and the prediction, and update the prediction model and hypothesis space based on the deviation; When the deviation meets the preset switching conditions, a candidate hypothesis is selected from the candidate hypotheses for switching, and the switched candidate hypothesis is used as the master hypothesis for the next frame. The pre-configuration is then re-executed before the next frame is rendered.

[0004] Furthermore, the step of predicting the projection parameters, seam status, interaction intent, and resource requirements of each view in future multiple frames based on the state information using a cross-view spatiotemporal prediction model includes: The cross-view spatiotemporal prediction model is constructed, which includes a projection parameter spatiotemporal field, a seam state spatiotemporal field, an interaction intent spatiotemporal field, and a resource requirement spatiotemporal field. The projection parameter spatiotemporal field, the seam state spatiotemporal field, the interaction intent spatiotemporal field, and the resource requirement spatiotemporal field take the main camera pose, the user interaction state, the rendering load of each view, and the historical prediction deviation as inputs. The projection parameters, seam status, interaction intent, and resource requirements of each view at the current and historical times are mapped to a unified spatiotemporal coordinate system to generate the spatiotemporal field of the projection parameters, the spatiotemporal field of the seam status, the spatiotemporal field of the interaction intent, and the spatiotemporal field of the resource requirements. Based on the rate of change of the main camera pose, the historical trajectory of the user interaction state, and the fluctuation trend of the rendering load of each view, the evolution direction and rate of the spatiotemporal field of the projection parameters, the spatiotemporal field of the seam state, the spatiotemporal field of the interaction intent, and the spatiotemporal field of the resource demand are determined. The historical prediction deviation is used as a correction term to dynamically adjust the prediction boundaries of the projection parameter spatiotemporal field, the seam state spatiotemporal field, the interaction intent spatiotemporal field, and the resource demand spatiotemporal field. The cross-view spatiotemporal prediction model outputs the projection parameters, seam status, interaction intent, and resource requirements of each view in future multiple frames based on the adjusted prediction boundary.

[0005] Furthermore, the step of using the historical prediction deviation as a correction term to dynamically adjust the prediction boundaries of the projection parameter spatiotemporal field, the seam state spatiotemporal field, the interaction intent spatiotemporal field, and the resource demand spatiotemporal field includes: The historical prediction deviations are classified according to the spatiotemporal field of the projection parameters, the spatiotemporal field of the seam status, the spatiotemporal field of the interaction intent, and the spatiotemporal field of the resource demand, respectively, to generate projection parameter deviation, seam status deviation, interaction intent deviation, and resource demand deviation. Each of the projection parameter deviation, seam status deviation, interaction intent deviation, and resource demand deviation contains the deviation component of the corresponding spatiotemporal field in the time dimension. Each of the categorized deviations is mapped to the unified spatiotemporal coordinate system to generate a deviation mineral deposit map, which is used to identify the distribution density and trend of the prediction deviations at each spatiotemporal location. Based on the areas in the deviation mineral deposit map where the deviation distribution density exceeds a preset density threshold or the change trend exceeds a preset trend threshold, high entropy areas are identified. The prediction boundary corresponding to the high-entropy region is expanded, and the prediction boundary corresponding to the non-high-entropy region is shrunk to generate an adjusted prediction boundary. Based on the adjusted prediction boundaries, update the boundary ranges of the projection parameter spatiotemporal field, the seam state spatiotemporal field, the interaction intent spatiotemporal field, and the resource demand spatiotemporal field.

[0006] Furthermore, the step of generating multiple candidate view configuration assumptions based on the prediction results includes: Based on the projection parameters, the seam state, the interaction intent, and the evolution direction and rate of the resource requirements, the sampling density of the hypothesis space is adjusted so that the sampling density increases in the region where the evolution rate exceeds a preset rate threshold and decreases in the region where the evolution rate is lower than the preset rate threshold. Based on the adjusted sampling density, multiple candidate view configuration hypotheses are generated in the hypothesis space. Each candidate view configuration hypothesis includes projection offset, seam overlap parameters, resource allocation, and interaction route configuration. The interaction probability and visual importance of each view are determined based on the interaction intent. Determine the estimated cost of each view based on the resource requirements described; Based on the interaction probability, the visual importance, and the estimated computational cost, a prior weight is assigned to each generated candidate view configuration hypothesis. The prior weight is used for subsequent cost evaluation and / or candidate hypothesis ranking.

[0007] Furthermore, the cost of evaluating the configuration hypotheses of each candidate view, selecting the master hypothesis, and retaining alternative hypotheses includes: For each candidate view, a cost function is constructed based on the hypothesis. The cost function includes geometric error terms, interaction error terms, delay error terms, resource error terms, and seam error terms. The cost function is a weighted sum of each error term. The weights of each error term and / or the order of the candidate hypotheses are determined based on the prior weights. The view proxy corresponding to each view calculates the local cost component of the view under the corresponding candidate view configuration assumption based on the local prediction results and local rendering load of the view. The coordinator receives the local cost components from each view agent and, in conjunction with global constraints, calculates the global cost of each candidate view configuration assumption. The candidate view configuration hypothesis with the lowest global cost is selected as the master hypothesis, and a preset number of candidate view configuration hypotheses are retained as alternative hypotheses in ascending order of global cost. When the preset switching conditions are met, a candidate hypothesis is selected from the candidate hypotheses in ascending order of global cost for switching.

[0008] Furthermore, the step of pre-configuring the projection segmentation, seam processing, resource allocation, and event routing of each view according to the current principal assumption before rendering includes: Based on the projection offset in the main assumption, an off-screen projection offset is applied to each view camera, and the complete view cone is divided into multiple sub-regions with strictly connected adjacent boundaries on the geometric projection, and each sub-region does not overlap with each other on the geometric projection. Based on the seam overlap parameters in the main hypothesis, an overlap band is constructed along the image space at the seam of adjacent views, wherein pixels from adjacent views are obtained within the overlap band by at least one of the following methods: performing boundary expansion rendering on adjacent views at the seam, sharing boundary textures, or reusing seam pixels of adjacent views by a compositor; the width and blending weight of the overlap band are determined, and temporal anti-aliasing dithering phase alignment parameters are configured. Based on the resource allocation in the main assumption, allocate graphics processing unit computing resources, rendering threads and transmission bandwidth to each view; Based on the interaction routing configuration in the master hypothesis, an event routing table is established with pointer identifier and view identifier as the joint routing key, and the user interaction intent prediction result is pre-associated with the hit detection benchmark of the target view.

[0009] In addition, the present invention also provides a multi-view splicing rendering and cross-view interaction system, which is used to store and execute the above-described methods.

[0010] Based on the above, this technical solution addresses the problems in existing multi-view stitching rendering where projection segmentation, seam processing, and resource allocation are independent, cross-view interaction relies on fixed mappings, and it is difficult to adapt to changes in the main camera pose, user cross-view operations, and rendering load fluctuations. It forms a closed loop of prediction, configuration, rendering, interaction, and feedback correction. By using a cross-view spatiotemporal prediction model to predict projection parameters, seam states, interaction intentions, and resource requirements for future multiple frames, projection segmentation, seam processing, resource allocation, and event routing for each view can be configured in advance before rendering, reducing temporary adjustments during the rendering stage. Through cost evaluation of candidate view configuration hypotheses and a master hypothesis and alternative hypothesis mechanism, configurations can be switched promptly when prediction deviations increase or resource states change. Transitional configurations can mitigate screen jumps caused by projection offsets, seam overlap parameters, and sudden changes in seam blending weights. By collecting the deviation between actual rendering results and predicted values, the prediction model, hypothesis space, and cost weights are updated respectively, enabling the system to continuously adapt to pose changes, interaction trajectories, and load fluctuations. By routing events according to pointer identifiers and the source physical view identifiers, and locking the touch-assigned view throughout the touch lifecycle, the continuity of interaction logic during cross-view swiping and multi-finger operations can be maintained, reducing touch attribution confusion and event interruptions. Cross-view collaborative gesture groups and a unified reference coordinate system support the recognition and distribution of cross-view gestures requiring the joint participation of multiple fingers. Dynamically adjusting the overlap width of seams and performing color correction can mitigate color differences and geometric discontinuities at seams between adjacent views. By combining interaction probability, visual importance, and estimated computational cost in resource allocation, both stitched image quality and interactive responsiveness can be balanced with limited computing power. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the method provided by the present invention. Detailed Implementation

[0012] like Figure 1 As shown in the figure. This technical solution provides a method and system for multi-view splicing rendering and cross-view interaction. The technical details are explained in detail below with reference to a specific application scenario.

[0013] This specific implementation uses a multi-screen splicing interactive system in an in-vehicle cockpit as an application scenario. The system includes three physical display units: an instrument panel physical display unit, a central control physical display unit, and a passenger-side physical display unit. These three physical display units are arranged horizontally to form a continuous splicing display area. Each physical display unit corresponds to one view. The main camera represents a virtual observation camera when the driver views the entire splicing display area. Users can perform click, drag, swipe, and multi-finger operations by touching the central control physical display unit or the passenger-side physical display unit. The system needs to maintain the continuity of the multi-view splicing screen and the continuity of cross-view interaction logic even with slight head movements, finger swipes across screens, and fluctuations in rendering load across views.

[0014] In this application, a multi-view stitching scene refers to a rendering and interaction scene in which multiple views jointly constitute a continuous display area. A view refers to an image channel rendered by a view camera and output to a physical display unit. The main camera refers to a virtual camera that determines the global viewing direction, stitching geometric reference, and world-space viewing relationship. Projection parameters refer to at least one of the view camera's projection matrix, view frustum parameters, projection offset, clipping plane, and viewport parameters. Seam status refers to at least one of the geometric connection error, image gradient, color difference, overlap band requirement, and historical deviation at the stitching boundary between adjacent views. Interaction intent refers to the prediction result of the physical view, target control, gesture type, and event routing target that the user may operate on in the future. Resource requirements refer to at least one of the graphics processing unit's computing resources, rendering threads, transmission bandwidth, video memory, and rendering latency.

[0015] I. Status Information Acquisition The system acquires the state information of the multi-view stitching scene before each rendering frame begins. This state information includes the main camera pose, user interaction state, rendering load of each view, and historical prediction deviation. The main camera pose can be provided by a head tracking module, eye tracking module, controller tracking module, or a preset observation point. The main camera pose includes at least one of position, posture, velocity, and angular velocity. The user interaction state includes at least one of the currently pressed pointer identifier, pointer coordinates, physical view identifier of the pointer source, historical touch trajectory, gesture stage, and interaction state machine state. The rendering load of each view includes at least one of graphics processing unit time, rendering thread utilization, rendering queue length, transmission bandwidth utilization, and video memory utilization. Historical prediction deviation includes the difference between predicted and actual values ​​over several past frames. The history window can be set to 8 to 32 frames. Historical prediction deviations are saved separately according to projection parameters, seam status, interaction intent, and resource requirements.

[0016] In one specific implementation, the instrument panel view has a low rendering load, the center console view has a high rendering load, and the passenger-side view has a medium rendering load. The user's finger slides from the center console physical display unit to the passenger-side physical display unit. The system obtains the pointer identifier of this touch event, the source physical view identifier (center console view), the current coordinates, and the historical trajectory. The system also obtains minute changes in the main camera's pose, as well as deviations in projection parameters, seam status, interaction intent, and resource requirements over the past 16 frames.

[0017] II. Cross-view spatiotemporal prediction The system constructs a cross-view spatiotemporal prediction model. This model includes the spatiotemporal field of projection parameters, the spatiotemporal field of seam status, the spatiotemporal field of interactive intent, and the spatiotemporal field of resource requirements. The spatiotemporal field refers to a data structure defined in a unified spatiotemporal coordinate system, used to represent the values, trends, and uncertainties of corresponding physical quantities in spatial location and time frame. The unified spatiotemporal coordinate system uses the upper left corner of the spliced ​​display area as its spatial origin, with horizontal and vertical coordinates normalized to 0 to 1, and the time axis represented by future frame numbers. Spatial location can correspond to a specific physical display unit, a specific view, a specific seam location, or a specific screen area.

[0018] The cross-view spatiotemporal prediction model takes the main camera pose, user interaction state, rendering load of each view, and historical prediction bias as input. The system maps the projection parameters, seam states, interaction intentions, and resource requirements of each view at the current and historical moments to a unified spatiotemporal coordinate system, generating spatiotemporal fields for projection parameters, seam states, interaction intentions, and resource requirements. During mapping, the local coordinates of each view are transformed to the unified spatiotemporal coordinate system based on the view's position in the spliced ​​display area, scaling relationship, and seam offset. The seam position is mapped to a boundary line or boundary band in the unified spatiotemporal coordinate system. In the time dimension, the current frame and historical frames are arranged by frame number.

[0019] The system determines the evolution direction and rate of four spatiotemporal fields based on the rate of change of the main camera pose, the historical trajectory of user interaction states, and the fluctuation trend of rendering load in each view. The evolution direction refers to the direction of change of the spatiotemporal field in both spatial and temporal dimensions. The evolution rate refers to the amount of change in the spatiotemporal field value within a unit frame. For example, when the main camera pose rotates to the right, the projection parameter spatiotemporal field expands to the right in the seam area between the center view and the passenger view. When the user's finger slides from the center view to the passenger view, the probability value of the interaction intent spatiotemporal field in the direction of the passenger view gradually increases. As the rendering load of the center view continues to increase, the value of the resource requirement spatiotemporal field in the corresponding area of ​​the center view gradually increases.

[0020] The system uses historical prediction bias as a correction term to dynamically adjust the prediction boundaries of the four spatiotemporal fields. The prediction boundary refers to the effective range of the spatiotemporal field in terms of time span, spatial extent, and confidence interval. During dynamic adjustment, historical prediction bias is categorized according to the four spatiotemporal fields, generating projection parameter bias, seam state bias, interaction intent bias, and resource demand bias. Each type of bias contains the bias component of the corresponding spatiotemporal field in the time dimension. The categorized biases are mapped to a unified spatiotemporal coordinate system to generate a bias map. The bias map is used to identify the distribution density and trend of prediction bias at each spatiotemporal location. The distribution density can be obtained by the ratio of the number of bias samples to the area of ​​the region within a unit spatiotemporal region. The trend can be obtained by the fitting slope of the bias samples in the time dimension or the difference between adjacent frames.

[0021] The system identifies high-entropy regions based on areas in the deviation mineral deposit map where the deviation distribution density exceeds a preset density threshold or the change trend exceeds a preset trend threshold. High-entropy regions indicate areas with high prediction uncertainty. The preset density threshold can be set to a normalized density of 0.6 to 0.8. The preset trend threshold can be set to 0.05 to 0.2 per frame. The prediction boundary corresponding to the high-entropy region is expanded, while the prediction boundary corresponding to the non-high-entropy region is contracted. The expansion ratio can be set to 1.1 to 1.5, and the contraction ratio can be set to 0.8 to 0.95. Based on the adjusted prediction boundaries, the boundary ranges of the four spatiotemporal fields are updated. The cross-view spatiotemporal prediction model outputs projection parameters, seam status, interaction intent, and resource requirements for each view in future multiple frames based on the adjusted prediction boundaries. These future multiple frames can be set to 3 to 8 frames.

[0022] III. Generation of Candidate View Configuration Assumptions The system generates multiple candidate view configuration hypotheses based on the prediction results. Each candidate view configuration hypothesis includes projection offset, seam overlap parameters, resource allocation, and interaction route configuration. A candidate view configuration hypothesis refers to a set of available view configuration parameter combinations. The hypothesis space refers to the set of parameters for the candidate view configuration hypotheses.

[0023] The system adjusts the sampling density of the hypothesis space based on the evolution direction and rate of projection parameters, seam status, interaction intent, and resource requirements. Sampling density refers to the number of candidate view configuration hypotheses generated per unit parameter area. The sampling density increases in regions where the evolution rate exceeds a preset rate threshold and decreases in regions where the evolution rate is below the preset rate threshold. The preset rate threshold can be set based on the median or mean of historical evolution rates. The sampling density in high-evolution-rate regions can be set to 2 to 5 times that of low-evolution-rate regions. Based on the adjusted sampling density, multiple candidate view configuration hypotheses are generated in the hypothesis space.

[0024] The system determines the interaction probability and visual importance of each view based on the interaction intent. Interaction probability refers to the probability that each view will be interacted with by the user within a certain number of future frames. Visual importance refers to the visible area, attention level, and main camera orientation weight of each view within the stitched image. The system determines the estimated computational cost of each view based on resource requirements. Estimated computational cost refers to the graphics processing unit time, thread usage, and bandwidth usage required to render the view. Based on the interaction probability, visual importance, and estimated computational cost, the system assigns prior weights to each candidate view's configuration hypothesis. These prior weights are used in at least one of subsequent cost evaluation and candidate hypothesis ranking.

[0025] In one specific implementation, a user's finger slides from the center console view to the passenger view. The spatiotemporal field of the interaction intent shows an increased probability of interaction with the passenger view. Assuming the generation module increases the sampling density near the projection offset, seam overlap parameters, and interaction routing configuration corresponding to the passenger view, five candidate view configuration hypotheses are generated. Candidate A increases the rendering resources of the passenger view, candidate B increases the seam overlap between the center console view and the passenger view, and candidate C pre-associates the event routing target to the passenger view.

[0026] IV. Cost Assessment of Candidate Hypotheses The system constructs a cost function for each candidate view configuration hypothesis. The cost function includes geometric error terms, interaction error terms, latency error terms, resource error terms, and seam error terms. The cost function is a weighted sum of these error terms. In a specific implementation, the global cost of the k-th candidate view configuration hypothesis can be expressed as:

[0027] in, This represents the global cost of the configuration hypothesis for the k-th candidate view. Represents the geometric error term. Indicates the interactive error term. This represents the delay error term. Indicates the resource error term. This indicates the seam error term. Represents the weight of the geometric error term. Indicates the weight of the interaction error term. Indicates the weight of the delay error term. Indicates the weight of the resource error term. This represents the weight of the seam error term. Each weight can be preset or adjusted based on prior weights. The sum of all weights can be normalized to 1.

[0028] The geometric error term measures the stitching geometric errors caused by projection offset and view frustum segmentation. The interaction error term measures the deviation between the event routing target and the predicted interaction intent. The latency error term measures the estimated rendering latency and event response latency. The resource error term measures the deviation between resource allocation and resource requirements. The seam error term measures color differences, gradient differences, and overlap band requirements at seams.

[0029] Each view's corresponding view proxy calculates the local cost component of that view under the corresponding candidate view configuration assumption based on the view's local prediction results and local rendering load. A view proxy is a rendering control object corresponding to a view, which can run in an independent thread, process, or graphics processing unit queue. The coordinator receives the local cost components from each view proxy and calculates the global cost for each candidate view configuration assumption in conjunction with global constraints. The coordinator is the scheduling module responsible for cross-view coordination. Global constraints include at least one of the following: total graphics processing unit budget, target frame rate, maximum allowable latency, total bandwidth limit, and synchronization barrier. The global cost can be a weighted sum of the local cost components of each view, or a combination of the maximum value of each view's local cost component and its weighted sum.

[0030] The system selects the candidate view configuration hypothesis with the lowest global cost as the primary hypothesis and retains a preset number of candidate view configuration hypotheses as alternative hypotheses, ordered by global cost in ascending order. The preset number can be set to 2 to 4. When a preset switching condition is met, an alternative hypothesis is selected from the alternative hypotheses in ascending order of global cost for switching. The weights of each error term and the order of the alternative hypotheses are determined based on prior weights. In one implementation, both weights and order are determined based on prior weights. In another implementation, only the order is determined based on prior weights, and the weights use fixed values.

[0031] V. Pre-configuration before rendering For the current frame to be rendered in the future, the system pre-configures the projection segmentation, seam handling, resource allocation, and event routing of each view according to the current master assumption before rendering. The current frame to be rendered refers to the frame that is about to enter the rendering pipeline. Pre-configuration refers to setting projection parameters, seam parameters, resource reservation, and event routing tables before the actual rendering command is submitted.

[0032] Based on the projection offset in the master hypothesis, the system applies an off-screen projection offset to each view camera. Off-screen projection offset refers to the offset applied to the projection matrix or view frustum of the view camera before the image is output to the physical display unit. The system divides the complete view frustum into multiple sub-regions with strictly contiguous boundaries in the geometric projection. These sub-regions do not overlap in the geometric projection. Each sub-region corresponds to the geometric projection range of a view. The boundaries between adjacent sub-regions are strictly contiguous in the geometric projection.

[0033] Based on the seam overlap parameters in the master hypothesis, the system constructs an overlap band along the image space at the seams of adjacent views. The non-overlapping geometric projections and the image space overlap band are not contradictory. Geometric projection segmentation ensures that each view covers different geometric regions, while the image space overlap band ensures smooth pixel transitions at the seams. Pixels from adjacent views are obtained within the overlap band through at least one of the following methods: boundary expansion rendering, shared boundary texture, or compositor reuse of seam pixels from adjacent views. Boundary expansion rendering refers to adjacent views additionally rendering several pixel columns or rows at the seams. Shared boundary texture refers to adjacent views sharing texture data for the seam region. Compositor reuse refers to a hardware or software compositor reading and blending the seam pixels from adjacent views. The system determines the width and blending weights of the overlap band and configures temporal anti-aliasing dithering phase alignment parameters. The temporal anti-aliasing dithering phase alignment parameters are used to ensure consistent dithering sampling phases in adjacent views at the seams, reducing flicker at the seams.

[0034] Based on the resource allocation in the master hypothesis, the system allocates graphics processing unit (GPU) computing resources, rendering threads, and transmission bandwidth to each view. Resource allocation can be adjusted according to the view's visual importance and interaction probability. Views with high interaction probability or high visual importance receive more GPU time and rendering threads. Upper and lower limits can also be set for resource allocation to prevent a single view from monopolizing resources.

[0035] Based on the interaction routing configuration in the master hypothesis, the system establishes an event routing table with pointer identifier and view identifier as the joint routing key. The event routing table is used to look up the corresponding interaction state machine based on the pointer identifier and the source physical view identifier. The system pre-associates the user interaction intent prediction results with the hit detection benchmark of the target view. The hit detection benchmark refers to the data structure used for ray detection, control detection, or collision detection, including at least one of the following: hierarchical bounding box, collider, UI control bounding box, depth map, and hit test cache.

[0036] VI. Multi-view parallel rendering and cross-view interaction Each view delegate independently constructs a scene view and submits it for rendering to its corresponding physical display unit based on the projection offset and rendering control parameters in the master hypothesis. Rendering control parameters include at least one of resolution, clipping plane, jitter phase, synchronization barrier, and framebuffer binding. The three views can be rendered in parallel. Each view is rendered independently to its corresponding physical display unit. In an alternative implementation, multiple views can also be rendered to a shared framebuffer first, and then output by the compositor to each physical display unit.

[0037] Before an input event enters the engine, it is marked with a source physical view identifier and a pointer identifier. The source physical view identifier indicates the physical display unit where the touch was pressed. The pointer identifier is a unique identifier for the touching finger, mouse pointer, or stylus. When a touch press event is detected, the system determines the source physical view identifier corresponding to the touch press event as the touch home view of that pointer, and locks the interaction state machine corresponding to the touch home view as the event routing target of that pointer within the current touch lifecycle. The touch lifecycle refers to the time period from touch press to touch release or touch cancellation. The touch home view refers to the physical view where the pointer was pressed.

[0038] During the touch lifecycle, the system predicts the target view identifier based on the historical touch trajectories and interaction intent predictions of the touch-owning view, and preloads the hit detection benchmark and interaction state machine corresponding to the target view identifier. The target view identifier refers to the physical view identifier that the finger is predicted to enter or operate on. Preloading refers to loading the hit detection benchmark, interaction state machine, and related resources into the cache or graphics processing unit in advance.

[0039] The system continuously routes subsequent input events within the touch lifecycle to the interaction state machine corresponding to the touch's home view, maintaining the continuity of interaction logic. Within the touch lifecycle, the system determines the physical view where the finger is currently located based on the current coordinates of the subsequent input event. The event coordinates are then back-projected into a world space ray using the projection parameters of the current physical view, and a hit detection is performed. A world space ray is a ray that originates from the viewpoint of the current physical view, passes through the screen point corresponding to the event coordinates, and enters the unified world space. The hit detection result is used to determine the manipulated object, control, or 3D position.

[0040] When the current physical view is inconsistent with the touch-assigned view, the system synchronizes the hit detection result to the interaction state machine corresponding to the touch-assigned view to maintain the continuity of interaction logic. Simultaneously, it calls the pre-loaded interaction state machine corresponding to the target view identifier, synchronizing the hit detection result to that state machine as well. In one implementation, the interaction state machine corresponding to the touch-assigned view acts as the main state machine, possessing the final response authority. The interaction state machine corresponding to the target view identifier acts as an auxiliary state machine, used for prediction, preloading, and state synchronization. The auxiliary state machine does not directly issue the final response, thus avoiding the same event being processed repeatedly by multiple state machines.

[0041] When the target view identifier matches the current actual physical view Figure 1When the target view identifier is not identical to the current physical view, hit detection is performed using the pre-loaded target view identifier as the baseline for hit detection, thus accelerating the process. If the target view identifier differs from the current physical view, hit detection is still performed using the projection parameters of the current physical view. This improves speed when predictions are correct and ensures coordinate accuracy when predictions are incorrect.

[0042] When a multi-finger operation is detected during the touch lifecycle, each finger is independently anchored to its initial touch home view, and event routing and hit detection are performed using a joint routing key of pointer identifier and view identifier. When multiple fingers cross, each finger maintains its own touch home view and interaction state machine. If a finger crosses into another physical view, the hit detection uses the projection parameters of the current actual physical view, but the event logic is still preferentially maintained by the touch home view state machine of that finger.

[0043] VII. Detailed Calculation of Joint Overlap Zone The system acquires the rate of change of the main camera pose, the image gradient at the corresponding seam location in the spatiotemporal field of the seam state, the inter-screen color difference between adjacent views, and the historical standard deviation of the corresponding seam location in the deviation mineral deposit map. The deviation mineral deposit map is generated by classifying historical prediction deviations according to projection parameters, seam state, interaction intent, and resource requirements, and mapping each classified deviation to a unified spatiotemporal coordinate system. The deviation mineral deposit map is used to identify the distribution density and trend of prediction deviations at each spatiotemporal location.

[0044] The system calculates the overlap band width at the seam between adjacent views based on the rate of change of the main camera pose, image gradient, inter-screen color difference, and the standard deviation of the historical deviation. The overlap band width increases with the increase of the rate of change of the main camera pose, image gradient, inter-screen color difference, and the standard deviation of the historical deviation. In one implementation, the overlap band width can be expressed as:

[0045] in, Indicates the width of the overlap band. Indicates the base width. This represents the rate of change of the main camera's pose. Represents the image gradient. Indicates color differences between screens. Indicates the historical deviation standard deviation. This represents the coefficient of pose change rate. Represents the image gradient coefficients. Indicates the color difference coefficient. This represents the historical deviation coefficient. The base width can be set from 0 pixels to 8 pixels. The pose change rate coefficient, image gradient coefficient, color difference coefficient, and historical deviation coefficient can be set empirically. The overlap band width can be set to an upper limit, for example, not exceeding 5% of the shorter side of the adjacent viewport, and not exceeding 64 pixels.

[0046] An overlap band is constructed at the seam between adjacent views based on the overlap band width. A blend weight is generated within the overlap band according to a normalized distance from the seam. The blend weight transitions smoothly at the seam. In one implementation, the blend weight can be represented as:

[0047] in, Indicates mixed weights, This represents the normalized distance from the seam. This represents the width of the overlap band. The blending weight tends to 0 at the edge of the overlap band and to 1 on the other side of the seam. Based on the blending weight within the overlap band, a weighted blending is performed on the pixels of adjacent views within the overlap band to generate the stitched image after seam processing. Blending can be performed in a linear color space or other color spaces.

[0048] VIII. Deviation Collection, Update and Switching After each frame is rendered, the system collects the actual projection parameters, actual seam status, actual interaction hit results associated with the current frame, and actual resource usage for each view. The system compares these actual values ​​with the predicted values ​​of the corresponding timestamps in the projection parameter spatiotemporal field, seam status spatiotemporal field, interaction intent spatiotemporal field, and resource requirement spatiotemporal field, generating projection parameter deviation, seam status deviation, interaction deviation, and resource deviation. The timestamps can correspond to frame sequence numbers, graphics processing unit timestamps, or display timestamps.

[0049] The system updates the model parameters of the cross-view spatiotemporal prediction model based on projection parameter deviation and seam state deviation, serving as the inner loop correction. Inner loop correction refers to the online adjustment of the prediction model's internal parameters. Model parameters include at least one of the following: neural network weights, statistical model parameters, filtering parameters, and rule parameters. The system updates the sampling density of the hypothesis space and the weights of each error term in the cost function based on interaction deviation and resource deviation, serving as the outer loop correction. Outer loop correction refers to the adjustment of the candidate hypothesis generation and cost evaluation rules.

[0050] The system counts the number of times each deviation exceeds its corresponding preset threshold within multiple consecutive frames. When the number of times any deviation exceeds its corresponding preset threshold reaches a preset threshold, the current master hypothesis is deemed invalid. This determination is included in the preset switching conditions. The preset threshold can be set to 3 to 8 times. In one implementation, the switching can also be determined by combining the global cost reduction level and the hysteresis coefficient. The hysteresis coefficient is used to control the sensitivity of the switching scheme. The larger the hysteresis coefficient, the more stable the switching, but the slower the response to changes in resource demand. The hysteresis coefficient can be set to 1.05 to 1.2.

[0051] After determining that the current master hypothesis is invalid, the system recalculates the global cost of the current candidate hypotheses based on the updated cost function, or regenerates candidate hypotheses based on the updated hypothesis space. The system selects a new master hypothesis from the current candidate hypotheses or the regenerated candidate hypotheses in ascending order of global cost. The new master hypothesis is determined as the master hypothesis for the next frame, and pre-configuration and rendering are performed according to the new master hypothesis before the next frame is rendered.

[0052] Before or during the rendering of the next frame, the system performs prediction, candidate generation, and evaluation for the frame immediately following it to update the set of candidate hypotheses. The master hypothesis selection result obtained from the prediction, candidate generation, and evaluation for the frame immediately following it does not overwrite the new master hypothesis already determined as the master hypothesis for the next frame, but rather serves as a candidate or alternative master hypothesis for the frame immediately following it. The system repeats the steps of deviation acquisition, updating, switching, pre-configuration, rendering, and prediction, candidate generation, and evaluation for subsequent frames in a frame-by-frame rolling manner. In one implementation, the frame pipeline depth is 2 to 3 frames. When the current frame is rendered, the master hypothesis for the next frame is already locked, and the master hypothesis for the frame immediately following it is in the candidate evaluation stage.

[0053] This application also provides a multi-view stitching rendering and cross-view interaction system, including a state acquisition module, a spatiotemporal prediction module, a hypothesis generation module, a hypothesis evaluation module, a pre-configuration module, a rendering and interaction module, a deviation acquisition and update module, and a switching module. The state acquisition module acquires the state information of the multi-view stitching scene. The state information includes the main camera pose, user interaction state, rendering load of each view, and historical prediction deviation. Based on the state information, the spatiotemporal prediction module predicts the projection parameters, seam states, interaction intentions, and resource requirements of each view in future multi-frames using a cross-view spatiotemporal prediction model. The hypothesis generation module generates multiple candidate view configuration hypotheses based on the prediction results. Each candidate view configuration hypothesis includes projection offset, seam overlap parameters, resource allocation, and interaction routing configuration. The hypothesis evaluation module evaluates the cost of each candidate view configuration hypothesis, selects the master hypothesis, and retains alternative hypotheses. The pre-configuration module pre-configures the projection segmentation, seam processing, resource allocation, and event routing of each view according to the current master hypothesis for the current frame to be rendered in future multi-frames. The rendering and interaction module performs parallel rendering of multiple views, with each view rendered independently to its corresponding physical display unit. Input events are routed according to pointer identifiers and source physical view identifiers, and touch assignment is locked. The deviation acquisition and update module collects the deviation between the actual rendering results and the prediction, and updates the prediction model and hypothesis space based on the deviation. When the deviation meets the preset switching conditions, the switching module selects a candidate hypothesis from the alternative hypotheses for switching, and uses the switched candidate hypothesis as the primary hypothesis for the next frame, re-executing the pre-configuration before rendering the next frame.

[0054] In alternative implementations, the view proxy and coordinator can be integrated within the same process or distributed across different computing nodes. Resource allocation can allocate only graphics processing unit time or allocate both threads and bandwidth simultaneously. Seam handling can employ only image space blending or a combination of boundary expansion rendering and shared boundary textures. Event routing can be based on a joint key of pointer and view identifiers, or gesture type or time window can be added as an additional key. The spatiotemporal prediction model can employ at least one of neural networks, Kalman filtering, particle filtering, or statistical regression. Overlap band width calculation can use linear combination, piecewise functions, or lookup tables. The number of frames predicted for future frames can be dynamically adjusted based on frame rate and computing power. The number of alternative hypotheses can be dynamically adjusted based on resource availability. All of the above alternative implementations remain consistent with the basic concept of this technical solution.

[0055] In the aforementioned in-vehicle cockpit multi-screen splicing interaction system, when multi-finger operation is detected during the touch lifecycle, cross-view collaborative gesture processing, master-slave switching transition, and seam overlap color correction can be further performed.

[0056] When multi-finger gestures are detected, the system first determines whether the fingers involved constitute a cross-view collaborative gesture. A cross-view collaborative gesture is a gesture that requires the coordinates of at least two fingers to be used in conjunction for recognition. For example, pinching to zoom with two fingers, rotating with two fingers, panning with two fingers, and swiping with three fingers are cross-view collaborative gestures. Single-finger taps, single-finger swipes, and taps with two fingers independently are not cross-view collaborative gestures.

[0057] During the judgment process, the system obtains the pointer identifier of each finger, the view to which the initial touch belongs, the current coordinates, and the touch trajectory. If there are two or more fingers, the touch time periods of each finger overlap, the gesture type belongs to a predefined set of collaborative gestures, and the coordinate changes of each finger meet the joint recognition conditions, then it is determined to constitute a cross-view collaborative gesture. The joint recognition conditions may include at least one of the following: changes in the distance between fingers, changes in rotation angle, and consistency of translation direction. If each finger operates independently in different physical views and there is no coordinate association required for joint recognition, then it is determined not to constitute a cross-view collaborative gesture, and each finger is still processed as an independent route.

[0058] When a cross-view collaborative gesture is formed, the system establishes a cross-view gesture group for each finger participating in the gesture and associates the pointer identifiers of each finger with the same gesture group identifier. A cross-view gesture group is a data structure that logically associates multiple fingers into a single gesture processing unit. The gesture group identifier is used to uniquely identify the gesture group. The cross-view gesture group can record the gesture group identifier, a list of pointer identifiers, the initial touch view to which each finger belongs, the current touch event, historical touch trajectories, and the gesture stage.

[0059] In a specific scenario, a user presses their left hand on the central control physical display unit and their right hand on the passenger-side physical display unit, simultaneously bringing both fingers together. The system detects this pinch-to-zoom gesture, establishes a cross-view gesture group, and associates the two pointer icons with the same gesture group icon. At this point, the central control physical display unit corresponds to the central control view, and the passenger-side physical display unit corresponds to the passenger-side view.

[0060] Throughout the touch lifecycle, the system maintains the initial touch home view of each finger unchanged. Even if a finger slides across the boundary of a physical display unit, the touch home view of that finger remains the same as the physical view it was in when it was pressed. For each finger within a cross-view gesture group, the system obtains its current touch event. The current touch event includes pointer identifier, current coordinates, event type, and timestamp.

[0061] The system backprojects the coordinates of each finger's current touch event to a unified reference coordinate system based on the projection parameters of the view to which the initial touch belongs. The unified reference coordinate system can be either the main camera's world coordinate system or the view coordinate system of a specified main view. The main camera's world coordinate system is the world space coordinate system established with the main camera as the observation reference. The view coordinate system of a specified main view is the local coordinate system of a view selected from multiple views, such as the selected control view as the specified main view. The selection method can be: the main camera's world coordinate system is used when the main camera's pose is valid and stable; the view coordinate system of the specified main view is used when the main camera's pose is unavailable or has a large deviation.

[0062] In a unified reference coordinate system, the system performs collaborative gesture recognition based on the touch trajectories of each finger within a cross-view gesture group, generating collaborative gesture commands. Collaborative gesture commands include at least one of the following: gesture type, operation amount, center point, target view identifier, and timestamp. Gesture types include zoom, rotation, translation, swipe, and combined gestures. Operation amounts include zoom ratio, rotation angle, and translation amount. The center point can be the geometric center of each finger's coordinates in the unified reference coordinate system. The target view identifier can be the view containing the geometric center of the view to which each finger belongs within the cross-view gesture group, the view with the highest predicted interaction intent value, or the view currently being focused on by the main camera.

[0063] The system distributes collaborative gesture commands to the interaction state machine corresponding to the target view, or the interaction state machines corresponding to the touch-assigned views of each finger within the cross-view gesture group, or both, based on the target view to which the collaborative gesture command is targeted and the touch-assigned view of each finger. In one implementation, if the target view of the collaborative gesture command is the central control view, the collaborative gesture command is distributed to the interaction state machine corresponding to the central control view. If the collaborative gesture crosses the central control view and the passenger view, the collaborative gesture command is distributed to the interaction state machines corresponding to the central control view and the passenger view, and one of these state machines is designated as the primary response state machine, while the other is designated as the synchronization state machine. The primary response state machine is used to execute the final operation, and the synchronization state machine is used to maintain the continuity of the interface state and interaction logic.

[0064] Within a cross-view gesture group, the system continues to route subsequent input events of each finger to the interaction state machine corresponding to its respective touch-owning view using the joint routing key of the pointer identifier and view identifier. The joint routing key is a routing lookup key composed of the pointer identifier and the identifier of the finger's initial touch-owning view. The system synchronizes the response results of the collaborative gesture command to the interaction state machine until the cross-view collaborative gesture ends. The conditions for the end of a cross-view collaborative gesture include at least one of the following: all participating fingers are lifted, all participating fingers are canceled, or the number of remaining fingers in the gesture group is less than the minimum number of fingers required for the collaborative gesture. After the gesture ends, the system releases the gesture group identifier and disconnects each pointer identifier from the gesture group identifier.

[0065] After determining that the current master hypothesis is invalid and selecting a new master hypothesis, directly changing the projection offset, seam overlap parameters, and seam blending weights according to the new master hypothesis may cause abrupt changes in the stitched image. Therefore, the system generates a transition configuration sequence before rendering the next frame.

[0066] The system acquires the projection offset, seam overlap parameters, and seam blending weights of the current master hypothesis as the starting configuration, and acquires the projection offset, seam overlap parameters, and seam blending weights of the new master hypothesis as the target configuration. The projection offset includes the projection matrix offset or view frustum offset of each view camera. The seam overlap parameters include at least one of the overlap band width, boundary expansion, and jitter phase alignment parameters. The seam blending weights include the blending ratio of each pixel within the overlap band.

[0067] The system generates a transition configuration sequence by performing temporal interpolation on the projection offset, seam overlap parameters, and seam blending weights between the starting and target configurations based on a preset number of transition frames. The preset number of transition frames can be set from 3 to 12 frames. The preset number of transition frames can be increased when the deviation is large, and decreased when resources are scarce or the frame rate is low.

[0068] In one implementation, the projection offset of the t-th transition frame can be expressed as:

[0069] in, This represents the projection offset of the t-th transition frame. This indicates the projection offset in the initial configuration. Indicates the projection offset in the target configuration. Indicates the preset number of transition frames. This indicates the current transition frame number. The seam overlap parameter and seam blending weight can use the same interpolation method. The seam overlap parameter can be expressed as:

[0070] in, This represents the seam overlap parameter for the t-th transition frame. This indicates the seam overlap parameter in the initial configuration. This represents the seam overlap parameter in the target configuration. The seam blending weight can be expressed as:

[0071] in, This represents the seam blending weight for the t-th transition frame. This indicates the seam blending weight in the initial configuration. This represents the seam blending weight in the target configuration. Interpolation methods can include linear interpolation, cosine interpolation, or sigmoid interpolation. Using cosine or sigmoid interpolation results in smoother transitions at the beginning and end of the transition phases.

[0072] Before rendering the next frame, the system pre-configures projection segmentation, seam handling, resource allocation, and event routing according to the new master assumption. Specifically, event routing immediately switches according to the interaction routing configuration of the new master assumption. This immediate switching is necessary because touch event routing requires high responsiveness; if event routing were still executed according to the old master assumption, pointer attribution errors or lag in the interaction state machine might occur. Projection segmentation, seam handling, and resource allocation can have target values ​​set according to the new master assumption, but actual rendering is performed according to the transition configuration sequence within the transition frame.

[0073] The system loads the transition configuration sequence into the rendering pipeline. Within each transition frame, the system performs rendering according to the corresponding transition configuration in the sequence. The seam blending weight gradually approaches the seam blending weight of the new master hypothesis as the number of transition frames increases. Once the preset number of transition frames is reached, the system performs rendering entirely according to the projection offset, seam overlap parameters, and seam blending weight of the new master hypothesis. If the master hypothesis is determined to be invalid again during the transition, the system can use the current transition configuration as the new starting configuration and regenerate the transition configuration sequence.

[0074] Before performing weighted blending on adjacent view pixels based on the blending weights within the overlap band, the system can also perform color correction on the pixels of adjacent views within the overlap band to reduce color differences between adjacent views.

[0075] The system acquires the color statistical characteristics of adjacent views within the overlap zone. These color statistical characteristics include at least one of average brightness, color histogram, and color space mean. Average brightness can be the mean of the brightness components of pixels within the overlap zone. The color histogram can be the distribution of pixels within the overlap zone across different color intervals. The color space mean can be the mean of pixels within the overlap zone in RGB, YUV, or Lab color spaces.

[0076] The system calculates the color deviation of adjacent views within the overlap zone based on color statistical characteristics and inter-screen color differences. The color deviation can be obtained by weighting at least one of the following: average brightness difference, color histogram distance, and color space mean difference. For example, the color deviation is obtained by normalizing the average brightness difference, color histogram distance, and color space mean difference of corresponding pixels in two adjacent views within the overlap zone, then summing them using weighted methods.

[0077] The system generates color correction parameters based on color deviation, the rate of change of the main camera pose, and the standard deviation of historical deviations. These parameters include at least one of brightness gain, color offset, or color transformation matrix. Brightness gain adjusts pixel brightness. Color offset adjusts color channel values. The color transformation matrix performs a linear transformation on the color channels.

[0078] In one implementation, the luminance gain can be expressed as:

[0079] in, Indicates brightness gain. Indicates color deviation. This represents the rate of change of the main camera's pose. Indicates the historical deviation standard deviation. This represents the color deviation coefficient. This represents the coefficient of pose change rate. This represents the historical deviation coefficient. The color deviation coefficient, pose change rate coefficient, and historical deviation coefficient can be set empirically or adjusted online. The brightness gain can be limited to a preset correction range, such as 0.9 to 1.1. The color offset can be limited to -0.1 to 0.1. The elements of the color transformation matrix can be limited to -0.1 to 0.1, or their norm can be limited to a preset value.

[0080] The color correction parameters increase with increasing color deviation, main camera pose change rate, and historical deviation standard deviation, but are limited within a preset correction range. This preset range is used to avoid over-correction that could lead to color distortion. If the color deviation, main camera pose change rate, or historical deviation standard deviation are small, the color correction parameters are close to uncorrected. If these values ​​increase, the color correction parameters increase in the correction direction, but do not exceed the preset correction range.

[0081] The system performs color correction on pixels in adjacent views within the overlap band based on color correction parameters, resulting in corrected adjacent view pixels. Color correction can be performed in the linear RGB color space, or in sRGB, YUV, or Lab color spaces. In alternative implementations, color correction can be performed on only one side of the adjacent views, or on both sides separately. The system then performs weighted blending on the corrected adjacent view pixels based on the blending weights within the overlap band, generating a stitched image with seam treatment. During weighted blending, the blending weights transition smoothly at the seams, thus maintaining brightness, color, and geometric continuity in the stitched image at the seams.

[0082] The above refers primarily to cross-view collaborative gesture processing, master hypothesis switching transitions, and seam overlap color correction, which can be combined with the aforementioned frame-by-frame scrolling process. During each frame's rendering, bias acquisition, prediction update, and master hypothesis switching, the system can enable the above processing as needed. In alternative implementations, cross-view collaborative gesture recognition can employ template matching, hidden Markov models, or neural network classifiers. Master hypothesis switching transitions can employ linear interpolation, cosine interpolation, or sigmoid interpolation. Color correction can employ gain correction, offset correction, or matrix correction. All of these alternative implementations remain consistent with the basic concept of this technical solution.

[0083] Further explanation is needed: The main camera pose is located in the main camera world coordinate system, with position represented by a 3D vector and attitude represented by a rotation matrix or quaternion. Historical prediction bias is normalized according to the corresponding physical quantity dimensions before entering the prediction model. Projection parameter bias is normalized according to the viewport width and height. Seam status bias is normalized according to the number of pixels or color difference. Interaction intent bias is normalized according to probability value or view identifier matching degree. Resource requirement bias is normalized according to budget proportion. Each view and physical display unit can correspond one-to-one, or multiple views can be output to different areas of the same physical display unit. View agents and coordinators can be deployed in the same process, or across threads, processes, or compute nodes. Before input events enter the engine, the engine can refer to at least one layer of the operating system input subsystem, graphics engine, or application framework. Timestamps uniformly adopt one of the following: frame sequence number, graphics processing unit timestamp, or display timestamp, and are aligned before comparing predicted and actual values. The current frame, next frame, and the next-next frame are arranged in display refresh order. The master hypothesis for the next frame is locked before the next frame is rendered. The master hypothesis in the next frame is only considered as a candidate or alternative and does not cover the master hypothesis in the next frame.

[0084] The unified spatiotemporal coordinate mapping can be represented as: , , ; in, and Represents spatial coordinates in a unified spacetime coordinate system. and Represents local coordinates of the view. and This indicates the offset of the view within the tiled display area. and Indicates the scaling factor. Represents a relative time coordinate. Indicates the current frame number. Indicates the starting frame number.

[0085] The evolution of the spatiotemporal field can be represented as .in, This represents any one of the following: the spatiotemporal field of projection parameters, the spatiotemporal field of seam status, the spatiotemporal field of interaction intent, or the spatiotemporal field of resource requirements. Indicates the rate of evolution. Indicates the time step. This represents the correction term obtained from historical prediction bias.

[0086] The distribution density of the deviation mineral deposit map can be expressed as: .in, Indicates the distribution density. This represents the number of deviation samples within a unit of spatiotemporal region. This indicates the area or volume of the region.

[0087] The trend of change can be represented as .in, Indicates the trend of change. This represents the deviation value of that spatiotemporal location.

[0088] The conditions for the high entropy region are or .in, This indicates a preset density threshold. This indicates a preset trend threshold. The high-entropy region is not defined by information entropy, but rather by a region with high prediction uncertainty.

[0089] The predicted boundary extension of the high-entropy region can be expressed as: The predicted boundary contraction in the non-high entropy region can be expressed as: .in, Indicates the prediction boundary before adjustment. Indicates the adjusted prediction boundary. This represents the adjustment factor, typically ranging from 0.05 to 0.3. The time-dimensional deviation component is constructed using the deviation sequence within the sliding window, first-order difference, and second-order difference. The deviation mineral deposit map can be generated using a unified spatiotemporal coordinate grid, with a grid resolution of 32×32×16.

[0090] Sampling density can be expressed as .in, This represents the sampling density of the k-th region. Indicates the basic sampling density. Indicates the influence coefficient of evolution rate. This indicates the evolution rate of the region.

[0091] The aforementioned frame-by-frame scrolling process forms a closed loop. Each frame executes state acquisition, cross-view spatiotemporal prediction, candidate hypothesis generation, cost evaluation, primary hypothesis and alternative hypothesis selection, pre-rendering pre-configuration, multi-view parallel rendering, event routing and hit detection, deviation acquisition, inner loop correction, and outer loop correction. If the deviation meets the preset switching conditions, the primary hypothesis is updated in the next frame, and projection offset, seam overlap parameters, and seam blending weights are executed according to the transition configuration sequence, and event routing is switched immediately. The touch-assigned view remains locked throughout the touch lifecycle. Cross-view gesture groups are released after the collaborative gesture ends. Color correction parameters are limited to a preset correction range. If a new primary hypothesis is unavailable or resources are insufficient, the previous valid primary hypothesis is maintained until a new primary hypothesis becomes available. The prediction results of the next frame are used to update the alternative hypothesis set, without overwriting the locked primary hypothesis for the next frame. The above supplements, combined with the aforementioned specific implementation methods, form a complete and implementable closed loop in terms of terminology, parameters, formulas, timing, and switching logic.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. The aforementioned stages, including state acquisition, cross-view spatiotemporal prediction, candidate hypothesis generation and evaluation, pre-configuration before rendering, multi-view parallel rendering, cross-view interaction, deviation acquisition and updating, master hypothesis switching, cross-view collaborative gesture processing, transition configuration, and seam color correction, can be executed cyclically in a frame-by-frame scrolling manner, forming a closed loop of prediction, configuration, rendering, feedback, and correction. Each module and step can be implemented entirely by software or by a combination of software and hardware. The number of views, the arrangement of physical display units, the type of prediction model, the weight of each error term in the cost function, the calculation method of the overlap band width, the setting method of color correction parameters, and the number of transition frames can all be adjusted according to specific application scenarios. Those skilled in the art can make various changes and substitutions without departing from the basic concept of the present invention. All such changes and substitutions should fall within the scope of protection of the present invention.

Claims

1. A method for multi-view stitching rendering and cross-view interaction, characterized in that, The method includes: Acquire the status information of the multi-view stitching scene, including the main camera pose, user interaction status, rendering load of each view, and historical prediction deviation; Based on the aforementioned state information, the projection parameters, seam status, interaction intent, and resource requirements of each view in future multiple frames are predicted using a cross-view spatiotemporal prediction model. Based on the prediction results, multiple candidate view configuration hypotheses are generated. Each candidate view configuration hypothesis includes projection offset, seam overlap parameters, resource allocation, and interaction route configuration. Evaluate the cost of each candidate view configuration hypothesis, select the master hypothesis, and retain alternative hypotheses; For the current frame to be rendered in the next few frames, pre-configure the projection segmentation, seam processing, resource allocation and event routing of each view according to the current master assumption before rendering; Perform multi-view parallel rendering, with each view rendered independently to its corresponding physical display unit. Input events are routed according to the pointer identifier and the source physical view identifier, and the touch attribute is locked. Collect the deviation between the actual rendering results and the prediction, and update the prediction model and hypothesis space based on the deviation; When the deviation meets the preset switching conditions, a candidate hypothesis is selected from the candidate hypotheses for switching, and the switched candidate hypothesis is used as the master hypothesis for the next frame. The pre-configuration is then re-executed before the next frame is rendered.

2. The multi-view stitching rendering and cross-view interaction method according to claim 1, characterized in that, Based on the state information, the prediction of projection parameters, seam status, interaction intent, and resource requirements for each view in future multiple frames using a cross-view spatiotemporal prediction model includes: The cross-view spatiotemporal prediction model is constructed, which includes a projection parameter spatiotemporal field, a seam state spatiotemporal field, an interaction intent spatiotemporal field, and a resource requirement spatiotemporal field. The projection parameter spatiotemporal field, the seam state spatiotemporal field, the interaction intent spatiotemporal field, and the resource requirement spatiotemporal field take the main camera pose, the user interaction state, the rendering load of each view, and the historical prediction deviation as inputs. The projection parameters, seam status, interaction intent, and resource requirements of each view at the current and historical times are mapped to a unified spatiotemporal coordinate system to generate the spatiotemporal field of the projection parameters, the spatiotemporal field of the seam status, the spatiotemporal field of the interaction intent, and the spatiotemporal field of the resource requirements. Based on the rate of change of the main camera pose, the historical trajectory of the user interaction state, and the fluctuation trend of the rendering load of each view, the evolution direction and rate of the spatiotemporal field of the projection parameters, the spatiotemporal field of the seam state, the spatiotemporal field of the interaction intent, and the spatiotemporal field of the resource demand are determined. The historical prediction deviation is used as a correction term to dynamically adjust the prediction boundaries of the projection parameter spatiotemporal field, the seam state spatiotemporal field, the interaction intent spatiotemporal field, and the resource demand spatiotemporal field. The cross-view spatiotemporal prediction model outputs the projection parameters, seam status, interaction intent, and resource requirements of each view in future multiple frames based on the adjusted prediction boundary.

3. The multi-view stitching rendering and cross-view interaction method according to claim 2, characterized in that, The step of using the historical prediction deviation as a correction term to dynamically adjust the prediction boundaries of the projection parameter spatiotemporal field, the seam state spatiotemporal field, the interaction intent spatiotemporal field, and the resource demand spatiotemporal field includes: The historical prediction deviations are classified according to the spatiotemporal field of the projection parameters, the spatiotemporal field of the seam status, the spatiotemporal field of the interaction intent, and the spatiotemporal field of the resource demand, respectively, to generate projection parameter deviation, seam status deviation, interaction intent deviation, and resource demand deviation. Each of the projection parameter deviation, seam status deviation, interaction intent deviation, and resource demand deviation contains the deviation component of the corresponding spatiotemporal field in the time dimension. Each of the categorized deviations is mapped to the unified spatiotemporal coordinate system to generate a deviation mineral deposit map, which is used to identify the distribution density and trend of the prediction deviations at each spatiotemporal location. Based on the areas in the deviation mineral deposit map where the deviation distribution density exceeds a preset density threshold or the change trend exceeds a preset trend threshold, high entropy areas are identified. The prediction boundary corresponding to the high-entropy region is expanded, and the prediction boundary corresponding to the non-high-entropy region is shrunk to generate an adjusted prediction boundary. Based on the adjusted prediction boundaries, update the boundary ranges of the projection parameter spatiotemporal field, the seam state spatiotemporal field, the interaction intent spatiotemporal field, and the resource demand spatiotemporal field.

4. The multi-view stitching rendering and cross-view interaction method according to claim 3, characterized in that, The assumptions for generating multiple candidate view configurations based on the prediction results include: Based on the projection parameters, the seam state, the interaction intent, and the evolution direction and rate of the resource requirements, the sampling density of the hypothesis space is adjusted so that the sampling density increases in the region where the evolution rate exceeds a preset rate threshold and decreases in the region where the evolution rate is lower than the preset rate threshold. Based on the adjusted sampling density, multiple candidate view configuration hypotheses are generated in the hypothesis space. Each candidate view configuration hypothesis includes projection offset, seam overlap parameters, resource allocation, and interaction route configuration. The interaction probability and visual importance of each view are determined based on the interaction intent. Determine the estimated cost of each view based on the resource requirements described; Based on the interaction probability, the visual importance, and the estimated computational cost, a prior weight is assigned to each generated candidate view configuration hypothesis. The prior weight is used for subsequent cost evaluation and / or candidate hypothesis ranking.

5. The multi-view stitching rendering and cross-view interaction method according to claim 4, characterized in that, The cost of evaluating the configuration hypotheses of each candidate view, selecting the master hypothesis and retaining alternative hypotheses, includes: For each candidate view, a cost function is constructed based on the hypothesis. The cost function includes geometric error terms, interaction error terms, delay error terms, resource error terms, and seam error terms. The cost function is a weighted sum of each error term. The weights of each error term and / or the order of the candidate hypotheses are determined based on the prior weights. The view proxy corresponding to each view calculates the local cost component of the view under the corresponding candidate view configuration assumption based on the local prediction results and local rendering load of the view. The coordinator receives the local cost components from each view agent and, in conjunction with global constraints, calculates the global cost of each candidate view configuration assumption. The candidate view configuration hypothesis with the lowest global cost is selected as the master hypothesis, and a preset number of candidate view configuration hypotheses are retained as alternative hypotheses in ascending order of global cost. When the preset switching conditions are met, a candidate hypothesis is selected from the candidate hypotheses in ascending order of global cost for switching.

6. The multi-view stitching rendering and cross-view interaction method according to claim 5, characterized in that, Before rendering, the pre-configuration of projection segmentation, seam handling, resource allocation, and event routing for each view according to the current master assumption includes: Based on the projection offset in the main assumption, an off-screen projection offset is applied to each view camera, and the complete view cone is divided into multiple sub-regions with strictly connected adjacent boundaries on the geometric projection, and each sub-region does not overlap with each other on the geometric projection. Based on the seam overlap parameters in the main hypothesis, an overlap band is constructed along the image space at the seam of adjacent views, wherein pixels from adjacent views are obtained within the overlap band by at least one of the following methods: performing boundary expansion rendering on adjacent views at the seam, sharing boundary textures, or reusing seam pixels of adjacent views by a compositor; the width and blending weight of the overlap band are determined, and temporal anti-aliasing dithering phase alignment parameters are configured. Based on the resource allocation in the main assumption, allocate graphics processing unit computing resources, rendering threads and transmission bandwidth to each view; Based on the interaction routing configuration in the master hypothesis, an event routing table is established with pointer identifier and view identifier as the joint routing key, and the user interaction intent prediction result is pre-associated with the hit detection benchmark of the target view.

7. The multi-view stitching rendering and cross-view interaction method according to claim 6, characterized in that, The process involves performing multi-view parallel rendering, with each view rendered independently to its corresponding physical display unit. Input events are routed according to pointer identifiers and source physical view identifiers, and touch attribution is locked, including: Each view agent independently constructs a scene view and submits it for rendering to the corresponding physical display unit based on the projection offset and rendering control parameters in the master hypothesis; Before an input event enters the engine, it is marked with the source physical view identifier and pointer identifier; When a touch press event is detected, the source physical view identifier corresponding to the touch press event is determined as the touch home view of the pointer, and the interaction state machine corresponding to the touch home view is locked as the event routing target of the pointer within the current touch lifecycle; During the touch lifecycle, based on the historical touch trajectory of the touch-attributed view and the predicted value of the interaction intent, the target view identifier is predicted, and the hit detection benchmark and interaction state machine corresponding to the target view identifier are preloaded. Subsequent input events within the touch lifecycle are continuously routed to the interaction state machine corresponding to the touch-owning view according to pointer identifiers, so as to maintain the continuity of interaction logic; During the touch lifecycle, the physical view where the finger is currently located is determined based on the current coordinates of the subsequent input event. The event coordinates are back-projected into a world space ray using the projection parameters of the current physical view, and a hit detection is performed. When the current physical view is inconsistent with the touch-assigned view, the hit detection result is synchronized to the interaction state machine corresponding to the touch-assigned view to maintain the continuity of the interaction logic; and the interaction state machine corresponding to the preloaded target view identifier is called to synchronize the hit detection result to the interaction state machine corresponding to the target view identifier. Specifically, when the target view identifier is consistent with the current physical view, the hit detection is performed using the hit detection benchmark corresponding to the preloaded target view identifier to accelerate the hit detection; when the target view identifier is inconsistent with the current physical view, the hit detection is still performed based on the projection parameters of the current physical view. When a multi-finger operation is detected during the touch lifecycle, each finger is independently anchored to its initial touch home view, and event routing and hit detection are performed using the combined routing key of the pointer identifier and the view identifier.

8. The multi-view stitching rendering and cross-view interaction method according to claim 5, characterized in that, The process of collecting the deviation between the actual rendering results and the prediction, updating the prediction model and hypothesis space based on the deviation, and switching to the alternative hypothesis when the deviation meets preset switching conditions includes: After each frame is rendered, the actual projection parameters, actual seam status, actual interaction hit results associated with the current frame, and actual resource usage of each view are collected and compared with the predicted values ​​of the corresponding timestamps in the projection parameter spatiotemporal field, the seam status spatiotemporal field, the interaction intent spatiotemporal field, and the resource demand spatiotemporal field, respectively, to generate projection parameter deviation, seam status deviation, interaction deviation, and resource deviation. Based on the projection parameter deviation and the seam state deviation, update the model parameters of the cross-view spatiotemporal prediction model as an inner loop correction; Based on the interaction bias and the resource bias, update the sampling density of the hypothesis space and the weights of each error term in the cost function as outer loop correction; The number of times each deviation exceeds the corresponding preset threshold within multiple consecutive frames is counted. When the number of times any deviation exceeds the corresponding preset threshold reaches the preset number threshold, the current master hypothesis is determined to be invalid. The preset switching condition includes this determination. After determining that the current master hypothesis is invalid, the global cost is recalculated for the current candidate hypothesis based on the updated cost function, or candidate hypotheses are regenerated based on the updated hypothesis space, and a new master hypothesis is selected from the current candidate hypothesis or the regenerated candidate hypothesis in ascending order of global cost. The new master hypothesis is determined as the master hypothesis for the next frame, and pre-configuration and rendering are performed according to the new master hypothesis before the next frame is rendered; Before or during the rendering of the next frame, prediction, candidate generation, and evaluation are performed for the next frame after that to update the set of alternative hypotheses. The result of the selection of the master hypothesis obtained by performing prediction, candidate generation, and evaluation for the next frame after that does not cover the new master hypothesis that has been determined as the master hypothesis for the next frame, but is used as a candidate or alternative master hypothesis for the next frame after that. Repeat the above steps of deviation acquisition, update, switching, pre-configuration, rendering, and prediction, candidate generation, and evaluation for subsequent frames in a frame-by-frame rolling manner.

9. The multi-view stitching rendering and cross-view interaction method according to claim 6, characterized in that, The step of determining the overlap band width and blending weight at the seam of adjacent views based on the seam overlap parameter in the main assumption includes: The system acquires the rate of change of the main camera pose, the image gradient of the corresponding seam position in the spatiotemporal field of the seam state, the inter-screen color difference between adjacent views, and the historical deviation standard deviation of the corresponding seam position in the deviation mineral deposit map. The deviation mineral deposit map is generated by classifying historical prediction deviations according to projection parameters, seam state, interaction intent, and resource requirements, and mapping each classified deviation to a unified spatiotemporal coordinate system. It is used to identify the distribution density and trend of prediction deviations at each spatiotemporal position. The overlap width at the seam between adjacent views is calculated based on the rate of change of the main camera pose, the image gradient, the inter-screen color difference, and the standard deviation of the historical deviation, wherein the overlap width increases with the increase of the rate of change of the main camera pose, the image gradient, the inter-screen color difference, and the standard deviation of the historical deviation. Based on the width of the overlap band, an overlap band is constructed at the seam of the adjacent views, and a blend weight is generated within the overlap band according to the normalized distance from the seam, the blend weight transitioning smoothly at the seam. Based on the blending weights within the overlapping band, the pixels of the adjacent views within the overlapping band are weighted and blended to generate a stitched image after seam processing.

10. A multi-view stitching rendering and cross-view interaction system, characterized in that, include: The status acquisition module is used to acquire the status information of the multi-view stitching scene. The status information includes the main camera pose, user interaction status, rendering load of each view, and historical prediction deviation. The spatiotemporal prediction module is used to predict the projection parameters, seam status, interaction intent and resource requirements of each view in future multiple frames based on the state information and through a cross-view spatiotemporal prediction model. The hypothesis generation module is used to generate multiple candidate view configuration hypotheses based on the prediction results. Each candidate view configuration hypothesis includes projection offset, seam overlap parameters, resource allocation, and interaction route configuration. The hypothesis evaluation module is used to evaluate the cost of each candidate view configuration hypothesis, select the master hypothesis, and retain alternative hypotheses; The pre-configuration module is used to pre-configure the projection segmentation, seam processing, resource allocation, and event routing of each view according to the current master assumption before rendering, for the current frame to be rendered in multiple future frames. The rendering and interaction module is used to perform parallel rendering of multiple views. Each view is rendered independently to its corresponding physical display unit. Input events are routed according to the pointer identifier and the physical view identifier of the source and the touch attribute is locked. The deviation acquisition and update module is used to collect the deviation between the actual rendering results and the prediction, and update the prediction model and hypothesis space based on the deviation. The switching module is used to select one alternative hypothesis from the alternative hypotheses for switching when the deviation meets the preset switching conditions, and to use the switched alternative hypothesis as the master hypothesis for the next frame, and to re-execute the pre-configuration before the next frame is rendered.