An ar video fusion superimposition method and system for ultra-low latency

Abstract

The application provides an AR video fusion superimposition method and system for ultra-low delay, which comprises the following steps: obtaining a display refresh event and a picture submission timestamp, calculating a next refresh time and calculating a remaining available time interval, and generating display synchronization constraint information; dynamically screening a synchronizable superimposition information set based on a remaining budget and an estimated execution time of virtual information; analyzing anchor points to generate geometric data, aggregating resources to a texture atlas, and calculating fusion weights combined with parameters such as urgency and stability; finally, the virtual and real scene pixels are weighted and fused by using a pixel shader, and the display is submitted. Through accurate control of the time window and intelligent screening of the content, the end-to-end delay is effectively reduced, the picture lag and trailing are avoided, the priority presentation of key information is ensured, and the smoothness, real-time performance and hardware energy efficiency of AR display are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of augmented reality, and in particular relates to an AR video fusion and overlay method and system for ultra-low latency. Background Technology

[0002] As augmented reality technology is increasingly applied in scenarios such as emergency command, industrial inspection, public safety, and complex on-site operations, AR fusion and overlay services for real-time real-scene video are gradually becoming an important basic capability for on-site information presentation. Existing systems typically use real-scene video captured by cameras as a base map, complete processing, rendering, and output at the video frame level, and then combine positioning calibration and tag anchoring results to overlay virtual tags, guidance information, or warning information onto the corresponding real-scene locations for display. While these solutions work in typical interactive or display scenarios, they exhibit significant limitations in real-time applications, such as alerts. Firstly, the processing chain often operates on a "complete frame" basis. The generation and overlay of virtual information typically only become visible to the user after a frame is processed and enters the next refresh cycle, resulting in a substantial perceptual lag between alert triggering and visibility. Secondly, the refresh boundaries and submission queue behavior of the display chain itself have a decisive impact on "when it becomes visible," but existing technologies often treat this as an uncontrollable external condition. The fusion side typically only focuses on the algorithm's completion time, lacking a usable window determination aligned with the display cycle, leading to situations where "logically triggered, but not yet effective on display." Furthermore, the quantity, area coverage, and style effects of virtual overlay content vary considerably in complex dynamic scenarios. Without a unified organizational method oriented towards the display cycle, uncertainty in fusion and submission can easily arise under high loads, further amplifying the visibility delay and overlay instability of alert information. In summary, existing AR video fusion and overlay technologies still lack a mechanism that can explicitly incorporate the display cycle boundary into the fusion control and complete the overall process from selecting overlay information to organizing executable rendering payloads and finally displaying the data within that boundary, thus making it difficult to achieve millisecond-level synchronous presentation and on-site availability while ensuring spatial consistency. Summary of the Invention

[0003] The purpose of this invention is to design an AR video fusion and overlay method and system for ultra-low latency, which can achieve ultra-low latency virtual-real fusion through display cycle synchronization and dynamic filtering mechanisms; intelligently sort and render content based on multi-dimensional weights to ensure that key information is displayed first; and optimize graphics pipeline resource scheduling to effectively eliminate screen stuttering and ghosting, significantly improving the smoothness, accuracy and energy efficiency of the AR experience.

[0004] To achieve the above objectives, a first aspect of the present invention provides an AR video fusion and overlay method for ultra-low latency, the method comprising:

[0005] Obtain the display refresh event timestamp and the most recent screen submission timestamp, calculate the next refresh time based on the display refresh event timestamp, calculate the time difference between the next refresh time and the most recent screen submission timestamp as the remaining available time interval of the current display cycle, and generate display synchronization constraint information that includes the deadline and cycle availability determination results;

[0006] Receive the set of virtual overlay information to be overlaid, obtain the remaining budget based on the difference between the deadline in the display synchronization constraint information and the current system time, calculate the expected execution time for each virtual overlay information in the set of virtual overlay information to be overlaid, and combine the projection area ratio, style complexity parameter and urgency parameter, and filter the virtual overlay information that meets the condition that the remaining budget is greater than or equal to the expected execution time as the set of virtual overlay information that can be synchronously overlaid.

[0007] The anchor reference of each virtual overlay information in the synchronously overlayable virtual overlay information set is parsed to generate screen geometric data and written to the instance buffer. Text resources and icon resources are aggregated to the texture atlas. The fusion weight is calculated based on the urgency parameter, projection stability parameter, projection area ratio and style complexity parameter and written to the constant buffer. An AR video fusion overlay unit set containing the texture atlas, the instance buffer and the constant buffer is constructed.

[0008] The real-time scene frame buffer is bound, and the AR video fusion and overlay unit set is traversed according to the drawing order. The fragment shader is used to perform linear combination calculation of the virtual overlay pixel color and the real-time scene pixel color based on the fusion weight to obtain the output pixel color. The fused frame buffer is then submitted to the display system.

[0009] Further, the step of calculating the time difference between the next refresh time and the most recent screen submission timestamp as the remaining available time interval for the current display cycle includes:

[0010] Record the first and second refresh timestamps of two consecutive refresh events, and calculate the difference between the second refresh timestamp and the first refresh timestamp as the refresh interval;

[0011] The sum of the display refresh event timestamp and the refresh interval is determined as the next refresh time;

[0012] Obtain the timestamp of the most recent screen submission from the graphics compositing interface;

[0013] The remaining available time interval is obtained by performing a subtraction operation.

[0014] Further, the step of calculating the estimated execution time for each piece of virtual overlay information in the set of virtual overlay information to be overlaid includes:

[0015] Obtain the area weight coefficient, complexity weight coefficient, and urgency adjustment coefficient from the configuration coefficients;

[0016] The estimated execution time is obtained by summing the product of the projection area ratio and the area weight coefficient, the product of the style complexity parameter and the complexity weight coefficient, and the quotient of the product of the preset baseline time and the urgency adjustment coefficient and the urgency parameter.

[0017] The higher the value of the urgency parameter, the smaller the calculated result of the expected execution time.

[0018] Furthermore, after generating the synchronously overlayable virtual overlay information set, the method further includes:

[0019] The virtual overlay information in the set of synchronously overlayable virtual overlay information is sorted;

[0020] The sorting rules are as follows: priority is given to arranging according to the urgency parameter from high to low; if the urgency parameters are the same, they are arranged according to the proportion of the projected area from small to large.

[0021] The sorted order will be used as the drawing order index when constructing the AR video fusion and overlay unit set later.

[0022] Furthermore, the projection stability parameters are obtained through the following methods:

[0023] Obtain the coordinates of the center point of the projection area of ​​the virtual overlay information in the current frame;

[0024] Obtain the coordinates of the center point of the projection area of ​​the same virtual overlay information in the previous frame;

[0025] Calculate the norm of the difference between the coordinates of the center point of the projection region in the current frame and the coordinates of the center point of the projection region in the previous frame;

[0026] Divide the difference norm by the screen scale to obtain the displacement ratio, and subtract the displacement ratio from 1 to obtain the projection stability parameter.

[0027] Further, the calculation of the fusion weights includes:

[0028] Obtain the urgency ratio, projection stability ratio, projection area ratio, style complexity level, and dimensionless configuration coefficient;

[0029] The sum of the product of the urgency ratio and the first coefficient, and the product of the projected stability ratio and the second coefficient, is calculated as a positive contribution term.

[0030] The product of the projected area ratio, the style complexity level, and the third coefficient is calculated as a penalty term.

[0031] Calculate the difference between the positive contribution term and the penalty term;

[0032] If the difference is less than a preset lower threshold, the fusion weight is set to the preset lower threshold; if the difference is greater than a preset upper threshold, the fusion weight is set to the preset upper threshold; if the difference is between the preset lower threshold and the preset upper threshold, the difference is directly used as the fusion weight.

[0033] Furthermore, the aggregation of text resources and icon resources into the texture atlas includes:

[0034] Allocate a writable texture atlas for the current display cycle;

[0035] The font rasterization module generates a glyph bitmap based on the text string index and writes the glyph bitmap into the first slot of the texture atlas;

[0036] The icon bitmap is read from the resource cache based on the icon resource handle, and the icon bitmap is written into the second slot of the texture atlas;

[0037] The texture sampling coordinate range corresponding to each virtual overlay information is recorded in the instance buffer, and the texture sampling coordinate range points to the first slot or the second slot.

[0038] Furthermore, the calculation of the linear combination of the virtual overlay pixel color and the real-time real-scene pixel color using the fragment shader based on the fusion weights specifically follows the following calculation logic:

[0039] The output pixel color is equal to the product of the virtual overlay pixel color and the fusion weight, plus the product of the real-time real-scene pixel color and the difference between 1 and the fusion weight.

[0040] The virtual overlay pixel color is obtained by sampling the texture atlas by the fragment shader, and the real-time scene pixel color comes from the bound real-time scene frame buffer.

[0041] Furthermore, the method also includes a maintenance step for the fusion processing time estimate:

[0042] Record the start time and submission completion time of the most recent N fusion operations, where N is a preset integer greater than 1;

[0043] Calculate the time difference between the start time and the submission completion time of each fusion operation to obtain N time consumption data;

[0044] The estimated fusion processing time is obtained by performing an arithmetic average or a weighted average on the N time-consuming data.

[0045] When generating display synchronization constraint information, if the remaining available time interval is greater than or equal to the estimated fusion processing time, the cycle availability determination result of the current display cycle is marked as available; otherwise, it is marked as unavailable.

[0046] A second aspect of the invention provides an AR video fusion and overlay system for ultra-low latency, the system comprising:

[0047] The constraint generation module is used to obtain the display refresh event timestamp and the most recent screen submission timestamp, calculate the next refresh time based on the display refresh event timestamp, calculate the time difference between the next refresh time and the most recent screen submission timestamp as the remaining available time interval of the current display cycle, and generate display synchronization constraint information that includes the deadline and cycle availability determination results.

[0048] The information filtering module is used to receive a set of virtual overlay information to be overlaid, obtain the remaining budget based on the difference between the deadline in the display synchronization constraint information and the current system time, calculate the expected execution time for each virtual overlay information in the set of virtual overlay information to be overlaid, and filter the virtual overlay information that meets the condition that the remaining budget is greater than or equal to the expected execution time as a set of virtual overlay information that can be synchronously overlaid.

[0049] The unit construction module parses the anchor reference of each virtual overlay information in the set of synchronously overlayable virtual overlay information to generate screen geometric data and write it into the instance buffer, aggregates text resources and icon resources into the texture atlas, calculates the fusion weight based on the urgency parameter, projection stability parameter, projection area ratio and style complexity parameter and writes it into the constant buffer, and constructs an AR video fusion overlay unit set including the texture atlas, the instance buffer and the constant buffer;

[0050] The fusion display module is used to bind the real-time scene frame buffer, traverse the AR video fusion and overlay unit set according to the drawing order index, use the fragment shader to perform linear combination calculation of the virtual overlay pixel color and the real-time scene pixel color based on the fusion weight to obtain the output pixel color, and submit the fused frame buffer to the display system.

[0051] The beneficial technical effects of the present invention are at least as follows:

[0052] To address the aforementioned issues, this invention provides an AR video fusion and overlay method and system for ultra-low latency. Its core lies in transforming the time boundary of the display refresh cycle into a calculable and transferable display synchronization constraint, and using this constraint to run through the entire process of generating, organizing, and outputting virtual overlay content, so that warning information and virtual tags can be fused and overlayed and actually displayed within the current display cycle. Specifically, this invention establishes an available submission window and deadline for the current period based on display refresh events and screen submission events during runtime, forming a unified display synchronization constraint. Under this constraint, the feasibility of each virtual information to be overlaid is determined, and the spatially anchored positional consistency is combined with time budget constraints to generate a set of synchronously overlaid virtual information that can take effect in the current period. This set is further organized into fusion overlay units that can be directly executed by the graphics pipeline. Multiple overlay contents are aggregated into a small number of submitable payloads through structures such as texture atlases and instance buffers. Fusion weights are formed based on urgency, coverage, style complexity, and projection stability to improve readability and stability in complex dynamic scenes. Finally, pixel-level fusion is completed and submitted for display within the time window limited by the display synchronization constraint, so that virtual labels and warning information can be synchronously visible at millisecond level while maintaining the same position as the real scene. This reduces the worst-case latency from warning triggering to visibility and improves the on-site usage effect. Attached Figure Description

[0053] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0054] Figure 1 This is a flowchart of an AR video fusion and overlay method for ultra-low latency according to the present invention.

[0055] Figure 2 This is a framework diagram of an AR video fusion and overlay system for ultra-low latency according to the present invention. Detailed Implementation

[0056] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0057] In one or more embodiments, such as Figure 1 As shown, a method for AR video fusion and overlay for ultra-low latency is disclosed, the method comprising the following:

[0058] S1: Obtain the display refresh event timestamp and the most recent screen submission timestamp, calculate the next refresh time based on the display refresh event timestamp, calculate the time difference between the next refresh time and the most recent screen submission timestamp as the remaining available time interval of the current display cycle, and generate display synchronization constraint information including the deadline and cycle availability determination results;

[0059] Specifically, during the startup and continuous operation of the AR video fusion service, this step constructs display synchronization constraint information around the display refresh rhythm to characterize the remaining time interval within the current display cycle that can be used for fusion and submission. This constraint information is directly derived from the time relationship between the display system refresh event and the screen submission event. Through time difference calculation under a unified time reference domain, a time boundary object that can be directly used by subsequent steps is formed, enabling the system to determine whether the current cycle has the actual display conditions before generating virtual tags and warning information.

[0060] This step requires input including the timestamp of the refresh event. And the most recent screen submission timestamp The real-time scene frame sequence is acquired by the terminal camera acquisition module. The camera sensor outputs image signals, which are processed by the hardware image processing unit and then enter the operating system buffer. The application layer obtains frame data through the video acquisition interface via callback, and simultaneously records the system timestamp. The display refresh event timestamp is generated by the synchronization callback triggered by the display system each time a refresh is completed, and the application layer records the time in the callback function. The timestamp of the most recent frame submission is returned by the graphics compositing interface, indicating the time point when the fused result of the previous frame entered the display pipeline. All the above times are based on the same system clock source to ensure the consistency of time difference calculation.

[0061] After receiving the latest refresh event timestamp, the system marks this time as the start of the current display cycle. It then calculates the refresh interval based on the difference between the two most recent refresh times and uses this to predict the next refresh time. The refresh interval is obtained by subtracting two consecutive refresh timestamps; this value remains stable over a short period. The next refresh time is obtained by adding the refresh interval to the current refresh time. Finally, the remaining available time interval for the current cycle is calculated.

[0062] ;

[0063] in, The next refresh time is calculated based on the time difference between the two most recent refreshes. The timestamp of the most recent submission of the merged image to the display system; This indicates the remaining time interval between the most recent commit and the next refresh boundary in the current display cycle. The method for obtaining this information is: record the times of the two most recent refreshes. and After obtaining the refresh interval, extrapolate to get the next refresh time; The actual submission and return time from the graphics composition interface; The result of the difference calculation between the two is used for subsequent time determination.

[0064] The system also maintains an estimate of the fusion processing time. This value is obtained by performing a sliding statistical analysis on the processing times of the most recent fusion operations, and is used to characterize the stable processing time required for a complete fusion and submission operation. The statistical process is based on averaging or weighted averaging the differences between historical fusion completion times and submission times to form a stable estimate. If the current display period is deemed usable for a new fusion operation, its expiration time is recorded; otherwise, it is recorded as an unavailable period. This step ultimately outputs display synchronization constraint information. This information includes the end time of the current display period. And the result of determining whether the current period can be used for a new fusion and overlay operation, where From the next refresh time Direct assignment will be obtained soon. Write The deadline field. This object will serve as the time reference for generating synchronously overlaid virtual information in the next step, allowing subsequent processing to revolve around the display cycle.

[0065] S2: Receive the set of virtual overlay information to be overlaid, obtain the remaining budget based on the difference between the deadline in the display synchronization constraint information and the current system time, calculate the expected execution time for each virtual overlay information in the set of virtual overlay information to be overlaid, combined with the projection area ratio, style complexity parameter and urgency parameter, and filter the virtual overlay information that meets the condition that the remaining budget is greater than or equal to the expected execution time as the set of virtual overlay information that can be synchronously overlaid.

[0066] Specifically, this step displays the synchronization constraint information output in step one. As the sole time boundary, the virtual overlay information flow continuously generated by the business side will be... Transformed into a set that "can be merged and overlaid within the current display cycle and then displayed". Its core principle comes from the classic deadline constraint in the field of real-time systems and scheduling: for any task to be completed before the deadline, the sum of "current time + estimated execution time" must not exceed the deadline. Mapping this constraint to this scheme, the deadline is determined by... The estimated execution time corresponds to the processing time for "completing a virtual overlay information and real-time scene image fusion and submission". This step implements this constraint into an executable determination for each virtual overlay information, and adds additional terms strongly related to the use case of this solution to the determination: overlay area size, style complexity, and urgency modulation, so as to prioritize the display of warning information in the current cycle when the budget is tight at the end of the cycle in complex scenarios.

[0067] In the input data, At least include the deadline (Calculated within the same reference domain in step one and written into the structure field); This is a set of virtual overlay information to be superimposed, written by the business module through a queue or message bus, typically including two categories: virtual tags and early warning information. Each piece of virtual overlay information is denoted as... Its fields can be directly implemented as a structure in engineering: anchor reference (an anchor identifier associated with the positioning calibration and tag anchoring result table), presentation content reference (such as text string index, icon resource handle), style parameters (such as stroke switch, transparency level, blink switch), and urgency parameter. (Written by the early warning strategy module; virtual tags are written to a lower level according to configuration, and early warning information is written to a higher level according to configuration). This step also reads the current system time. Obtained from the same system clock interface as in step one, making and The remaining budget for the current period can be obtained by directly calculating the difference.

[0068] First, explicitly define the "remaining budget" in the deadline constraint as The initial source of this formula is the mathematical definition of time difference and the definition of slacktime in real-time scheduling: on the same time base, the difference between the deadline and the current time is the remaining available time budget. The calculation is as follows:

[0069] ;

[0070] in, Taken from The cutoff field; Taken from the current reading of the system clock; This is the budget for the current period from "now" to "end". Both sides of the formula are time quantities under the same time base, and the difference is still a time quantity, which is logical. This budget will be used as the basis for all subsequent judgments. A unified boundary is established, thus fully incorporating the output of step one into every decision of this step.

[0071] Then, it is necessary to obtain each one Estimated execution time Its construction originates from the additive overhead model (linear cost model) commonly used in engineering, frequently found in graphics rendering and real-time system execution time estimation: the total time consumption can be broken down into several measurable feature terms that are approximately linearly related to "area size" and "effect complexity," and their weights are obtained through offline calibration. Combining the scenario constraint of "early warnings first, labels second" in this scheme, an urgency modulation term is added to make early warning information more likely to enter the current cycle under the same budget. The derivation process starts from two engineering principles: "fusion time is affected by the coverage size of the overlay area" and "fusion time is affected by style complexity": the larger the overlay area, the more pixels need to participate in the blending calculation; the more complex the style (e.g., stroke, shadow, flicker), the more blending and branching processing is required. The urgency modulation term is based on... Form entry: when When the value is higher, this item decreases, indicating a greater tendency to approve within the same budget; when lower and When the value is large, this factor increases, indicating a greater tendency to postpone non-urgent tasks that consume a large portion of the budget. The final estimation formula is:

[0072] ;

[0073] in, for The projection area ratio on the current real-time scene is a proportional value; it is obtained by reading... The anchor point references the current world coordinate system and retrieves its position and orientation from the anchoring result table. The current camera projection matrix and view matrix are obtained from the rendering pipeline (the AR runtime / rendering engine outputs this matrix during each frame). The bounding geometry associated with the anchor point (point labels use the configured screen radius, and region labels use the configured planar range) is projected onto the image plane to obtain a two-dimensional bounding region. The pixel coverage of this region is divided by the total number of pixels in the entire frame to obtain the proportion. . This is a style complexity parameter, with a value that is dimensionless and determined by... The style fields are mapped according to the configuration table. For example, "plain text", "text + stroke", "icon overlay", and "icon + blink" are mapped to different levels. In engineering, this mapping can be implemented as a table lookup: style flag combination key to complexity level value. The urgency parameter is a normalized level value, generated by the early warning strategy module. Write at time (e.g., mapped to a risk level table) (proportional value). The configuration coefficients are derived from the calibration of the fusion pipeline on the target device before deployment: under fixed real-scene frame conditions, fusion is performed on sample overlay items with different area ratios, different style levels, and different urgency levels, and the time sequence "from the start of fusion to the return of the submission call" is recorded. Then, the weights are obtained by least squares fitting; the fitted coefficients are fixed as configuration parameters and released with the version. The three terms on the right side of the formula are all linear combinations of "coefficient × dimensionless feature". and They are comparable at the same time base and meet the dimensional consistency requirements of the deadline constraint.

[0074] In obtaining With each of Then, the feasibility inequality of the deadline directly from the real-time scheduling can be simultaneously superimposed: substituting "current time + estimated execution time not exceeding the deadline" and rearranging the terms, we can obtain... During the project execution, for Iterate through each line to find the inequality. Joined The remaining entries are kept in the queue to await re-evaluation in the next display cycle. To ensure that subsequent build and overlay units fall more stably within budget, this step will also write... A deterministic ordering should be performed at this time: prioritizing by urgency. Sort by urgency from highest to lowest; if urgency is the same, sort by proportion of projected area. Sorting from smallest to largest makes it easier to include more high-value items when budgets are tight. This sorting doesn't require a new formula and can be implemented using a comparator: first compare... Compare again The comparison result is used in the sorting function.

[0075] The output of this step is a set of virtual overlay information that can be synchronously superimposed. Its elements all come from Furthermore, the original anchor references, content references, style fields, and urgency fields are retained unchanged. Only the feasibility of the deadline and the deterministic sorting are used to form the executable subset of the current cycle, which serves as the input for the next step of building the fusion and overlay unit.

[0076] S3: Parse the anchor reference of each virtual overlay information in the set of synchronously overlayable virtual overlay information to generate screen geometric data and write it into the instance buffer; aggregate text resources and icon resources into the texture atlas; calculate the fusion weight based on the urgency parameter, projection stability parameter, projection area ratio and style complexity parameter and write it into the constant buffer; construct an AR video fusion overlay unit set including the texture atlas, the instance buffer and the constant buffer.

[0077] Specifically, this step uses the set of synchronously overlaid virtual information output from step two. As input, each piece of virtual overlay information is used. The organization is a structured carrier that can directly enter the graphics pipeline to perform fusion and overlay, and aggregates to form a collection of AR video fusion and overlay units. Step two has already converted the display synchronization constraints into line-by-line constraints. The executable subset, therefore, this step further transforms the "information" into an "executable, drawable, and mergeable" execution structure: on the one hand, anchor references are resolved into screen geometry and written to the instance buffer; on the other hand, text / icon resources are aggregated into texture atlases to reduce submission batches, while also providing each... Generate fusion weight parameters and write them into a constant buffer so that the subsequent display stage can seamlessly overlay virtual labels and warning information onto the real-time scene within one or a few submissions.

[0078] For each Its urgency Projected area ratio Style complexity The definitions and values ​​from step two are used in all cases, and their meanings are not changed: The normalized configuration of warning level or label level comes from the business module. The result of the coverage ratio calculation in the image after anchor point geometric projection. The complexity level is derived from the style field mapped to the configuration table. This step additionally generates a projection stability. , used to reflect The degree of screen position jitter in adjacent frames. The calculation approach is as follows: use the position difference between adjacent frames to measure stability, and normalize it to... The proportions are used to combine with other proportion features. Specifically, the rendering engine obtains the proportions when building the overlay unit in the previous frame. The coordinates of the center point of the projection area are obtained and cached. In the current frame, the coordinates of the center point are obtained again, the norm of the difference between the two is calculated and normalized according to the screen scale to obtain the displacement ratio. Then, the complementarity is taken to obtain the stability. This normalization process only involves basic operations such as addition, subtraction, multiplication, and square root extraction, and the output is a dimensionless proportional quantity. Consistency facilitates combination within the same formula.

[0079] Fusion weights The design draws upon classic computer graphics compositing models as its foundational formulas. Firstly, the source-over model in the Porter-Duff compositing rule: the output color can be represented as a linear combination of the source and background colors with varying opacity coefficients; therefore, "opacity / weight" is naturally a proportional parameter used to control the overlay intensity. Secondly, the commonly used linear scoring model in engineering: multiple dimensionless features are linearly superimposed with weights to obtain an adjustable score. Based on these two classic starting points, this step will... Defined as "the positive contribution of urgency and stability minus the occlusion / overhead penalty caused by area and complexity", and clipped to the usable range using a clipping operator, we get:

[0080] ;

[0081] in, The urgency level is a percentage (written by the early warning strategy module or tag management module according to the configuration mapping). (fields) The projection stability ratio is obtained by normalizing the difference between the projection center points of adjacent frames and written into the instance record of this step. The proportion of the projected area (calculated in step two and followed by) transfer), The style complexity level (by (style field obtained from table lookup) The dimensionless configuration coefficients are fixed by the parameter tuning before the device goes online, for example, by performing grid search or linear regression fitting on "readability score / occlusion degree / fusion time" through multiple sets of scene recording data. The engineering implementation involves segmented trimming: if the result within the parentheses is less than the lower limit, the lower limit is used; if it is greater than the upper limit, the upper limit is used; otherwise, the original value is retained. Both sides of this formula are dimensionless proportional quantities. Dimensionless The coefficient is dimensionless. Since it is a dimensionless product, the quantity inside the parentheses and the quantity after clipping are... All values ​​are dimensionless proportions, consistent with common sense, and can be directly used as transparency / weight parameters in Porter-Duff source overlay compositing. Compared to the original Porter-Duff's "given transparency" approach, this formula adds an adaptive term driven by scene and task attributes: using Enhance the intensity of overlaying early warning information, using To suppress unstable superposition when jitter is large, use As a regular expression penalty term, it constrains the occlusion and rendering burden of large-area, highly complex content within the same cycle, thus better aligning with the goal of "real-time display of dynamic information in complex scenarios".

[0082] After obtaining each of Next, this step constructs the execution payload of the overlay unit. Screen geometry generation is achieved through the rendering engine's matrix transformation interface: reading... The anchor point reference is obtained from the anchoring result table to retrieve the anchor point spatial pose; according to The type selection encompasses the geometry template (point annotation template, region annotation template, etc., template parameters are from the configuration table). After instantiating the template in the anchor point coordinate system, it is mapped to the screen plane through the view matrix and projection matrix, generating a two-dimensional vertex array and writing it to the instance buffer. Texture resource aggregation adopts the texture atlas method: a writable atlas is allocated for the current display cycle. Text content is generated by the font rasterization module according to the string index to create glyph bitmaps and written to the atlas slot. Icon content is read by the resource cache according to the handle bitmap and written to the atlas slot. Each instance record is written to its atlas sampling coordinate range. Blending parameters are written to the constant buffer: Along with necessary color / stroke style parameters, these are written so that they can be directly used during source overlay compositing in the subsequent fragment shading stage. Multiple instance records sharing the same atlas texture and the same draw pipeline state are aggregated into a single overlay unit. This overlay unit contains an atlas reference, instance buffer reference, constant buffer reference, and draw order index. The draw order index follows the... The order (step two has already been organized according to urgency priority) ensures that information with high urgency is naturally at a higher level when synthesized in the same frame.

[0083] The output of this step is a collection of AR video fusion and overlay units. Each overlay unit contains an atlas texture reference, an instance buffer reference, a constant buffer reference, and a draw order index, and each... Corresponding fusion weights This is written into the directly executable rendering parameters. In the next step, this output is directly used to perform fusion and overlay on the real-time scene and output it for display, so that the synchronously overlayable information selected in step two can be presented within the current display cycle with minimal submission overhead and more stable readability.

[0084] S4: Bind the real-time scene frame buffer, traverse the AR video fusion and overlay unit set according to the drawing order index, use the fragment shader to perform linear combination calculation of the virtual overlay pixel color and the real-time scene pixel color based on the fusion weight to obtain the output pixel color, and submit the fused frame buffer to the display system.

[0085] Specifically, this step uses the AR video fusion and overlay unit set output from step three. As the sole input, it completes pixel-level fusion and overlay of the real-time scene within the current display cycle and submits the result to the display system. Each overlay unit already contains a texture reference, an instance buffer (containing a 2D vertex array and sampled coordinates), and blending weights. And the drawing order index. The real-time scene frame buffer is written to the GPU frame buffer after the frames captured by the camera are converted by the graphics pipeline, serving as the fusion base. This step completes the underlying frame binding, the drawing of each overlay unit, and the final submission operation within the same graphics context, so that the virtual labels and warning information can directly enter the display scanning process within the current refresh cycle.

[0086] The pixel blending formula originates from the Porter-Duff source-over model in computer graphics. The original expression of this model is: the output pixel color is equal to a linear combination of the source pixel color with its transparency and the target pixel color with its complement transparency. In the standard model, if the source color is... The target color is Transparency is Then output color satisfy This step will use the fusion weights obtained in step three. It is directly used as the transparency parameter, and the virtual overlay pixels are used as the source color. Real-time real-view pixels as target color The resulting fusion expression is:

[0087] ;

[0088] in, The value is obtained by the fragment shader through atlas texture sampling, and its value comes from the text or icon bitmap written into the atlas in step three. Real-world image textures bound to the current frame buffer; For step three, based on Calculate and write the proportion to the constant buffer; This is the pixel color vector that will ultimately be written to the target frame buffer. Both sides of this formula are color vectors. and It is a dimensionless proportional quantity that satisfies the common sense of linear combination and the consistency of dimension.

[0089] The fusion execution process is as follows: First, bind the real-time scene image as the underlying texture, and copy or directly use it as the initial content of the rendering target frame buffer; then, according to... The drawing order index is used to process overlay units sequentially from low priority to high priority. For each overlay unit, its atlas texture reference and instance buffer are loaded, and the constant buffer is set. Vertex shading and rasterization are performed, and the above blending formula is applied to each pixel within the coverage area during the fragment stage. Because... The units in the image have been sorted by urgency. High-urgency warning information is overlaid on the previously drawn ordinary labels in the later drawing stage, thus forming a visual hierarchy.

[0090] After all overlay units are drawn, the rendering interface of the graphics API is called to submit the current frame buffer. This interface adds the merged frame to the display queue, which is then scanned and output by the display system within the current refresh cycle. Since step one ensures a usable time budget for this cycle, steps two and three perform time and structure control on the overlay information. The submission action occurs before the cycle ends, thus enabling virtual tags and warning information to be presented synchronously with the real-time scene within the same display cycle. The entire process... Starting from the structured overlay unit, the pixel-level output is calculated using the Porter-Duff linear synthesis formula, and then enters the display refresh process through the graphics submission interface, forming a complete closed loop from algorithm derivation to physical display, achieving the final technical effect of ultra-low latency AR video fusion and overlay.

[0091] In one or more embodiments, such as Figure 2 As shown, an AR video fusion and overlay system for ultra-low latency is disclosed, the system comprising:

[0092] The constraint generation module is used to obtain the display refresh event timestamp and the most recent screen submission timestamp, calculate the next refresh time based on the display refresh event timestamp, calculate the time difference between the next refresh time and the most recent screen submission timestamp as the remaining available time interval of the current display cycle, and generate display synchronization constraint information that includes the deadline and cycle availability determination results.

[0093] The information filtering module is used to receive a set of virtual overlay information to be overlaid, obtain the remaining budget based on the difference between the deadline in the display synchronization constraint information and the current system time, calculate the expected execution time for each virtual overlay information in the set of virtual overlay information to be overlaid, and filter the virtual overlay information that meets the condition that the remaining budget is greater than or equal to the expected execution time as a set of virtual overlay information that can be synchronously overlaid.

[0094] The unit construction module parses the anchor reference of each virtual overlay information in the set of synchronously overlayable virtual overlay information to generate screen geometric data and write it into the instance buffer, aggregates text resources and icon resources into the texture atlas, calculates the fusion weight based on the urgency parameter, projection stability parameter, projection area ratio and style complexity parameter and writes it into the constant buffer, and constructs an AR video fusion overlay unit set including the texture atlas, the instance buffer and the constant buffer;

[0095] The fusion display module is used to bind the real-time scene frame buffer, traverse the AR video fusion and overlay unit set according to the drawing order index, use the fragment shader to perform linear combination calculation of the virtual overlay pixel color and the real-time scene pixel color based on the fusion weight to obtain the output pixel color, and submit the fused frame buffer to the display system.

[0096] It is worth noting that the specific workflow of the AR video fusion and overlay system for ultra-low latency provided in this embodiment of the invention is the same as that of the AR video fusion and overlay method for ultra-low latency described in the above embodiment, and will not be repeated here.

[0097] This invention also provides an AR video fusion and overlay device for ultra-low latency, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps described in the above embodiment of an AR video fusion and overlay method for ultra-low latency, for example... Figure 1 The steps S1 to S4 described above; or, when the processor executes the computer program, it implements the functions of each module in the above system embodiments.

[0098] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the ultra-low latency AR video fusion and overlay device.

[0099] The aforementioned AR video fusion and overlay device for ultra-low latency can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the device may also include input / output devices, network access devices, buses, etc.

[0100] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the ultra-low latency AR video fusion and overlay device, connecting various parts of the device via various interfaces and lines.

[0101] The memory can be used to store the computer program and / or modules. The processor implements various functions of the ultra-low latency AR video fusion and overlay device by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on the operation of the air conditioner controller, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0102] The module integrated into the ultra-low latency AR video fusion and overlay device, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0103] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0104] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for AR video fusion and overlay for ultra-low latency, characterized in that, The method includes: Obtain the display refresh event timestamp and the most recent screen submission timestamp, calculate the next refresh time based on the display refresh event timestamp, calculate the time difference between the next refresh time and the most recent screen submission timestamp as the remaining available time interval of the current display cycle, and generate display synchronization constraint information that includes the deadline and cycle availability determination results; The system receives a set of virtual overlay information to be overlaid. Based on the difference between the deadline in the display synchronization constraint information and the current system time, it obtains the remaining budget. For each piece of virtual overlay information in the set, it calculates the expected execution time by combining the projection area ratio, style complexity parameter, and urgency parameter. Virtual overlay information that satisfies the condition that the remaining budget is greater than or equal to the expected execution time is selected as a set of virtual overlay information that can be synchronized. The projection area ratio is obtained by reading the anchor point reference of the virtual overlay information, obtaining the position and orientation of the anchor point in the current world coordinate system from the anchoring result table, obtaining the current camera projection matrix and view matrix from the rendering pipeline, projecting the bounding geometry associated with the anchor point onto the image plane to obtain a two-dimensional bounding region, and dividing the pixel coverage of this region by the total number of pixels in the frame. The style complexity parameter is obtained by mapping the style field of the virtual overlay information according to the configuration table; The urgency parameter is a normalized level quantity written by the early warning strategy module when generating virtual overlay information; The anchor reference of each virtual overlay information in the synchronously overlayable virtual overlay information set is parsed to generate screen geometric data and written into the instance buffer. The anchor reference is the anchor identifier in the associated positioning calibration and tag anchoring result table. Text resources and icon resources are aggregated into the texture atlas. The fusion weight is calculated based on the urgency parameter, projection stability parameter, projection area ratio and style complexity parameter and written into the constant buffer. An AR video fusion overlay unit set containing the texture atlas, the instance buffer and the constant buffer is constructed. The real-time scene frame buffer is bound, and the AR video fusion and overlay unit set is traversed according to the drawing order. The fragment shader is used to perform linear combination calculation of the virtual overlay pixel color and the real-time scene pixel color based on the fusion weight to obtain the output pixel color. The fused frame buffer is then submitted to the display system.

2. The AR video fusion and overlay method for ultra-low latency according to claim 1, characterized in that, The calculation of the time difference between the next refresh time and the most recent screen submission timestamp as the remaining available time interval for the current display cycle includes: Record the first and second refresh timestamps of two consecutive refresh events, and calculate the difference between the second refresh timestamp and the first refresh timestamp as the refresh interval; The sum of the display refresh event timestamp and the refresh interval is determined as the next refresh time; Obtain the timestamp of the most recent screen submission from the graphics compositing interface; The remaining available time interval is obtained by performing a subtraction operation.

3. The AR video fusion and overlay method for ultra-low latency according to claim 1, characterized in that, The calculation of the estimated execution time for each piece of virtual overlay information in the set of virtual overlay information to be overlaid includes: Obtain the area weight coefficient, complexity weight coefficient, and urgency adjustment coefficient from the configuration coefficients; The estimated execution time is obtained through the following calculation: ； in, This refers to the proportion of the projected area of ​​the virtual overlay information onto the current real-time scene. This is a style complexity parameter. This is the urgency parameter, and its value is a normalized level. The configuration factor is derived from the calibration of the fusion pipeline of the target equipment before going live. The estimated execution time is the time to be executed. The higher the value of the urgency parameter, the shorter the calculated estimated execution time.

4. The AR video fusion and overlay method for ultra-low latency according to claim 1, characterized in that, After generating the synchronously overlayable virtual overlay information set, the method further includes: The virtual overlay information in the set of synchronously overlayable virtual overlay information is sorted; The sorting rules are as follows: priority is given to arranging according to the urgency parameter from high to low; if the urgency parameters are the same, they are arranged according to the proportion of the projected area from small to large. The sorted order will be used as the drawing order index when constructing the AR video fusion and overlay unit set later.

5. The AR video fusion and overlay method for ultra-low latency according to claim 1, characterized in that, The methods for obtaining the projection stability parameters include: Obtain the coordinates of the center point of the projection area of ​​the virtual overlay information in the current frame; Obtain the coordinates of the center point of the projection area of ​​the same virtual overlay information in the previous frame; Calculate the norm of the difference between the coordinates of the center point of the projection region in the current frame and the coordinates of the center point of the projection region in the previous frame; Divide the difference norm by the screen scale to obtain the displacement ratio, and subtract the displacement ratio from 1 to obtain the projection stability parameter.

6. The AR video fusion and overlay method for ultra-low latency according to claim 1, characterized in that, The calculation of the fusion weights includes: Obtain urgency parameters, projection stability parameters, projection area ratio, style complexity parameters, and dimensionless configuration coefficients; The sum of the product of the urgency parameter and the first coefficient, and the product of the projected stability parameter and the second coefficient, is calculated as a positive contribution term; The product of the projected area ratio, the style complexity parameter, and the third coefficient is calculated as a penalty term; Calculate the difference between the positive contribution term and the penalty term; If the difference is less than a preset lower threshold, the fusion weight is set to the preset lower threshold; if the difference is greater than a preset upper threshold, the fusion weight is set to the preset upper threshold; if the difference is between the preset lower threshold and the preset upper threshold, the difference is directly used as the fusion weight.

7. The AR video fusion and overlay method for ultra-low latency according to claim 1, characterized in that, The aggregation of text resources and icon resources to the texture atlas includes: Allocate a writable texture atlas for the current display cycle; The font rasterization module generates a glyph bitmap based on the text string index and writes the glyph bitmap into the first slot of the texture atlas; The icon bitmap is read from the resource cache based on the icon resource handle, and the icon bitmap is written into the second slot of the texture atlas; The texture sampling coordinate range corresponding to each virtual overlay information is recorded in the instance buffer, and the texture sampling coordinate range points to the first slot or the second slot.

8. The AR video fusion and overlay method for ultra-low latency according to claim 1, characterized in that, The method of using the fragment shader to perform a linear combination calculation of the virtual overlay pixel color and the real-time real-scene pixel color based on the fusion weight follows the following calculation logic: The output pixel color is equal to the product of the virtual overlay pixel color and the fusion weight, plus the product of the real-time real-scene pixel color and the difference between 1 and the fusion weight. The virtual overlay pixel color is obtained by sampling the texture atlas by the fragment shader, and the real-time scene pixel color comes from the bound real-time scene frame buffer.

9. The AR video fusion and overlay method for ultra-low latency according to claim 1, characterized in that, The method also includes a step for maintaining the fusion processing time estimate: Record the start time and submission completion time of the most recent N fusion operations, where N is a preset integer greater than 1; Calculate the time difference between the start time and the submission completion time of each fusion operation to obtain N time consumption data; The estimated fusion processing time is obtained by performing an arithmetic average or a weighted average on the N time-consuming data. When generating display synchronization constraint information, if the remaining available time interval is greater than or equal to the estimated fusion processing time, the cycle availability determination result of the current display cycle is marked as available; otherwise, it is marked as unavailable.

10. An AR video fusion and overlay system for ultra-low latency, characterized in that, The system includes: The constraint generation module is used to obtain the display refresh event timestamp and the most recent screen submission timestamp, calculate the next refresh time based on the display refresh event timestamp, calculate the time difference between the next refresh time and the most recent screen submission timestamp as the remaining available time interval of the current display cycle, and generate display synchronization constraint information that includes the deadline and cycle availability determination results. The information filtering module receives a set of virtual overlay information to be overlaid, obtains the remaining budget based on the difference between the deadline in the display synchronization constraint information and the current system time, calculates the expected execution time for each virtual overlay information in the set of virtual overlay information to be overlaid, and filters the virtual overlay information that meets the condition that the remaining budget is greater than or equal to the expected execution time as a set of virtual overlay information that can be synchronously overlaid. The method for obtaining the projection area ratio is as follows: read the anchor point reference of the virtual overlay information, obtain the position and orientation of the anchor point in the current world coordinate system from the anchoring result table, obtain the current camera projection matrix and view matrix from the rendering pipeline, project the bounding geometry associated with the anchor point onto the image plane to obtain a two-dimensional bounding region, and divide the pixel coverage of the region by the total number of pixels in the whole frame. The style complexity parameter is obtained by mapping the style field of the virtual overlay information according to the configuration table; The urgency parameter is a normalized level quantity written by the early warning strategy module when generating virtual overlay information; The unit construction module parses the anchor reference of each virtual overlay information in the synchronously overlayable virtual overlay information set to generate screen geometric data and write it into the instance buffer. The anchor reference is the anchor identifier in the associated positioning calibration and tag anchoring result table. It aggregates text resources and icon resources into the texture atlas, calculates the fusion weight based on the urgency parameter, projection stability parameter, projection area ratio and style complexity parameter and writes it into the constant buffer, and constructs an AR video fusion overlay unit set containing the texture atlas, the instance buffer and the constant buffer. The fusion display module is used to bind the real-time scene frame buffer, traverse the AR video fusion and overlay unit set according to the drawing order index, use the fragment shader to perform linear combination calculation of the virtual overlay pixel color and the real-time scene pixel color based on the fusion weight to obtain the output pixel color, and submit the fused frame buffer to the display system.

Images